* feat(core): add image support in tool returns [LET-7140]
Enable tool_return to support both string and ImageContent content parts,
matching the pattern used for user message inputs. This allows tools
executed client-side to return images back to the agent.
Changes:
- Add LettaToolReturnContentUnion type for text/image content parts
- Update ToolReturn schema to accept Union[str, List[content parts]]
- Update converters for each provider:
- OpenAI Chat Completions: placeholder text for images
- OpenAI Responses API: full image support
- Anthropic: full image support with base64
- Google: placeholder text for images
- Add resolve_tool_return_images() for URL-to-base64 conversion
- Make create_approval_response_message_from_input() async
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(core): support images in Google tool returns as sibling parts
Following the gemini-cli pattern: images in tool returns are sent as
sibling inlineData parts alongside the functionResponse, rather than
inside it.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* test(core): add integration tests for multi-modal tool returns [LET-7140]
Tests verify that:
- Models with image support (Anthropic, OpenAI Responses API) can see
images in tool returns and identify the secret text
- Models without image support (Chat Completions) get placeholder text
and cannot see the actual image content
- Tool returns with images persist correctly in the database
Uses secret.png test image containing hidden text "FIREBRAWL" that
models must identify to pass the test.
Also fixes misleading comment about Anthropic only supporting base64
images - they support URLs too, we just pre-resolve for consistency.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: simplify tool return image support implementation
Reduce code verbosity while maintaining all functionality:
- Extract _resolve_url_to_base64() helper in message_helper.py (eliminates duplication)
- Add _get_text_from_part() helper for text extraction
- Add _get_base64_image_data() helper for image data extraction
- Add _tool_return_to_google_parts() to simplify Google implementation
- Add _image_dict_to_data_url() for OpenAI Responses format
- Use walrus operator and list comprehensions where appropriate
- Add integration_test_multi_modal_tool_returns.py to CI workflow
Net change: -120 lines while preserving all features and test coverage.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(tests): improve prompt for multi-modal tool return tests
Make prompts more direct to reduce LLM flakiness:
- Simplify tool description: "Retrieves a secret image with hidden text. Call this function to get the image."
- Change user prompt from verbose request to direct command: "Call the get_secret_image function now."
- Apply to both test methods
This reduces ambiguity and makes tool calling more reliable across different LLM models.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix bugs
* test(core): add google_ai/gemini-2.0-flash-exp to multi-modal tests
Add Gemini model to test coverage for multi-modal tool returns. Google AI already supports images in tool returns via sibling inlineData parts.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(ui): handle multi-modal tool_return type in frontend components
Convert Union<string, LettaToolReturnContentUnion[]> to string for display:
- ViewRunDetails: Convert array to '[Image here]' placeholder
- ToolCallMessageComponent: Convert array to '[Image here]' placeholder
Fixes TypeScript errors in web, desktop-ui, and docker-ui type-checks.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Caren Thomas <carenthomas@gmail.com>
* feat: byok provider models in db also
* make tests and sync api
* fix inconsistent state with recreating provider of same name
* fix sync on byok creation
* update revision
* move stripe code for testing purposes
* revert
* add refresh byok models endpoint
* just stage publish api
* add tests
* reorder revision
* add test for name clashes
feat: update documentation and add new tutorials for memory blocks and agent collaboration
- Updated navigation paths in docs.yml to reflect new tutorial locations.
- Added comprehensive guides on shared memory blocks and attaching/detaching memory blocks.
- Enhanced existing documentation for memory blocks with examples and best practices.
- Corrected API key references in prebuilt tools documentation.
These changes aim to improve user understanding and facilitate multi-agent collaboration through shared memory systems.
* feat: Add conversation_id filtering to message list and search endpoints
Add optional conversation_id parameter to filter messages by conversation:
- client.agents.messages.list
- client.messages.list
- client.messages.search
Changes:
- Added conversation_id field to MessageSearchRequest and SearchAllMessagesRequest schemas
- Added conversation_id filtering to list_messages in message_manager.py
- Updated get_agent_recall_async and get_all_messages_recall_async in server.py
- Added conversation_id query parameter to router endpoints
- Updated Turbopuffer client to support conversation_id filtering in searches
Fixes#8320🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Charles Packer <cpacker@users.noreply.github.com>
* add conversation_id to message and tpuf
* default messages filter for backward compatibility
* add test and auto gen
* fix integration test
* fix test
* update test
---------
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: Charles Packer <cpacker@users.noreply.github.com>
Co-authored-by: christinatong01 <christina@letta.com>
* feat: add message_types filter to list messages endpoint
Add the ability to filter messages by type when listing message history
via GET /v1/agents/{agent_id}/messages. This brings parity with the
create message endpoint which already supports include_return_message_types.
Changes:
- Add message_types query parameter to list_messages endpoint in agents.py
- Add message_types parameter to get_agent_recall_async in server.py
- Filter messages by message_type after LettaMessage conversion
- Add test for message_types filtering
Closes#8277
Written by Cameron ◯ Letta Code
> "Simplicity is the ultimate sophistication." - Leonardo da Vinci
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: regenerate OpenAPI spec and SDK for message_types filter
🐧 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
Written by Cameron ◯ Letta Code
"The only way to do great work is to love what you do." - Steve Jobs
---------
Co-authored-by: Letta <noreply@letta.com>
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore: next
* chore; udpate secrets
* feat: allow client-side tools to be specified in request
Add `client_tools` field to LettaRequest to allow passing tool schemas
at message creation time without requiring server-side registration.
When the agent calls a client-side tool, execution pauses with
stop_reason=requires_approval for the client to provide tool returns.
- Add ClientToolSchema class for request-level tool schemas
- Merge client tools with agent tools in _get_valid_tools()
- Treat client-side tool calls as requiring approval
- Add integration tests for client-side tools flow
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* test: add comprehensive end-to-end test for client-side tools
Update integration test to verify the complete flow:
- Agent calls client-side tool and pauses
- Client provides tool return with secret code
- Agent processes and responds
- User asks about the code, agent recalls it
- Validate full conversation history makes sense
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* update apis
* fix: client-side tools schema format and test assertions
- Use flat schema format for client tools (matching t.json_schema)
- Support both object and dict access for client tools
- Fix stop_reason assertions to access .stop_reason attribute
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: simplify client_tools access pattern
ClientToolSchema objects always have .name attribute
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add client_tools parameter to LettaAgentV2 for API compatibility
V2 agent doesn't use client_tools but needs the parameter
to match the base class signature.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* revert: remove client_tools from LettaRequestConfig
Client-side tools don't work with background jobs since
there's no client present to provide tool returns.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add client_tools parameter to SleeptimeMultiAgent classes
Add client_tools to step() and stream() methods in:
- SleeptimeMultiAgentV3
- SleeptimeMultiAgentV4
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: regenerate API specs for client_tools support
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: add zai provider support
* add zai_api_key secret to deploy-core
* add to justfile
* add testing, provider integration skill
* enable zai key
* fix zai test
* clean up skill a little
* small changes
* initial commit
* Add database migration for compaction_settings field
This migration adds the compaction_settings column to the agents table
to support customized summarization configuration for each agent.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix
* rename
* update apis
* fix tests
* update web test
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Kian Jones <kian@letta.com>
