OpenAI Chat Completions requires tool message content to be a string,
so images in tool returns were silently replaced with [Image omitted].
Now: text stays in the tool return, images get injected as a user
message right after. The model actually sees what the tool saw.
to_openai_dict also cleaned up — image handling lives in
to_openai_dicts_from_list where it can inject the extra message.
* auto fixes
* auto fix pt2 and transitive deps and undefined var checking locals()
* manual fixes (ignored or letta-code fixed)
* fix circular import
* remove all ignores, add FastAPI rules and Ruff rules
* add ty and precommit
* ruff stuff
* ty check fixes
* ty check fixes pt 2
* error on invalid
Avoid failing message-list endpoints when historical send_message tool calls are missing the expected message argument by logging and skipping malformed entries during conversion.
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* fix: handle system messages with mixed TextContent + ImageContent
System messages injected by external tools (e.g. packify.ai MCP) can
contain both TextContent and ImageContent. The assertions in
to_openai_responses_dicts and to_anthropic_dict expected exactly one
TextContent, causing AssertionError in production.
Extract all text parts and join them, matching how to_openai_dict
already handles this case.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: replace asserts with logger.warning + graceful skip
Asserts are the wrong tool for production input validation — if a
system message has only non-text content, we should warn and skip
rather than crash the request.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: add conversation_id to message search results
Add conversation_id field to all *MessageListResult classes
(SystemMessageListResult, UserMessageListResult, ReasoningMessageListResult,
AssistantMessageListResult) so that conversation IDs are returned from
the /messages/search endpoint alongside agent IDs.
Fixes#9055
Co-authored-by: Charles Packer <cpacker@users.noreply.github.com>
* chore: regenerate SDK and OpenAPI spec
Regenerate autogenerated files after adding conversation_id to
message search result schemas.
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
---------
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: Charles Packer <cpacker@users.noreply.github.com>
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
fix: gracefully skip assistant messages with empty content in LLM format conversion
**Problem:**
Context window calculation crashed with AssertionError when converting messages
to Google/Anthropic/OpenAI format:
```
AssertionError at line 2047: assert self.tool_calls is not None or
text_content is not None or len(self.content) > 1
```
This happened when loading agents with old/malformed messages that had
`content=None` or `content=[]` in the database.
**Root Cause:**
The Message ORM model allows `content: Optional[List[...]] = None` (line 252),
but format conversion methods assumed content would always have extractable text
or tool calls.
Scenarios that triggered crashes:
1. Assistant message with `content=None` (old migrations/edge cases)
2. Assistant message with `content=[]` (message creation bugs)
3. Assistant message with single non-text content that doesn't match extraction logic
**Fix:**
Replaced assertions with defensive checks in 3 conversion methods:
1. `to_google_dict()` (line 2054) - Return None to skip unconvertible messages
2. `to_openai_responses_api_dicts()` (line 1476) - Return early to skip
3. `to_anthropic_dict()` (line 1794) - Return None to skip
Pattern: Check for empty content, return None/early to skip gracefully.
**Result:**
- Context window calculation no longer crashes on malformed/old messages
- Messages with no convertible content are silently skipped
- Consistent with existing Anthropic reasoning-only message handling (line 1308)
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* feat(core): add image support in tool returns [LET-7140]
Enable tool_return to support both string and ImageContent content parts,
matching the pattern used for user message inputs. This allows tools
executed client-side to return images back to the agent.
Changes:
- Add LettaToolReturnContentUnion type for text/image content parts
- Update ToolReturn schema to accept Union[str, List[content parts]]
- Update converters for each provider:
- OpenAI Chat Completions: placeholder text for images
- OpenAI Responses API: full image support
- Anthropic: full image support with base64
- Google: placeholder text for images
- Add resolve_tool_return_images() for URL-to-base64 conversion
- Make create_approval_response_message_from_input() async
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(core): support images in Google tool returns as sibling parts
Following the gemini-cli pattern: images in tool returns are sent as
sibling inlineData parts alongside the functionResponse, rather than
inside it.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* test(core): add integration tests for multi-modal tool returns [LET-7140]
Tests verify that:
- Models with image support (Anthropic, OpenAI Responses API) can see
images in tool returns and identify the secret text
- Models without image support (Chat Completions) get placeholder text
and cannot see the actual image content
- Tool returns with images persist correctly in the database
Uses secret.png test image containing hidden text "FIREBRAWL" that
models must identify to pass the test.
Also fixes misleading comment about Anthropic only supporting base64
images - they support URLs too, we just pre-resolve for consistency.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: simplify tool return image support implementation
Reduce code verbosity while maintaining all functionality:
- Extract _resolve_url_to_base64() helper in message_helper.py (eliminates duplication)
- Add _get_text_from_part() helper for text extraction
- Add _get_base64_image_data() helper for image data extraction
- Add _tool_return_to_google_parts() to simplify Google implementation
- Add _image_dict_to_data_url() for OpenAI Responses format
- Use walrus operator and list comprehensions where appropriate
- Add integration_test_multi_modal_tool_returns.py to CI workflow
Net change: -120 lines while preserving all features and test coverage.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(tests): improve prompt for multi-modal tool return tests
Make prompts more direct to reduce LLM flakiness:
- Simplify tool description: "Retrieves a secret image with hidden text. Call this function to get the image."
- Change user prompt from verbose request to direct command: "Call the get_secret_image function now."
- Apply to both test methods
This reduces ambiguity and makes tool calling more reliable across different LLM models.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix bugs
* test(core): add google_ai/gemini-2.0-flash-exp to multi-modal tests
Add Gemini model to test coverage for multi-modal tool returns. Google AI already supports images in tool returns via sibling inlineData parts.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(ui): handle multi-modal tool_return type in frontend components
Convert Union<string, LettaToolReturnContentUnion[]> to string for display:
- ViewRunDetails: Convert array to '[Image here]' placeholder
- ToolCallMessageComponent: Convert array to '[Image here]' placeholder
Fixes TypeScript errors in web, desktop-ui, and docker-ui type-checks.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Caren Thomas <carenthomas@gmail.com>
This fixes a 400 INVALID_ARGUMENT error from Google's Gemini API where
function calls were missing required thought_signature in functionCall parts.
Changes:
- Allow signatures when self.model is None (backwards compatibility for
older messages that may not have had their model field set)
- Only add thought_signature to the FIRST function call for parallel
tool calls, per Google's docs
- Take the first non-None signature found (don't keep overwriting)
Reference: https://ai.google.dev/gemini-api/docs/thought-signaturesCloses#8589🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: datadog-official[bot] <datadog-official[bot]@users.noreply.github.com>
Co-authored-by: Kian Jones <11655409+kianjones9@users.noreply.github.com>
* fix: handle missing tool_call_id in Anthropic message conversion
- Add null check for self.tool_returns before iterating
- Fall back to message's tool_call_id when tool_return.tool_call_id is None
- Improve error message to show actual tool name from message.name
- Only raise error if no valid tool_call_id is available from either source
This fixes the error "Anthropic API requires tool_use_id to be set" that
occurs when a ToolReturn object in the database doesn't have tool_call_id
set, by using the message-level tool_call_id as a fallback.
Fixes#8379🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: datadog-official[bot] <datadog-official[bot]@users.noreply.github.com>
Co-Authored-By: Letta <noreply@letta.com>
* fix: restrict tool_call_id fallback to single tool returns
The message-level `self.tool_call_id` is set to the first tool return's ID
for legacy compatibility. For parallel tool calls with multiple tool_returns,
using this as a fallback would incorrectly assign the first tool return's ID
to all subsequent returns missing their own ID.
This change:
- Only allows the fallback when there's exactly one tool return
- For multiple tool returns, each must have its own ID or raise an error
- Adds tool return index to error messages for better debugging
Co-authored-by: Kian Jones <kianjones9@users.noreply.github.com>
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: datadog-official[bot] <datadog-official[bot]@users.noreply.github.com>
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Kian Jones <11655409+kianjones9@users.noreply.github.com>
* feat: Add conversation_id filtering to message list and search endpoints
Add optional conversation_id parameter to filter messages by conversation:
- client.agents.messages.list
- client.messages.list
- client.messages.search
Changes:
- Added conversation_id field to MessageSearchRequest and SearchAllMessagesRequest schemas
- Added conversation_id filtering to list_messages in message_manager.py
- Updated get_agent_recall_async and get_all_messages_recall_async in server.py
- Added conversation_id query parameter to router endpoints
- Updated Turbopuffer client to support conversation_id filtering in searches
Fixes#8320🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Charles Packer <cpacker@users.noreply.github.com>
* add conversation_id to message and tpuf
* default messages filter for backward compatibility
* add test and auto gen
* fix integration test
* fix test
* update test
---------
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: Charles Packer <cpacker@users.noreply.github.com>
Co-authored-by: christinatong01 <christina@letta.com>
* fix: prevent empty reasoning messages in streaming interfaces
Prevents empty "Thinking..." indicators from appearing in clients by
filtering out reasoning messages with no content at the source.
Changes:
- Gemini: Don't emit ReasoningMessage when only thought_signature exists
- Gemini: Only emit reasoning content if text is non-empty
- Anthropic: Don't emit ReasoningMessage for BetaSignatureDelta
- Anthropic: Only emit reasoning content if thinking text is non-empty
This fixes the issue where providers send signature metadata before
actual thinking content, causing empty reasoning blocks to appear
in the UI after responses complete.
Affects: Gemini reasoning, Anthropic extended thinking
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: handle Anthropic thinking signature correctly
- Only include 'signature' in Anthropic message payload if it is not None (fixes BadRequestError).
- Capture and attach 'signature' to ReasoningMessage in streaming interface.
* fix(anthropic): attach signature to last reasoning message in stream
---------
Co-authored-by: Letta <noreply@letta.com>
fix AssistantMessage validation error when content is dict
validate_function_response can return either a string or dict, but
AssistantMessage.content expects a string. When a tool returns a dict
like {'tofu': 1, 'mofu': 1, 'bofu': 1}, it needs to be JSON-serialized
before passing to AssistantMessage.
Fixes: pydantic_core._pydantic_core.ValidationError: 2 validation errors for AssistantMessage
* claude code first pass
* rename routes
* search_messages and list_messages
* revert agents messagesearch
* generate api
* fix backend for list all messages
* request for message search
* return list of letta message
* add tests
* error in archive endpoint
* archive delete return type wrong
* optional params for archive creation
* add passage to tpuf on create
* fix archive manager
* support global passage search
* search by agent
* just do basic org wide search for now
* change message test to be about fresh data, cleanup after
---------
Co-authored-by: Ari Webb <ari@letta.com>
* fix(core): sanitize messages to anthropic in the main path the same way (or similar) to how we do it in the token counter
* fix: also patch poison error in backend by filtering lazily
* fix: remap streaming errors (what the fuck)
* fix: dedupe tool clals
* fix: cleanup, removed try/catch
* fix: patch hole in the fallback summarizer where we weren't actually truncating
* fix: remove no-op
* chore: comment
* fix: simplify the new fallback
* fix: properly handle images in summarizer payload
* first hack
* clean up
* first implementation working
* revert package-lock
* remove openai test
* error throw
* typo
* Update integration_test_send_message_v2.py
* Update integration_test_send_message_v2.py
* refine test
* Only make changes for openai non streaming
* Add tests
---------
Co-authored-by: Ari Webb <ari@letta.com>
Co-authored-by: Matt Zhou <mattzh1314@gmail.com>
* letta coded
* migrate to stainless from fern
* revert core workflows
* fix if statement
* fix typo
* run on self-hosted ci runners
* add empty check
* change file
* fix upstream renaming and special character escaping
* fix letta-code with opus
* remove random client type import
* remove env localhost
* remove fialing tests
* ignore ts for now
* fix caching maybe
* tar.gz -> whl
* retain name metadata
* don't build on cache hit
* add sdk_v1 tests
* refactor not to use warnings.warn
* temp circular import fix maybe unecessary/bnad
* fix Deprecation warning
* fix deprecation warning and mcp thing?
* revert changes to mcp server test
* fix deprecation warning
* claude coded first pass
* fix test cases to expect errors instead
* fix this
* let's see how letta-code did
* claude
* fix tests, remove dangling comments, retrofit all managers functions with decorator
* revert to main for these since we are not erroring on invalid tool and block ids
* reorder decorators
* finish refactoring test cases
* reorder agent_manager decorators and fix test tool manager
* add decorator on missing managers
* fix id sources
* remove redundant check
* uses enum now
* move to enum
* fix: patch the issue with the GET failing (tested)
* fix(web): patch for the FE UI
* parse func_response
---------
Co-authored-by: Caren Thomas <carenthomas@gmail.com>
* wip
* Fix parallel tool calling interface
* wip
* wip adapt using id field
* Integrate new multi tool return schemas into parallel tool calling
* Remove example script
* Reset changes to llm stream adapter since old agent loop should not enable parallel tool calling
* Clean up fallback logic for extracting tool calls
* Remove redundant check
* Simplify logic
* Clean up logic in handle ai response
* Fix tests
* Write anthropic dict conversion to be back compatible
* wip
* Double write tool call id for legacy reasons
* Fix override args failures
* Patch for approvals
* Revert comments
* Remove extraneous prints
