* feat(core): add image support in tool returns [LET-7140]
Enable tool_return to support both string and ImageContent content parts,
matching the pattern used for user message inputs. This allows tools
executed client-side to return images back to the agent.
Changes:
- Add LettaToolReturnContentUnion type for text/image content parts
- Update ToolReturn schema to accept Union[str, List[content parts]]
- Update converters for each provider:
- OpenAI Chat Completions: placeholder text for images
- OpenAI Responses API: full image support
- Anthropic: full image support with base64
- Google: placeholder text for images
- Add resolve_tool_return_images() for URL-to-base64 conversion
- Make create_approval_response_message_from_input() async
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(core): support images in Google tool returns as sibling parts
Following the gemini-cli pattern: images in tool returns are sent as
sibling inlineData parts alongside the functionResponse, rather than
inside it.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* test(core): add integration tests for multi-modal tool returns [LET-7140]
Tests verify that:
- Models with image support (Anthropic, OpenAI Responses API) can see
images in tool returns and identify the secret text
- Models without image support (Chat Completions) get placeholder text
and cannot see the actual image content
- Tool returns with images persist correctly in the database
Uses secret.png test image containing hidden text "FIREBRAWL" that
models must identify to pass the test.
Also fixes misleading comment about Anthropic only supporting base64
images - they support URLs too, we just pre-resolve for consistency.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: simplify tool return image support implementation
Reduce code verbosity while maintaining all functionality:
- Extract _resolve_url_to_base64() helper in message_helper.py (eliminates duplication)
- Add _get_text_from_part() helper for text extraction
- Add _get_base64_image_data() helper for image data extraction
- Add _tool_return_to_google_parts() to simplify Google implementation
- Add _image_dict_to_data_url() for OpenAI Responses format
- Use walrus operator and list comprehensions where appropriate
- Add integration_test_multi_modal_tool_returns.py to CI workflow
Net change: -120 lines while preserving all features and test coverage.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(tests): improve prompt for multi-modal tool return tests
Make prompts more direct to reduce LLM flakiness:
- Simplify tool description: "Retrieves a secret image with hidden text. Call this function to get the image."
- Change user prompt from verbose request to direct command: "Call the get_secret_image function now."
- Apply to both test methods
This reduces ambiguity and makes tool calling more reliable across different LLM models.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix bugs
* test(core): add google_ai/gemini-2.0-flash-exp to multi-modal tests
Add Gemini model to test coverage for multi-modal tool returns. Google AI already supports images in tool returns via sibling inlineData parts.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(ui): handle multi-modal tool_return type in frontend components
Convert Union<string, LettaToolReturnContentUnion[]> to string for display:
- ViewRunDetails: Convert array to '[Image here]' placeholder
- ToolCallMessageComponent: Convert array to '[Image here]' placeholder
Fixes TypeScript errors in web, desktop-ui, and docker-ui type-checks.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Caren Thomas <carenthomas@gmail.com>
* fix: wrap turbopuffer vector writes in thread pool
Turbopuffer library does CPU-intensive base64 encoding of vectors
synchronously in async functions (_async_transform_recursive →
b64encode_vector), blocking the event loop during file uploads.
Solution: Created _run_turbopuffer_write_in_thread() helper that runs
turbopuffer writes in an isolated event loop within a worker thread.
Applied to all vector write operations:
- insert_tools()
- insert_archival_memories()
- insert_messages()
- insert_file_passages()
This prevents pybase64.b64encode_as_string() from blocking the main
event loop during vector encoding.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: wrap all turbopuffer operations in thread pool
Extended the thread pool wrapping to ALL turbopuffer write operations,
including delete operations, for complete isolation from the main event loop.
All turbopuffer namespace.write() calls now run in isolated event loops
within worker threads, preventing any potential CPU work from blocking.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: Add conversation_id filtering to message list and search endpoints
Add optional conversation_id parameter to filter messages by conversation:
- client.agents.messages.list
- client.messages.list
- client.messages.search
Changes:
- Added conversation_id field to MessageSearchRequest and SearchAllMessagesRequest schemas
- Added conversation_id filtering to list_messages in message_manager.py
- Updated get_agent_recall_async and get_all_messages_recall_async in server.py
- Added conversation_id query parameter to router endpoints
- Updated Turbopuffer client to support conversation_id filtering in searches
Fixes#8320🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Charles Packer <cpacker@users.noreply.github.com>
* add conversation_id to message and tpuf
* default messages filter for backward compatibility
* add test and auto gen
* fix integration test
* fix test
* update test
---------
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: Charles Packer <cpacker@users.noreply.github.com>
Co-authored-by: christinatong01 <christina@letta.com>
This fixes the asyncpg.exceptions.CharacterNotInRepertoireError that occurs
when tool returns contain null bytes (0x00), which PostgreSQL TEXT columns
reject in UTF-8 encoding.
Changes:
- Add sanitize_null_bytes() function to recursively remove null bytes from strings
- Update json_dumps() to sanitize data before serialization
- Apply sanitization in converters.py for tool_calls, tool_returns, approvals, and message_content
- Add comprehensive unit tests
Fixes#8014🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Kian Jones <11655409+kianjones9@users.noreply.github.com>
* fix: run PBKDF2 in thread pool to prevent event loop freeze
Problem: Event loop freezes for 100-500ms during secret decryption, blocking
all HTTP requests and async operations. The diagnostic monitor detected the
main thread stuck in PBKDF2 HMAC SHA256 computation at:
apps/core/letta/helpers/crypto_utils.py:51 (_derive_key)
apps/core/letta/schemas/secret.py:161 (get_plaintext)
Root cause: PBKDF2 with 100k iterations is intentionally CPU-intensive for
security, but running it synchronously on the main thread blocks the event loop.
Stack trace showed:
Thread 1 (Main): PBKDF2HMAC -> SHA256_Final -> sha256_block_data_order_avx2
Event loop watchdog: Detected freeze at 01:11:44 (request started 01:12:03)
Solution:
1. Run PBKDF2 in ThreadPoolExecutor to avoid blocking event loop
2. Add async versions of encrypt/decrypt methods
3. Add LRU cache for derived keys (deterministic results)
4. Add async get_plaintext_async() method to Secret class
Changes:
- apps/core/letta/helpers/crypto_utils.py:
- Added ThreadPoolExecutor for crypto operations
- Added @lru_cache(maxsize=256) to _derive_key_cached()
- Added _derive_key_async() using loop.run_in_executor()
- Added encrypt_async() and decrypt_async() methods
- Added warnings to sync methods about blocking behavior
- apps/core/letta/schemas/secret.py:
- Added get_plaintext_async() method
- Added warnings to get_plaintext() about blocking behavior
Benefits:
- Event loop no longer freezes during secret decryption
- HTTP requests continue processing while crypto runs in background
- Derived keys are cached, reducing CPU usage for repeated operations
- Backward compatible - sync methods still work for non-async code
Performance impact:
- Before: 100-500ms event loop block per decryption
- After: 100-500ms in thread pool (non-blocking) + LRU cache hits ~0.1ms
Next steps (follow-up PRs):
- Migrate all async callsites to use get_plaintext_async()
- Add metrics to track sync vs async usage
- Consider reducing PBKDF2 iterations if security allows
* update
* test
---------
Co-authored-by: Letta Bot <jinjpeng@gmail.com>
* initial commit
* Add database migration for compaction_settings field
This migration adds the compaction_settings column to the agents table
to support customized summarization configuration for each agent.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix
* rename
* update apis
* fix tests
* update web test
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Kian Jones <kian@letta.com>
* fix: exclude common API key prefixes from encryption detection
Add a list of known API key prefixes (OpenAI, Anthropic, GitHub, AWS,
Slack, etc.) to prevent is_encrypted() from incorrectly identifying
plaintext credentials as encrypted values.
* update
* test
* fix: detect and fail on malformed approval responses
* fix: guard against None approvals in utils.py
* fix: add extra warning
* fix: stop silent drops in deserialize_approvals
* fix: patch v3 stream error handling to prevent sending end_turn after an error occurs, and ensures stop_reason is always set when an error occurs
* fix: Prevents infinite client hangs by ensuring a terminal event is ALWAYS sent
* fix: Ensures terminal events are sent even if inner stream generator fails to
send them
* change my PR to match Caren's
* add path parameter validation for agent id first
* remove old import
* remove old agent_id_pattern pattern
* add example and fix max/min calculation to include hyphen
* fix regex string interpolation
* example deprecated in favour of examples
* openapi autogen
* change template test to expect 422
* fix 422 swallow
* expect 422 or 400
* rewrite error codes
* fix hallucinated uuid
* tweaked error message test
* print docker logs on failure
* Implement child tool rules args override
* Add zod types
* Run fern autogen and put ToolCallNode in new field
* Fix test_tool_rule_solver.py
* Fix types
* Fix types again
* Add tests to tool rule solver
