* auto fixes
* auto fix pt2 and transitive deps and undefined var checking locals()
* manual fixes (ignored or letta-code fixed)
* fix circular import
* remove all ignores, add FastAPI rules and Ruff rules
* add ty and precommit
* ruff stuff
* ty check fixes
* ty check fixes pt 2
* error on invalid
* fix: handle Anthropic overloaded_error in streaming interfaces
* fix: handle Unicode surrogates in OpenAI requests
Sanitize Unicode surrogate pairs before sending requests to OpenAI API.
Surrogate pairs (U+D800-U+DFFF) are UTF-16 encoding artifacts that cause
UnicodeEncodeError when encoding to UTF-8.
Fixes Datadog error: 'utf-8' codec can't encode character '\ud83c' in
position 326605: surrogates not allowed
* fix: handle UnicodeEncodeError from lone Unicode surrogates in OpenAI requests
Improved sanitize_unicode_surrogates() to explicitly filter out lone
surrogate characters (U+D800 to U+DFFF) which are invalid in UTF-8.
Previous implementation used errors='ignore' which could still fail in
edge cases. New approach directly checks Unicode code points and removes
any surrogates before data reaches httpx encoding.
Also added sanitization to stream_async_responses() method which was
missing it.
Fixes: 'utf-8' codec can't encode character '\ud83c' in position X:
surrogates not allowed
**Problem:**
Error logs showed empty detail fields when exceptions had no message:
```
Error during step processing:
Run run-xxx stopped with unknown error: , error_data: {...'detail': ''}
```
This made debugging production issues difficult as the actual error type
was hidden.
**Root Cause:**
Python exceptions created with no arguments (e.g., `Exception()` or caught
and re-raised in certain ways) have `str(e) == ""`:
```python
e = Exception()
str(e) # Returns ""
repr(e) # Returns "Exception()"
```
When exceptions with empty string representations were caught, all logging
and error messages showed blank details.
**Fix:**
Use `str(e) or repr(e)` fallback pattern in 3 places:
1. `letta_agent_v3.py` stream() exception handler (line 406)
2. `letta_agent_v3.py` step() exception handler (line 928)
3. `streaming_service.py` generic exception handler (line 469)
**Result:**
- Error logs now show `Exception()` or similar instead of empty string
- Helps identify exception types even when message is missing
- Better production debugging without changing exception handling logic
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
fix: use shared event + .athrow() to properly set stream_was_cancelled flag
**Problem:**
When a run is cancelled via /cancel endpoint, `stream_was_cancelled` remained
False because `RunCancelledException` was raised in the consumer code (wrapper),
which closes the generator from outside. This causes Python to skip the
generator's except blocks and jump directly to finally with the wrong flag value.
**Solution:**
1. Shared `asyncio.Event` registry for cross-layer cancellation signaling
2. `cancellation_aware_stream_wrapper` sets the event when cancellation detected
3. Wrapper uses `.athrow()` to inject exception INTO generator (not consumer-side raise)
4. All streaming interfaces check event in `finally` block to set flag correctly
5. `streaming_service.py` handles `RunCancelledException` gracefully, yields [DONE]
**Changes:**
- streaming_response.py: Event registry + .athrow() injection + graceful handling
- openai_streaming_interface.py: 3 classes check event in finally
- gemini_streaming_interface.py: Check event in finally
- anthropic_*.py: Catch RunCancelledException
- simple_llm_stream_adapter.py: Create & pass event to interfaces
- streaming_service.py: Handle RunCancelledException, yield [DONE], skip double-update
- routers/v1/{conversations,runs}.py: Pass event to wrapper
- integration_test_human_in_the_loop.py: New test for approval + cancellation
**Tests:**
- test_tool_call with cancellation (OpenAI models) ✅
- test_approve_with_cancellation (approval flow + concurrent cancel) ✅
**Known cosmetic warnings (pre-existing):**
- "Run already in terminal state" - agent loop tries to update after /cancel
- "Stream ended without terminal event" - background streaming timing race
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* feat: allow client-side tools to be specified in request
Add `client_tools` field to LettaRequest to allow passing tool schemas
at message creation time without requiring server-side registration.
When the agent calls a client-side tool, execution pauses with
stop_reason=requires_approval for the client to provide tool returns.
- Add ClientToolSchema class for request-level tool schemas
- Merge client tools with agent tools in _get_valid_tools()
- Treat client-side tool calls as requiring approval
- Add integration tests for client-side tools flow
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* test: add comprehensive end-to-end test for client-side tools
Update integration test to verify the complete flow:
- Agent calls client-side tool and pauses
- Client provides tool return with secret code
- Agent processes and responds
- User asks about the code, agent recalls it
- Validate full conversation history makes sense
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* update apis
* fix: client-side tools schema format and test assertions
- Use flat schema format for client tools (matching t.json_schema)
- Support both object and dict access for client tools
- Fix stop_reason assertions to access .stop_reason attribute
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: simplify client_tools access pattern
ClientToolSchema objects always have .name attribute
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add client_tools parameter to LettaAgentV2 for API compatibility
V2 agent doesn't use client_tools but needs the parameter
to match the base class signature.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* revert: remove client_tools from LettaRequestConfig
Client-side tools don't work with background jobs since
there's no client present to provide tool returns.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add client_tools parameter to SleeptimeMultiAgent classes
Add client_tools to step() and stream() methods in:
- SleeptimeMultiAgentV3
- SleeptimeMultiAgentV4
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: regenerate API specs for client_tools support
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: add zai provider support
* add zai_api_key secret to deploy-core
* add to justfile
* add testing, provider integration skill
* enable zai key
* fix zai test
* clean up skill a little
* small changes
* fix: detect and fail on malformed approval responses
* fix: guard against None approvals in utils.py
* fix: add extra warning
* fix: stop silent drops in deserialize_approvals
* fix: patch v3 stream error handling to prevent sending end_turn after an error occurs, and ensures stop_reason is always set when an error occurs
* fix: Prevents infinite client hangs by ensuring a terminal event is ALWAYS sent
* fix: Ensures terminal events are sent even if inner stream generator fails to
send them
* fix: patch sse streaming errors
* fix: don't re-raise, but log explicitly with sentry
* chore: cleanup comments
* fix: revert change from #5907, also make sure to write out a [DONE] to close the stream
* Fix agent loop continuing after cancellation in letta_agent_v3
Bug: When a run is cancelled, _check_run_cancellation() sets
self.should_continue=False and returns early from _step(), but the outer
for loop (line 245) continues to the next iteration, executing subsequent
steps even though cancellation was requested.
Symptom: User hits cancel during step 1, backend marks run as cancelled,
but agent continues executing steps 2, 3, etc.
Root cause: After the 'async for chunk in response' loop completes (line 255),
there was no check of self.should_continue before continuing to the next
iteration of the outer step loop.
Fix: Added 'if not self.should_continue: break' check after the inner loop
to exit the outer step loop when cancellation is detected. This makes v3
consistent with v2 which already had this check (line 306-307).
🐾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* add integration tests
* fix: misc fixes required to get cancellations to work on letta code localhost
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Sarah Wooders <sarahwooders@gmail.com>
