Commit Graph

11 Commits

Author SHA1 Message Date
cthomas
c162de5127 fix: use shared event + .athrow() to properly set stream_was_cancelle… (#9019)
fix: use shared event + .athrow() to properly set stream_was_cancelled flag

**Problem:**
When a run is cancelled via /cancel endpoint, `stream_was_cancelled` remained
False because `RunCancelledException` was raised in the consumer code (wrapper),
which closes the generator from outside. This causes Python to skip the
generator's except blocks and jump directly to finally with the wrong flag value.

**Solution:**
1. Shared `asyncio.Event` registry for cross-layer cancellation signaling
2. `cancellation_aware_stream_wrapper` sets the event when cancellation detected
3. Wrapper uses `.athrow()` to inject exception INTO generator (not consumer-side raise)
4. All streaming interfaces check event in `finally` block to set flag correctly
5. `streaming_service.py` handles `RunCancelledException` gracefully, yields [DONE]

**Changes:**
- streaming_response.py: Event registry + .athrow() injection + graceful handling
- openai_streaming_interface.py: 3 classes check event in finally
- gemini_streaming_interface.py: Check event in finally
- anthropic_*.py: Catch RunCancelledException
- simple_llm_stream_adapter.py: Create & pass event to interfaces
- streaming_service.py: Handle RunCancelledException, yield [DONE], skip double-update
- routers/v1/{conversations,runs}.py: Pass event to wrapper
- integration_test_human_in_the_loop.py: New test for approval + cancellation

**Tests:**
- test_tool_call with cancellation (OpenAI models) 
- test_approve_with_cancellation (approval flow + concurrent cancel) 

**Known cosmetic warnings (pre-existing):**
- "Run already in terminal state" - agent loop tries to update after /cancel
- "Stream ended without terminal event" - background streaming timing race

👾 Generated with [Letta Code](https://letta.com)

Co-authored-by: Letta <noreply@letta.com>
2026-01-29 12:44:04 -08:00
Kian Jones
82e5d70807 fix: prevent empty reasoning messages in streaming interfaces (#7207)
* fix: prevent empty reasoning messages in streaming interfaces

Prevents empty "Thinking..." indicators from appearing in clients by
filtering out reasoning messages with no content at the source.

Changes:
- Gemini: Don't emit ReasoningMessage when only thought_signature exists
- Gemini: Only emit reasoning content if text is non-empty
- Anthropic: Don't emit ReasoningMessage for BetaSignatureDelta
- Anthropic: Only emit reasoning content if thinking text is non-empty

This fixes the issue where providers send signature metadata before
actual thinking content, causing empty reasoning blocks to appear
in the UI after responses complete.

Affects: Gemini reasoning, Anthropic extended thinking

👾 Generated with [Letta Code](https://letta.com)

Co-Authored-By: Letta <noreply@letta.com>

* fix: handle Anthropic thinking signature correctly

- Only include 'signature' in Anthropic message payload if it is not None (fixes BadRequestError).
- Capture and attach 'signature' to ReasoningMessage in streaming interface.

* fix(anthropic): attach signature to last reasoning message in stream

---------

Co-authored-by: Letta <noreply@letta.com>
2026-01-12 10:57:19 -08:00
Charles Packer
1f7165afc4 fix: patch counting of tokens for anthropic (#6458)
* fix: patch counting of tokens for anthropic

* fix: patch ui to be simpler

* fix: patch undercounting bug in anthropic when caching is on
2025-12-15 12:02:19 -08:00
Charles Packer
4af6465226 feat(core+web): store raw usage data on streams (and visualize properly in ADE) (#6452)
* feat(core): store raw usage data on streams

* fix(web): various fixes to deal w/ hardcoding against openai
2025-12-15 12:02:19 -08:00
Charles Packer
131891e05f feat: add tracking of advanced usage data (eg caching) [LET-6372] (#6449)
* feat: init refactor

* feat: add helper code

* fix: missing file + test

* fix: just state/publish api
2025-12-15 12:02:19 -08:00
Matthew Zhou
6f57ae829a fix: Reduce string growth for anthropic (#5974)
Reduce string growth for anthropic
2025-11-13 15:36:55 -08:00
cthomas
6b37ef2cb7 fix: special otid handling for approval request (#5726) 2025-10-24 15:14:31 -07:00
cthomas
4823416af9 feat: default unpack assistant message content [LET-5404] (#5707)
feat: default unpack assistant message content
2025-10-24 15:14:20 -07:00
cthomas
73dcc0d4b7 feat: latest hitl + parallel tool call changes (#5565) 2025-10-24 15:12:49 -07:00
Matthew Zhou
643ec8fe2f fix: Double write tool call deltas [LET-5545] (#5461)
* Double write tool call deltas

* Fix
2025-10-24 15:12:11 -07:00
Matthew Zhou
7511b0f4fe feat: Write anthropic streaming interface that supports parallel tool calling [LET-5355] (#5295)
Write anthropic streaming interface that supports parallel tool calling
2025-10-09 15:25:21 -07:00