* add self compaction method with proper caching (pass in tools, don't refresh sys prompt beforehand) + sliding fallback
* updated prompts for self compaction
* add tests for self, self_sliding_window modes and w/o refresh messages before compaction
* add cache logging to summarization
* better handling to prevent agent from continuing convo on self modes
* if mode changes via summarize endpoint, will use default prompt for the new mode
---------
Co-authored-by: Amy Guan <amy@letta.com>
Adds a diagnostic log at the streaming chokepoint in LettaAgentV3.stream()
to detect when any LettaMessage chunk is yielded without an id or otid field.
This helps trace the root cause of client-side id/otid inconsistencies.
* auto fixes
* auto fix pt2 and transitive deps and undefined var checking locals()
* manual fixes (ignored or letta-code fixed)
* fix circular import
* remove all ignores, add FastAPI rules and Ruff rules
* add ty and precommit
* ruff stuff
* ty check fixes
* ty check fixes pt 2
* error on invalid
* fix(core): stabilize system prompt refresh and expand git-memory coverage
Only rebuild system prompts on explicit refresh paths so normal turns preserve prefix-cache stability, including git/custom prompt layouts. Add integration coverage for memory filesystem tree structure and recompile/reset system-message updates via message-id retrieval.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix(core): recompile system prompt around compaction and stabilize source tests
Force system prompt refresh before/after compaction in LettaAgentV3 so repaired system+memory state is used and persisted across subsequent turns. Update source-system prompt tests to explicitly recompile before raw preview assertions instead of assuming automatic rebuild timing.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* Add log probabilities support for RL training
This enables Letta server to request and return log probabilities from
OpenAI-compatible providers (including SGLang) for use in RL training.
Changes:
- LLMConfig: Add return_logprobs and top_logprobs fields
- OpenAIClient: Set logprobs in ChatCompletionRequest when enabled
- LettaLLMAdapter: Add logprobs field and extract from response
- LettaResponse: Add logprobs field to return log probs to client
- LettaRequest: Add return_logprobs/top_logprobs for per-request override
- LettaAgentV3: Store and pass logprobs through to response
- agents.py: Handle request-level logprobs override
Usage:
response = client.agents.messages.create(
agent_id=agent_id,
messages=[...],
return_logprobs=True,
top_logprobs=5,
)
print(response.logprobs) # Per-token log probabilities
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* Add multi-turn token tracking for RL training via SGLang native endpoint
- Add TurnTokenData schema to track token IDs and logprobs per turn
- Add return_token_ids flag to LettaRequest and LLMConfig
- Create SGLangNativeClient for /generate endpoint (returns output_ids)
- Create SGLangNativeAdapter that uses native endpoint
- Modify LettaAgentV3 to accumulate turns across LLM calls
- Include turns in LettaResponse when return_token_ids=True
* Fix: Add SGLang native adapter to step() method, not just stream()
* Fix: Handle Pydantic Message objects in SGLang native adapter
* Fix: Remove api_key reference from LLMConfig (not present)
* Fix: Add missing 'created' field to ChatCompletionResponse
* Add full tool support to SGLang native adapter
- Format tools into prompt in Qwen-style format
- Parse tool calls from <tool_call> tags in response
- Format tool results as <tool_response> in user messages
- Set finish_reason to 'tool_calls' when tools are called
* Use tokenizer.apply_chat_template for proper tool formatting
- Add tokenizer caching in SGLang native adapter
- Use apply_chat_template when tokenizer available
- Fall back to manual formatting if not
- Convert Letta messages to OpenAI format for tokenizer
* Fix: Use func_response instead of tool_return for ToolReturn content
* Fix: Get output_token_logprobs from meta_info in SGLang response
* Fix: Allow None in output_token_logprobs (SGLang format includes null)
* chore: remove unrelated files from logprobs branch
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add missing call_type param to adapter constructors in letta_agent_v3
The SGLang refactor dropped call_type=LLMCallType.agent_step when extracting
adapter creation into conditional blocks. Restores it for all 3 spots (SGLang
in step, SimpleLLM in step, SGLang in stream).
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* just stage-api && just publish-api
* fix: update expected LLMConfig fields in schema test for logprobs support
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: remove rllm provider references
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* just stage-api && just publish-api
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-65-206.ec2.internal>
Co-authored-by: Letta <noreply@letta.com>
* refactor: extract compact logic to shared function
Extract the compaction logic from LettaAgentV3.compact() into a
standalone compact_messages() function that can be shared between
the agent and temporal workflows.
Changes:
- Create apps/core/letta/services/summarizer/compact.py with:
- compact_messages(): Core compaction logic
- build_summarizer_llm_config(): LLM config builder for summarization
- CompactResult: Dataclass for compaction results
- Update LettaAgentV3.compact() to use compact_messages()
- Update temporal summarize_conversation_history activity to use
compact_messages() instead of the old Summarizer class
- Add use_summary_role parameter to SummarizeParams
This ensures consistent summarization behavior across different
execution paths and prevents drift as we improve the implementation.
* chore: clean up verbose comments
* fix: correct CompactionSettings import path
* fix: correct count_tokens import from summarizer_sliding_window
* fix: update test patch path for count_tokens_with_tools
After extracting compact logic to compact.py, the test was patching
the old location. Update the patch path to the new module location.
* fix: update test to use build_summarizer_llm_config from compact.py
The function was moved from LettaAgentV3._build_summarizer_llm_config
to compact.py as a standalone function.
* fix: add early check for system prompt size in compact_messages
Check if the system prompt alone exceeds the context window before
attempting summarization. The system prompt cannot be compacted,
so fail fast with SystemPromptTokenExceededError.
* fix: properly propagate SystemPromptTokenExceededError from compact
The exception handler in _step() was not setting the correct stop_reason
for SystemPromptTokenExceededError, which caused the finally block to
return early and swallow the exception.
Add special handling to set stop_reason to context_window_overflow_in_system_prompt
when SystemPromptTokenExceededError is caught.
* revert: remove redundant SystemPromptTokenExceededError handling
The special handling in the outer exception handler is redundant because
stop_reason is already set in the inner handler at line 943. The actual
fix for the test was the early check in compact_messages(), not this
redundant handling.
* fix: correctly re-raise SystemPromptTokenExceededError
The inner exception handler was using 'raise e' which re-raised the outer
ContextWindowExceededError instead of the current SystemPromptTokenExceededError.
Changed to 'raise' to correctly re-raise the current exception. This bug
was pre-existing but masked because _check_for_system_prompt_overflow was
only called as a fallback. The new early check in compact_messages() exposed it.
* revert: remove early check and restore raise e to match main behavior
* fix: set should_continue=False and correctly re-raise exception
- Add should_continue=False in SystemPromptTokenExceededError handler (matching main's _check_for_system_prompt_overflow behavior)
- Fix raise e -> raise to correctly propagate SystemPromptTokenExceededError
Note: test_large_system_prompt_summarization still fails locally but passes on main.
Need to investigate why exception isn't propagating correctly on refactored branch.
* fix: add SystemPromptTokenExceededError handler for post-step compaction
The post-step compaction (line 1066) was missing a SystemPromptTokenExceededError
exception handler. When compact_messages() raised this error, it would be caught
by the outer exception handler which would:
1. Set stop_reason to "error" instead of "context_window_overflow_in_system_prompt"
2. Not set should_continue = False
3. Get swallowed by the finally block (line 1126) which returns early
This caused test_large_system_prompt_summarization to fail because the exception
never propagated to the test.
The fix adds the same exception handler pattern used in the retry compaction flow
(line 941-946), ensuring proper state is set before re-raising.
This issue only affected the refactored code because on main, _check_for_system_prompt_overflow()
was an instance method that set should_continue/stop_reason BEFORE raising. In the refactor,
compact_messages() is a standalone function that cannot set instance state, so the caller
must handle the exception and set the state.
**Error:**
```
TypeError: LettaAgentV2.__init__() got an unexpected keyword argument 'conversation_id'
```
**Trace:** https://letta.grafana.net/goto/afbk4da3fuxhcf?orgId=stacks-1189126
**Problem:**
The `POST /v1/conversations/{conversation_id}/compact` endpoint was failing
because `LettaAgentV3` inherits from `LettaAgentV2` without overriding
`__init__`, so passing `conversation_id` to the constructor failed.
**Fix:**
1. Add `__init__` to `LettaAgentV3` that accepts optional `conversation_id`
2. Remove redundant `conversation_id` param from `_checkpoint_messages` -
use `self.conversation_id` consistently instead
3. Clean up internal callers that were passing `conversation_id=self.conversation_id`
Backward compatible - existing code creating `LettaAgentV3(agent_state, actor)`
still works since `conversation_id` defaults to `None`.
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
**Problem:**
Runs failed with error:
```
Argument step_id does not match type <class 'str'>; is None of type <class 'NoneType'>
```
This happened when processing approval responses where the original
approval request message had `step_id=None`.
**Root Cause:**
Line 672 in `_step()` directly used `approval_request.step_id`:
```python
step_id = approval_request.step_id # Can be None!
step_metrics = await self.step_manager.get_step_metrics_async(step_id=step_id, ...)
```
`Message.step_id` is `Optional[str]` (default None), but `get_step_metrics_async`
has `step_id: str` with `@enforce_types` validation.
Old approval messages or edge cases could have `step_id=None`, causing
the enforce_types decorator to reject the call.
**Fix:**
Check if `step_id is None` and generate a new step_id + initialize step
checkpoint if needed, instead of assuming step_id always exists.
**Note:**
Similar issue exists in letta_agent_v2.py and temporal agents, but v2
is deprecated.
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
fix: load default provider config when summarizer uses different provider
**Problem:**
Summarization failed when agent used one provider (e.g., Google AI) but
summarizer config specified a different provider (e.g., Anthropic):
```python
# Agent LLM config
model_endpoint_type='google_ai', handle='gemini-something/gemini-2.5-pro',
context_window=100000
# Summarizer config
model='anthropic/claude-haiku-4-5-20251001'
# Bug: Resulting summarizer_llm_config mixed Google + Anthropic settings
model='claude-haiku-4-5-20251001', model_endpoint_type='google_ai', # ❌ Wrong endpoint!
context_window=100000 # ❌ Google's context window, not Anthropic's default!
```
This sent Claude requests to Google AI endpoints with incorrect parameters.
**Root Cause:**
`_build_summarizer_llm_config()` always copied the agent's LLM config as base,
then patched model/provider fields. But this kept all provider-specific settings
(endpoint, context_window, etc.) from the wrong provider.
**Fix:**
1. Parse provider_name from summarizer handle
2. Check if it matches agent's model_endpoint_type (or provider_name for custom)
3. **If YES** → Use agent config as base, override model/handle (same provider)
4. **If NO** → Load default config via `provider_manager.get_llm_config_from_handle()` (new provider)
**Example Flow:**
```python
# Agent: google_ai/gemini-2.5-pro
# Summarizer: anthropic/claude-haiku
provider_name = "anthropic" # Parsed from handle
provider_matches = ("anthropic" == "google_ai") # False ❌
# Different provider → load default Anthropic config
base = await provider_manager.get_llm_config_from_handle(
handle="anthropic/claude-haiku",
actor=self.actor
)
# Returns: model_endpoint_type='anthropic', endpoint='https://api.anthropic.com', etc. ✅
```
**Result:**
- Summarizer with different provider gets correct default config
- No more mixing Google endpoints with Anthropic models
- Same-provider summarizers still inherit agent settings efficiently
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
**Problem:**
Error logs showed empty detail fields when exceptions had no message:
```
Error during step processing:
Run run-xxx stopped with unknown error: , error_data: {...'detail': ''}
```
This made debugging production issues difficult as the actual error type
was hidden.
**Root Cause:**
Python exceptions created with no arguments (e.g., `Exception()` or caught
and re-raised in certain ways) have `str(e) == ""`:
```python
e = Exception()
str(e) # Returns ""
repr(e) # Returns "Exception()"
```
When exceptions with empty string representations were caught, all logging
and error messages showed blank details.
**Fix:**
Use `str(e) or repr(e)` fallback pattern in 3 places:
1. `letta_agent_v3.py` stream() exception handler (line 406)
2. `letta_agent_v3.py` step() exception handler (line 928)
3. `streaming_service.py` generic exception handler (line 469)
**Result:**
- Error logs now show `Exception()` or similar instead of empty string
- Helps identify exception types even when message is missing
- Better production debugging without changing exception handling logic
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* feat: add agent_id, run_id, step_id to summarization provider traces
Summarization LLM calls were missing telemetry context (agent_id,
agent_tags, run_id, step_id), making it impossible to attribute
summarization costs to specific agents or trace them back to the
step that triggered compaction.
Changes:
- Add step_id param to simple_summary() and set_telemetry_context()
- Add agent_id, agent_tags, run_id, step_id to summarize_all() and
summarize_via_sliding_window()
- Update Summarizer class to accept and pass telemetry context
- Update LettaAgentV3.compact() to pass full telemetry context
- Update LettaAgentV2.summarize_conversation_history() with run_id/step_id
- Update LettaAgent (v1) streaming methods with run_id param
- Add run_id/step_id to SummarizeParams for Temporal activities
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: update test mock to accept new summarization params
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: centralize telemetry logging at LLM client level
Moves telemetry logging from individual adapters to LLMClientBase:
- Add TelemetryStreamWrapper for streaming telemetry on stream close
- Add request_async_with_telemetry() for non-streaming requests
- Add stream_async_with_telemetry() for streaming requests
- Add set_telemetry_context() to configure agent_id, run_id, step_id
Updates adapters and agents to use new pattern:
- LettaLLMAdapter now accepts agent_id/run_id in constructor
- Adapters call set_telemetry_context() before LLM requests
- Removes duplicate telemetry logging from adapters
- Enriches traces with agent_id, run_id, call_type metadata
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: accumulate streaming response content for telemetry
TelemetryStreamWrapper now extracts actual response data from chunks:
- Content text (concatenated from deltas)
- Tool calls (id, name, arguments)
- Model name, finish reason, usage stats
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: move streaming telemetry to caller (option 3)
- Remove TelemetryStreamWrapper class
- Add log_provider_trace_async() helper to LLMClientBase
- stream_async_with_telemetry() now just returns raw stream
- Callers log telemetry after processing with rich interface data
Updated callers:
- summarizer.py: logs content + usage after stream processing
- letta_agent.py: logs tool_call, reasoning, model, usage
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: pass agent_id and run_id to parent adapter class
LettaLLMStreamAdapter was not passing agent_id/run_id to parent,
causing "unexpected keyword argument" errors.
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: add conversation_id parameter to context endpoint [LET-6989]
Add optional conversation_id query parameter to retrieve_agent_context_window.
When provided, the endpoint uses messages from the specific conversation
instead of the agent's default message_ids.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: regenerate SDK after context endpoint update [LET-6989]
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* feat: add isolated blocks support for conversations
Allows conversations to have their own copies of specific memory blocks (e.g., todo_list) that override agent defaults, enabling conversation-specific state isolation.
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* undo
* update apis
* test
* cleanup
* fix tests
* simplify
* move override logic
* patch
---------
Co-authored-by: Letta <noreply@letta.com>
When streaming, the LLM adapter needs to know which tools require
approval so it can emit ApprovalRequestMessage instead of ToolCallMessage.
Client-side tools were being passed to the agent but not included in
the requires_approval_tools list passed to the streaming interface.
This caused the streaming interface to emit tool_call_message for
client tools, but the stop_reason was still requires_approval,
resulting in empty approvals arrays on the client side.
🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* feat: allow client-side tools to be specified in request
Add `client_tools` field to LettaRequest to allow passing tool schemas
at message creation time without requiring server-side registration.
When the agent calls a client-side tool, execution pauses with
stop_reason=requires_approval for the client to provide tool returns.
- Add ClientToolSchema class for request-level tool schemas
- Merge client tools with agent tools in _get_valid_tools()
- Treat client-side tool calls as requiring approval
- Add integration tests for client-side tools flow
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* test: add comprehensive end-to-end test for client-side tools
Update integration test to verify the complete flow:
- Agent calls client-side tool and pauses
- Client provides tool return with secret code
- Agent processes and responds
- User asks about the code, agent recalls it
- Validate full conversation history makes sense
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* update apis
* fix: client-side tools schema format and test assertions
- Use flat schema format for client tools (matching t.json_schema)
- Support both object and dict access for client tools
- Fix stop_reason assertions to access .stop_reason attribute
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: simplify client_tools access pattern
ClientToolSchema objects always have .name attribute
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add client_tools parameter to LettaAgentV2 for API compatibility
V2 agent doesn't use client_tools but needs the parameter
to match the base class signature.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* revert: remove client_tools from LettaRequestConfig
Client-side tools don't work with background jobs since
there's no client present to provide tool returns.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add client_tools parameter to SleeptimeMultiAgent classes
Add client_tools to step() and stream() methods in:
- SleeptimeMultiAgentV3
- SleeptimeMultiAgentV4
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: regenerate API specs for client_tools support
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* fix: validate parallel tool calls with tool rules at create/update time
Move validation from runtime to agent create/update time for better UX.
Add client-side enforcement to truncate parallel tool calls when disabled
(handles providers like Gemini that ignore the setting).
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* update apis
* undo
---------
Co-authored-by: Letta <noreply@letta.com>
* initial commit
* Add database migration for compaction_settings field
This migration adds the compaction_settings column to the agents table
to support customized summarization configuration for each agent.
🐾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix
* rename
* update apis
* fix tests
* update web test
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Kian Jones <kian@letta.com>
