Change the log level for expected MCP tool execution failures (ToolError,
McpError) from warning to debug in fastmcp_client.py to match base_client.py.
These errors occur when an LLM calls an MCP tool with missing/invalid
arguments - they are expected user-facing issues from external MCP servers,
not system errors that should trigger production alerts.
Fixes#8845🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: datadog-official[bot] <datadog-official[bot]@users.noreply.github.com>
Co-authored-by: Kian Jones <11655409+kianjones9@users.noreply.github.com>
MCP tool errors (ToolError, McpError) are expected user-facing errors
from external MCP servers (e.g., "No connected account found"). These
were propagating through @trace_method decorator and being recorded
as errors in Datadog APM.
Changes:
- Add try/except to catch expected MCP errors in ExternalMCPToolExecutor
- Return ToolExecutionResult with status="error" instead of re-raising
- Log expected errors at INFO level instead of letting them trace as ERROR
- Remove stray 'pass' statement that was a no-op
Fixes#8685🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: datadog-official[bot] <datadog-official[bot]@users.noreply.github.com>
Co-authored-by: Kian Jones <11655409+kianjones9@users.noreply.github.com>
When Ollama providers are synced to DB via sync_base_providers(), the
default_prompt_formatter field is lost because ProviderCreate doesn't
include it. When loading from DB and calling cast_to_subtype(), Pydantic
validation fails because the field is required.
This was a latent bug exposed when provider models persistence was
re-enabled in 0.16.2. The field was always required but never persisted.
Adding a default value ("chatml") fixes the issue. The field isn't
actually used in the current implementation - the model_wrapper line
is commented out in list_llm_models_async() since Ollama now uses
OpenAI-compatible endpoints.
Fixes: letta-ai/letta-code#587👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* feat: centralize telemetry logging at LLM client level
Moves telemetry logging from individual adapters to LLMClientBase:
- Add TelemetryStreamWrapper for streaming telemetry on stream close
- Add request_async_with_telemetry() for non-streaming requests
- Add stream_async_with_telemetry() for streaming requests
- Add set_telemetry_context() to configure agent_id, run_id, step_id
Updates adapters and agents to use new pattern:
- LettaLLMAdapter now accepts agent_id/run_id in constructor
- Adapters call set_telemetry_context() before LLM requests
- Removes duplicate telemetry logging from adapters
- Enriches traces with agent_id, run_id, call_type metadata
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: accumulate streaming response content for telemetry
TelemetryStreamWrapper now extracts actual response data from chunks:
- Content text (concatenated from deltas)
- Tool calls (id, name, arguments)
- Model name, finish reason, usage stats
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* refactor: move streaming telemetry to caller (option 3)
- Remove TelemetryStreamWrapper class
- Add log_provider_trace_async() helper to LLMClientBase
- stream_async_with_telemetry() now just returns raw stream
- Callers log telemetry after processing with rich interface data
Updated callers:
- summarizer.py: logs content + usage after stream processing
- letta_agent.py: logs tool_call, reasoning, model, usage
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: pass agent_id and run_id to parent adapter class
LettaLLMStreamAdapter was not passing agent_id/run_id to parent,
causing "unexpected keyword argument" errors.
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: add provider trace backend abstraction for multi-backend telemetry
Introduces a pluggable backend system for provider traces:
- Base class with async/sync create and read interfaces
- PostgreSQL backend (existing behavior)
- ClickHouse backend (via OTEL instrumentation)
- Socket backend (writes to Unix socket for crouton sidecar)
- Factory for instantiating backends from config
Refactors TelemetryManager to use backends with support for:
- Multi-backend writes (concurrent via asyncio.gather)
- Primary backend for reads (first in config list)
- Graceful error handling per backend
Config: LETTA_TELEMETRY_PROVIDER_TRACE_BACKEND (comma-separated)
Example: "postgres,socket" for dual-write to Postgres and crouton
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* feat: add protocol version to socket backend records
Adds PROTOCOL_VERSION constant to socket backend:
- Included in every telemetry record sent to crouton
- Must match ProtocolVersion in apps/crouton/main.go
- Enables crouton to detect and reject incompatible messages
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: remove organization_id from ProviderTraceCreate calls
The organization_id is now handled via the actor parameter in the
telemetry manager, not through ProviderTraceCreate schema. This fixes
validation errors after changing ProviderTraceCreate to inherit from
BaseProviderTrace which forbids extra fields.
🐙 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* consolidate provider trace
* add clickhouse-connect to fix bug on main lmao
* auto generated sdk changes, and deployment details, and clikchouse prefix bug and added fields to runs trace return api
* auto generated sdk changes, and deployment details, and clikchouse prefix bug and added fields to runs trace return api
* consolidate provider trace
* consolidate provider trace bug fix
---------
Co-authored-by: Letta <noreply@letta.com>
* feat: add TypeScript tool support for E2B sandbox execution
This change implements TypeScript tool support using the same E2B path as Python tools:
- Add TypeScript execution script generator (typescript_generator.py)
- Modify E2B sandbox to detect TypeScript tools and use language='ts'
- Add npm package installation for TypeScript tool dependencies
- Add validation requiring json_schema for TypeScript tools
- Add comprehensive integration tests for TypeScript tools
TypeScript tools:
- Require explicit json_schema (no docstring parsing)
- Use JSON serialization instead of pickle for results
- Support async functions with top-level await
- Support npm package dependencies via npm_requirements field
Closes#8793
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
* fix: disable AgentState for TypeScript tools & add letta-client injection
Based on Sarah's feedback:
1. AgentState is a legacy Python-only feature, disabled for TS tools
2. Added @letta-ai/letta-client npm package injection for TypeScript
(similar to letta_client for Python)
Changes:
- base.py: Explicitly set inject_agent_state=False for TypeScript tools
- typescript_generator.py: Inject LettaClient initialization code
- e2b_sandbox.py: Auto-install @letta-ai/letta-client for TS tools
- Added tests verifying both behaviors
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Sarah Wooders <sarahwooders@users.noreply.github.com>
Co-Authored-By: Letta <noreply@letta.com>
* Update core-integration-tests.yml
* fix: convert TypeScript test fixtures to async
The OrganizationManager and UserManager no longer have sync methods,
only async variants. Updated all fixtures to use:
- create_organization_async
- create_actor_async
- create_or_update_tool_async
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: skip Python AST parsing for TypeScript tools in sandbox base
The _init_async method was calling parse_function_arguments (which uses
Python's ast.parse) before checking if the tool was TypeScript, causing
SyntaxError when running TypeScript tools.
Moved the is_typescript_tool() check to happen first, skipping Python
AST parsing entirely for TypeScript tools.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* letta_agent_id
* skip ast parsing for s
* add tool execution test
---------
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: Kian Jones <kian@letta.com>
The import was `from datetime import datetime` but the code used
`datetime.timezone.utc` which is incorrect - timezone is a sibling
class in the datetime module, not an attribute of the datetime class.
This caused the error: "type object 'datetime.datetime' has no
attribute 'timezone'"
PR #8682 added code that references settings.use_clickhouse_for_provider_traces
but never added the field to Settings, causing AttributeError in prod.
🤖 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
feat(core): add run_id filter to internal runs endpoint
Adds the ability to filter runs by a specific run ID in the
internal runs list endpoint.
🐛 Generated with [Letta Code](https://letta.com)
Co-authored-by: Letta <noreply@letta.com>
* chore: deprecate identities/groups APIs and remove from SDK
- Mark all /v1/identities/* endpoints as deprecated
- Mark all /v1/groups/* endpoints as deprecated
- Remove identities, groups, and batches resources from stainless.yml
- Batch API remains active but hidden from SDK
👾 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* chore: update autogenerated SDK files
* chore: regenerate SDK and OpenAPI spec
Run `just stage-api` and `just publish-api` to sync generated files.
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
* chore: remove schedule API from stainless SDK
Remove schedule subresource from stainless.yml to hide scheduled messages
endpoints from the SDK generation.
👾 Generated with [Letta Code](https://letta.com)
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
---------
Co-authored-by: Letta <noreply@letta.com>
Co-authored-by: letta-code <248085862+letta-code@users.noreply.github.com>
Co-authored-by: Sarah Wooders <sarahwooders@users.noreply.github.com>
* feat: add /v1/runs/{run_id}/trace endpoint for OTEL traces
- Add new endpoint to retrieve filtered OTEL spans for a run
- Filter to only return UI-relevant spans (agent_step, tool executions, root span, TTFT)
- Skip Postgres writes when ClickHouse is enabled for provider traces
- Add USE_CLICKHOUSE_FOR_PROVIDER_TRACES env var to helm/justfile
- Move typecheck CI to self-hosted runners
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* fix: add missing clickhouse_provider_traces.py
The telemetry_manager.py imports ClickhouseProviderTraceReader from
this module, but the file was not included when splitting the PR.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
* autogen
* fix: add trace.retrieve to stainless.yml for SDK generation
Adds the runs.trace.retrieve method mapping so Stainless generates
the useRunsServiceRetrieveTraceForRun hook.
🤖 Generated with [Letta Code](https://letta.com)
Co-Authored-By: Letta <noreply@letta.com>
---------
Co-authored-by: Letta <noreply@letta.com>
