Commit Graph

6 Commits

Author SHA1 Message Date
Kian Jones
273ca9ec44 feat(tests): add crouton telemetry tests (#9000)
* test: add comprehensive provider trace telemetry tests

Add two test files for provider trace telemetry:

1. test_provider_trace.py - Integration tests for:
   - Basic agent steps (streaming and non-streaming)
   - Tool calls
   - Telemetry context fields (agent_id, agent_tags, step_id, run_id)
   - Multi-step conversations
   - Request/response JSON content

2. test_provider_trace_summarization.py - Unit tests for:
   - simple_summary() telemetry context passing
   - summarize_all() telemetry pass-through
   - summarize_via_sliding_window() telemetry pass-through
   - Summarizer class runtime vs constructor telemetry
   - LLMClient.set_telemetry_context() method

🤖 Generated with [Letta Code](https://letta.com)

Co-Authored-By: Letta <noreply@letta.com>

* test: add telemetry tests for tool generation, adapters, and agent versions

Add comprehensive unit tests for provider trace telemetry:

- TestToolGenerationTelemetry: Verify /generate-tool endpoint sets
  call_type="tool_generation" and has no agent context
- TestLLMClientTelemetryContext: Verify LLMClient.set_telemetry_context
  accepts all telemetry fields
- TestAdapterTelemetryAttributes: Verify base adapter and subclasses
  (LettaLLMRequestAdapter, LettaLLMStreamAdapter) support telemetry attrs
- TestSummarizerTelemetry: Verify Summarizer stores and passes telemetry
- TestAgentAdapterInstantiation: Verify LettaAgentV2 creates Summarizer
  with correct agent_id

🤖 Generated with [Letta Code](https://letta.com)

Co-Authored-By: Letta <noreply@letta.com>

* ci: add provider trace telemetry tests to unit test workflow

Add the new provider trace test files to the CI matrix:
- test_provider_trace_backends.py
- test_provider_trace_summarization.py
- test_provider_trace_agents.py

🤖 Generated with [Letta Code](https://letta.com)

Co-Authored-By: Letta <noreply@letta.com>

* fix: update socket backend test to match new record structure

The socket backend record structure changed - step_id/run_id are now
at top level, and model/usage are nested in request/response objects.

🤖 Generated with [Letta Code](https://letta.com)

Co-Authored-By: Letta <noreply@letta.com>

* fix: add step_id to V1 agent telemetry context

Pass step_id to set_telemetry_context in both streaming and non-streaming
paths in LettaAgent (v1). The step_id is available via step_metrics.id
in the non-streaming path and passed explicitly in the streaming path.

🤖 Generated with [Letta Code](https://letta.com)

Co-Authored-By: Letta <noreply@letta.com>

---------

Co-authored-by: Letta <noreply@letta.com>
2026-01-29 12:43:53 -08:00
cthomas
baa72e43c3 test: update provider trace tests to use sdk client (#2866) 2025-09-12 19:12:31 -07:00
Kian Jones
45c4dbd5e8 chore(ci): Add uv support and use for unit tests (#4127)
* cherrypick just relevant commits?

* make work with poetry

* update poetry?

* regen?

* change tests and dev to dependency groups instead of optional extras

* Fix Poetry/UV compatibility issues

- Fix sqlite-vec dependency: Remove optional flag from Poetry section to match main deps
- Regenerate poetry.lock to sync with pyproject.toml changes
- Test both package managers successfully:
  - Poetry: `poetry install --with dev --with test -E postgres -E external-tools -E cloud-tool-sandbox`
  - UV: `uv sync --group dev --group test --extra postgres --extra external-tools --extra cloud-tool-sandbox`

Resolves Poetry lock sync errors and ensures sqlite-vec is available for tests.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* more robust pip install

* Fix fern SDK wheel installation in CI workflow

Replace unreliable command substitution with proper error handling:
- Check if directory exists before attempting to find wheels
- Store wheel file path in variable to avoid empty arguments
- Provide clear error messages when directory/wheels are missing
- Prevents "required arguments were not provided" error in uv pip install

Fixes: error: the following required arguments were not provided: <PACKAGE>

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* debugging

* trigger CI

* ls

* revert whl installation to -e

* programmatic HIT version insertion

* version templating properly

* set var properly

* labelling

* remove version insertion

* ?

* try using sed '2r /dev/stdin'

* version

* try again smh

* not trigger on poetry version

* only add once

* filter only for project not poetry

* hand re-construct the file

* save tail?

* fix docker command

* please please please

* rename test -> tests

* update poetry and rename group to -E

* move async into tests extra and regen lock files and add sqlite extra

* remove loading cached venv from cloud api integration

* add uv dependency to CI runners

* test removing the custom event loop

* regen poetry.lock and try to fix async tests

* wrap async pg exception and event loop tweak in plugins

* remove event loop from plugins test and remove caching from cloud-api-integration-test

* migrate all tests away from event loop for pytest-asyncio

* pin firecrawl

* pin e2b

* take claude's suggestion

* deeper down the claude rabbit hole

* increase timeout for httpbin.org

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-08-26 11:51:31 -07:00
Andy Li
f9bb757a98 feat: support for agent loop job cancelation (#2837) 2025-07-02 14:31:16 -07:00
cthomas
a332b34830 feat: make create_async route consistent with other message routes (#2877) 2025-06-19 13:51:51 -07:00
Andy Li
0a54b998a8 feat: track llm provider traces and tracking steps in async agent loop (#2219) 2025-05-19 15:50:56 -07:00