Ani Tunturi 93337ce680 fix: pass images from tool returns to vision models ...

OpenAI Chat Completions requires tool message content to be a string,
so images in tool returns were silently replaced with [Image omitted].

Now: text stays in the tool return, images get injected as a user
message right after. The model actually sees what the tool saw.

to_openai_dict also cleaned up — image handling lives in
to_openai_dicts_from_list where it can inject the extra message.

2026-03-21 12:41:37 -04:00

..

adapters

Add billing context to LLM telemetry traces (#9745)

2026-03-03 18:34:13 -08:00

agents

Add billing context to LLM telemetry traces (#9745)

2026-03-03 18:34:13 -08:00

cli

chore: enable F821, F401, W293 (#9503)

2026-02-24 10:55:08 -08:00

client

chore: remove sync db (#4873)

2025-10-07 17:50:45 -07:00

data_sources

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

exceptions

feat: global exception middleware (#6017)

2025-11-13 15:36:55 -08:00

functions

fix(memory): standardize tool parameter names (#9552)

2026-02-24 10:55:24 -08:00

groups

fix: add billing_context to SleeptimeMultiAgent V3/V4 stream methods (#9753)

2026-03-03 18:34:15 -08:00

helpers

fix: accept base64 images in tool returns, not just letta-hosted

2026-03-21 12:41:20 -04:00

humans

merge this (#4759)

2025-09-17 15:47:40 -07:00

interfaces

fix(core): raise LLMEmptyResponseError for empty Anthropic responses (#9624)

2026-03-03 18:34:01 -08:00

jobs

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

llm_api

Add billing context to LLM telemetry traces (#9745)

2026-03-03 18:34:13 -08:00

local_llm

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

model_specs

feat: add gpt-5.3-chat-latest model support (#9746)

2026-03-03 18:34:15 -08:00

monitoring

feat: dump thread state on event loop hang (#8388)

2026-01-12 10:57:48 -08:00

openai_backcompat

merge this (#4759)

2025-09-17 15:47:40 -07:00

orm

fix: lazy load conversations [LET-7682] (#9629)

2026-03-03 18:34:01 -08:00

otel

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

personas

merge this (#4759)

2025-09-17 15:47:40 -07:00

plugins

chore: enable F821, F401, W293 (#9503)

2026-02-24 10:55:08 -08:00

prompts

Add modes self and self_sliding_window for prompt caching (#9372)

2026-02-24 10:55:26 -08:00

schemas

fix: pass images from tool returns to vision models

2026-03-21 12:41:37 -04:00

serialize_schemas

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

server

feat: new agent id query param for default convo (#9756)

2026-03-03 18:34:15 -08:00

services

Add billing context to LLM telemetry traces (#9745)

2026-03-03 18:34:13 -08:00

templates

merge this (#4759)

2025-09-17 15:47:40 -07:00

types

chore: enable F821, F401, W293 (#9503)

2026-02-24 10:55:08 -08:00

__init__.py

chore: bump version

2026-03-03 18:34:51 -08:00

agent.py

chore: remove sync db (#4873)

2025-10-07 17:50:45 -07:00

config_file.py

feat: change default context window from 32000 to 128000 (#9673)

2026-03-03 18:34:01 -08:00

config.py

merge this (#4759)

2025-09-17 15:47:40 -07:00

constants.py

Fix: Change Z.ai context window to account for max_token subtraction (#9710)

2026-03-03 18:34:02 -08:00

database_utils.py

chore: sync 0.12.0 version (#3023)

2025-10-08 16:10:51 -07:00

errors.py

fix(core): raise LLMEmptyResponseError for empty Anthropic responses (#9624)

2026-03-03 18:34:01 -08:00

interface.py

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

log_context.py

feat(logs): Enrich logs with context-aware primtive types (#5949)

2025-11-13 15:36:55 -08:00

log.py

feat: Ship traces to datadog and add trace correlation (#6311)

2025-11-24 19:10:26 -08:00

main.py

chore: enable F821, F401, W293 (#9503)

2026-02-24 10:55:08 -08:00

pytest.ini

merge this (#4759)

2025-09-17 15:47:40 -07:00

settings.py

feat(core): increase Gemini timeout to 10 minutes (#9714)

2026-03-03 18:34:02 -08:00

streaming_interface.py

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

streaming_utils.py

merge this (#4759)

2025-09-17 15:47:40 -07:00

system.py

Add modes self and self_sliding_window for prompt caching (#9372)

2026-02-24 10:55:26 -08:00

test_gemini.py

feat(gemini): add 3.1 pro preview support (#9553)

2026-02-24 10:55:11 -08:00

utils.py

chore: add ty + pre-commit hook and repeal even more ruff rules (#9504)

2026-02-24 10:55:11 -08:00

validators.py

feat: new agent id query param for default convo (#9756)

2026-03-03 18:34:15 -08:00