Fimeg/letta-server
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix(core): handle Anthropic overloaded errors and Unicode encoding issues (#9305)

Browse Source 
* fix: handle Anthropic overloaded_error in streaming interfaces

* fix: handle Unicode surrogates in OpenAI requests

Sanitize Unicode surrogate pairs before sending requests to OpenAI API.
Surrogate pairs (U+D800-U+DFFF) are UTF-16 encoding artifacts that cause
UnicodeEncodeError when encoding to UTF-8.

Fixes Datadog error: 'utf-8' codec can't encode character '\ud83c' in
position 326605: surrogates not allowed

* fix: handle UnicodeEncodeError from lone Unicode surrogates in OpenAI requests

Improved sanitize_unicode_surrogates() to explicitly filter out lone
surrogate characters (U+D800 to U+DFFF) which are invalid in UTF-8.

Previous implementation used errors='ignore' which could still fail in
edge cases. New approach directly checks Unicode code points and removes
any surrogates before data reaches httpx encoding.

Also added sanitization to stream_async_responses() method which was
missing it.

Fixes: 'utf-8' codec can't encode character '\ud83c' in position X:
surrogates not allowed
This commit is contained in:
Kian Jones
2026-02-05 16:00:36 -08:00
committed by Caren Thomas
parent 93249b96f5
commit 6f746c5225
12 changed files with 98 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tests/integration_test_summarizer.py
View File

@@ -1157,7 +1157,7 @@ async def test_sliding_window_cutoff_index_does_not_exceed_message_count(server:
summary, remaining_messages = await summarize_via_sliding_window(
actor=actor,
llm_config=llm_config,
agent_llm_config=llm_config, # case where agent and summarizer have same config
agent_llm_config=llm_config, # case where agent and summarizer have same config
summarizer_config=summarizer_config,
in_context_messages=messages,
)