cthomas
c05f3cec0b
fix: wrap markitdown PDF processing in asyncio.to_thread ( #8614 )
...
MarkItDown.convert() does blocking file I/O and CPU-intensive PDF parsing.
This was blocking the event loop during file uploads.
Now wraps the entire markitdown pipeline (tempfile write, convert, cleanup)
in asyncio.to_thread() to run in thread pool.
🐾 Generated with [Letta Code](https://letta.com )
Co-authored-by: Letta <noreply@letta.com >
2026-01-19 15:54:37 -08:00
cthomas
a775bbfdf2
feat: offload llama index init logic ( #8387 )
2026-01-12 10:57:48 -08:00
cthomas
6f810d95d8
feat: add semaphore to limit embeddings creation ( #6261 )
2025-11-24 19:10:11 -08:00
Kian Jones
15bc124ddc
fix: don't fail on empty pdf ( #6043 )
...
don't fail on empty pdf
2025-11-13 15:36:56 -08:00
Matthew Zhou
3a551f4bb7
feat: Add turbopuffer embedder by default [LET-4253] ( #4476 )
...
* Adapt to turbopuffer embedder
* Make turbopuffer search more efficient over all source ids
* Combine turbopuffer and pinecone hybrid
* Fix test sources
2025-09-08 18:46:41 -07:00
Matthew Zhou
9358bec393
fix: Remove possibility of empty strings when updating ( #4149 )
2025-08-24 17:28:11 -07:00
Matthew Zhou
b16f5ffc99
feat: Polishing open files tools ( #3575 )
...
Co-authored-by: Charles Packer <packercharles@gmail.com >
Co-authored-by: Shubham Naik <shub@letta.com >
Co-authored-by: Shubham Naik <shub@memgpt.ai >
Co-authored-by: cthomas <caren@letta.com >
Co-authored-by: jnjpng <jin@letta.com >
Co-authored-by: Jin Peng <jinjpeng@Jins-MacBook-Pro.local >
Co-authored-by: Cameron Pfiffer <cameron@pfiffer.org >
Co-authored-by: Kian Jones <11655409+kianjones9@users.noreply.github.com >
Co-authored-by: Kian Jones <kian@Kians-MacBook-Pro.local >
2025-07-29 15:46:51 -07:00
Matthew Zhou
357e30fc55
fix: Adjust immediate complete for pinecone file state machine ( #3546 )
2025-07-24 14:29:42 -07:00
Matthew Zhou
be7f4be4d8
fix: Fix state transitions for file processing ( #3541 )
2025-07-24 13:48:08 -07:00
Matthew Zhou
1a4aeb5f9f
feat: Add transition state validation ( #3509 )
2025-07-23 14:08:41 -07:00
Matthew Zhou
cc431b2e97
fix: Fix pinecone embedder ( #3502 )
2025-07-22 23:39:50 -07:00
Matthew Zhou
d446fcbd27
chore: Finish renaming agent serialization manager ( #3497 )
2025-07-22 18:01:00 -07:00
Andy Li
9ca1664ed8
feat: support for providers
2025-07-22 16:09:50 -07:00
Sarah Wooders
5e9231095a
feat: support markitdown instead of mistral ( #3451 )
...
Co-authored-by: Matt Zhou <mattzh1314@gmail.com >
2025-07-22 09:58:18 -07:00
Kevin Lin
043eaca869
feat: pass in embedding chunk size ( #3323 )
2025-07-21 15:11:28 -07:00
Matthew Zhou
b0138ecad6
feat: Support exporting/importing sources and files for agent file ( #3414 )
2025-07-18 16:20:18 -07:00
Matthew Zhou
723d44d816
feat: Fix empty splitlines bug ( #3317 )
2025-07-14 10:50:15 -07:00
Matthew Zhou
c94b227a32
feat: Improve performance on files related data models ( #3285 )
2025-07-10 18:00:35 -07:00
Andy Li
d362b96bb8
chore: sentry log grouping and file upload stats
2025-07-09 10:29:23 -07:00
Matthew Zhou
4c8e9af4bd
feat: Add error on out of range for open_files ( #3214 )
2025-07-07 16:40:25 -07:00
Matthew Zhou
9551f1257a
feat: Further split structured documents ( #3199 )
2025-07-07 13:22:50 -07:00
Andy Li
fa23566f24
chore: proper logging for sentry ( #3195 )
2025-07-07 12:37:39 -07:00
Matthew Zhou
e5cb964d90
feat: Make chunking more robust and file type specific ( #3188 )
2025-07-07 11:05:01 -07:00
Sarah Wooders
961d14f975
chore: reduce logger.error to logger.debug for file upload ( #3186 )
2025-07-07 09:49:31 -07:00
Matthew Zhou
9605d1f79c
feat: Add pinecone for cloud embedding ( #3160 )
2025-07-03 22:37:55 -07:00
Matthew Zhou
802361493f
fix: Change to only derive json_schema for custom tools if missing ( #3129 )
2025-07-01 17:13:31 -07:00
Matthew Zhou
e0c688dc32
feat: Add telemetry for file uploads ( #3128 )
2025-07-01 16:19:58 -07:00
Matthew Zhou
b39a4adcbe
feat: Finish batch size param on embedding config ( #3115 )
2025-07-01 13:41:49 -07:00
Matthew Zhou
62c14fcfe1
feat: Add csv support for files ( #3077 )
2025-06-27 15:04:24 -07:00
Matthew Zhou
edb20fa8d9
feat: Rework and simplify files tooling ( #3058 )
2025-06-27 13:23:44 -07:00
Matthew Zhou
ccd521f27b
feat: Adjust upload files to return FileMetadata instead of job ( #3031 )
2025-06-25 15:26:35 -07:00
Matthew Zhou
fd1bff065a
feat: Attach file blocks when creating an agent with source ( #3027 )
2025-06-25 14:08:00 -07:00
Matthew Zhou
82cf7aac38
feat: Use default letta embedder if OpenAI key is not provided ( #2899 )
2025-06-18 15:23:43 -07:00
Matthew Zhou
698d99a66e
feat: Ungate file upload for simple MIME types even without Mistral API key ( #2898 )
2025-06-18 15:11:30 -07:00
Matthew Zhou
a1a203c943
feat: Parallel web search tool ( #2890 )
2025-06-18 14:07:51 -07:00
Matthew Zhou
b4215d71d9
feat: Add content aware line chunking ( #2707 )
2025-06-09 13:03:25 -07:00
Matthew Zhou
2568039ab9
feat: Add code file support to file uploads ( #2702 )
2025-06-07 23:45:30 -07:00
cthomas
c300e58db9
chore: create file manager ( #2693 )
2025-06-07 16:14:41 -07:00
Matthew Zhou
ddb9186947
feat: Implement grep tool ( #2694 )
2025-06-06 17:36:49 -07:00
Matthew Zhou
318a7c769b
feat: Search files returns citations of the filenames that were searched ( #2689 )
2025-06-06 15:34:03 -07:00
Matthew Zhou
09ec9ffeea
feat: Add file metadata processing status ( #2665 )
2025-06-06 11:11:39 -07:00
Matthew Zhou
ef52a852a3
feat: Add line metadata and warnings to file blocks ( #2663 )
2025-06-05 17:02:28 -07:00
Matthew Zhou
f8e2185f37
feat: Add support for simple text files ( #2630 )
2025-06-04 14:36:25 -07:00
Matthew Zhou
fb0dbbdf0d
feat: Store entire parsed file content in separate table ( #2618 )
2025-06-03 18:51:45 -07:00
Matthew Zhou
5886f3819a
feat: Adapt lifecycle management of file <-> agent association ( #2591 )
2025-06-02 17:34:12 -07:00
Matthew Zhou
9c44b9ae6a
feat: Add mistral for cloud document parsing ( #2562 )
2025-05-30 21:06:28 -07:00
