Fimeg/letta-server
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

tweak

Browse Source
This commit is contained in:
Kian Jones
2025-06-20 16:24:29 -07:00
parent 8e370c101d
commit ebe6aa68ce
5 changed files with 311 additions and 9317 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.github/scripts/model-sweep/generate_model_sweep_markdown.py vendored
View File

@@ -406,7 +406,7 @@ def process_model_sweep_report(input_file, output_file, config_file=None, debug=
model_anchor = model_info['name'].replace('/', '_').replace(':', '_').replace('-', '_').lower()
# Build row with left-aligned first column, centered others
row_parts = [f"[`{model_info['name']}`](#{provider_name.lower().replace(' ', '_')}_{model_anchor}_details)".ljust(column_widths['model'] + 20)] # Add extra space for link syntax
row_parts = [f"`{model_info['name']}`".ljust(column_widths['model'])]
for feature in feature_names:
row_parts.append(f"{model_info['feature_support'][feature]:^{column_widths[feature]}}")
row_parts.extend([
@@ -421,7 +421,7 @@ def process_model_sweep_report(input_file, output_file, config_file=None, debug=
for model_info in model_data:
model_anchor = model_info['name'].replace('/', '_').replace(':', '_').replace('-', '_').lower()
mdx_lines.append(f"#### {model_info['name']} {{#{provider_name.lower().replace(' ', '_')}_{model_anchor}_details}}")
mdx_lines.append(f"#### {model_info['name']}")
mdx_lines.append("")
# Add test details

618
.github/scripts/model-sweep/model_sweep_report.mdx vendored
View File

File diff suppressed because it is too large Load Diff

216
.github/scripts/model-sweep/test_markdown.mdx vendored
View File

@@ -1,216 +0,0 @@
---
title: Support Models
generated: 2025-06-16T17:37:43.167565
---
# Supported Models
## Overview
Letta routinely runs automated scans against available providers and models. These are the results of the latest scan.
Ran 2464 tests against 154 models across 7 providers on June 16th, 2025
## anthropic
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `claude-3-5-haiku-20241022` | ✅ | ✅ | ✅ | 200,000 | 2025-06-16 |
| `claude-3-5-sonnet-20241022` | ✅ | ✅ | ✅ | 200,000 | 2025-06-16 |
| `claude-3-7-sonnet-20250219` | ✅ | ✅ | ✅ | 200,000 | 2025-06-16 |
| `claude-sonnet-4-20250514` | ✅ | ✅ | ✅ | 200,000 | 2025-06-16 |
| `claude-opus-4-20250514` | ✅ | ✅ | ⚠️ | 200,000 | 2025-06-16 |
| `claude-3-5-sonnet-20240620` | ⚠️ | ❌ | ✅ | 200,000 | 2025-06-16 |
| `claude-3-haiku-20240307` | ⚠️ | ❌ | ✅ | 200,000 | 2025-06-16 |
| `claude-3-opus-20240229` | ⚠️ | ❌ | ✅ | 200,000 | 2025-06-16 |
| `claude-3-sonnet-20240229` | ❌ | ❌ | ❌ | 200,000 | 2025-06-16 |
---
## openai
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `gpt-4.1` | ✅ | ✅ | ✅ | 1,047,576 | 2025-06-16 |
| `gpt-4.1-2025-04-14` | ✅ | ✅ | ✅ | 1,047,576 | 2025-06-16 |
| `gpt-4.1-nano-2025-04-14` | ✅ | ✅ | ✅ | 1,047,576 | 2025-06-16 |
| `gpt-4o` | ✅ | ✅ | ✅ | 128,000 | 2025-06-16 |
| `gpt-4o-2024-05-13` | ✅ | ✅ | ✅ | 128,000 | 2025-06-16 |
| `gpt-4-turbo` | ✅ | ✅ | ⚠️ | 8,192 | 2025-06-16 |
| `gpt-4.1-mini` | ✅ | ✅ | ⚠️ | 1,047,576 | 2025-06-16 |
| `gpt-4.5-preview` | ✅ | ✅ | ⚠️ | 128,000 | 2025-06-16 |
| `gpt-4.5-preview-2025-02-27` | ✅ | ✅ | ⚠️ | 128,000 | 2025-06-16 |
| `gpt-4o-2024-08-06` | ✅ | ✅ | ⚠️ | 128,000 | 2025-06-16 |
| `gpt-4-0613` | ✅ | ✅ | ❌ | 8,192 | 2025-06-16 |
| `gpt-4-1106-preview` | ✅ | ✅ | ❌ | 128,000 | 2025-06-16 |
| `gpt-4-turbo-2024-04-09` | ✅ | ⚠️ | ✅ | 128,000 | 2025-06-16 |
| `gpt-4.1-mini-2025-04-14` | ⚠️ | ✅ | ✅ | 1,047,576 | 2025-06-16 |
| `gpt-4.1-nano` | ⚠️ | ✅ | ✅ | 1,047,576 | 2025-06-16 |
| `gpt-4o-2024-11-20` | ⚠️ | ✅ | ✅ | 8,192 | 2025-06-16 |
| `gpt-4-turbo-preview` | ✅ | ⚠️ | ❌ | 128,000 | 2025-06-16 |
| `gpt-4-0125-preview` | ⚠️ | ✅ | ❌ | 128,000 | 2025-06-16 |
| `gpt-4o-mini` | ⚠️ | ⚠️ | ⚠️ | 128,000 | 2025-06-16 |
| `gpt-4o-mini-2024-07-18` | ⚠️ | ⚠️ | ❌ | 128,000 | 2025-06-16 |
| `gpt-4` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o1` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o1-2024-12-17` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o3` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o3-2025-04-16` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o3-mini` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o3-mini-2025-01-31` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o3-pro` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `o3-pro-2025-06-10` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
---
## google_ai
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `gemini-1.5-pro` | ✅ | ✅ | ✅ | 2,000,000 | 2025-06-16 |
| `gemini-1.5-pro-002` | ✅ | ✅ | ✅ | 2,000,000 | 2025-06-16 |
| `gemini-1.5-pro-latest` | ✅ | ✅ | ✅ | 2,000,000 | 2025-06-16 |
| `gemini-2.5-flash-preview-04-17-thinking` | ✅ | ✅ | ✅ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-pro-preview-03-25` | ✅ | ✅ | ✅ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-pro-preview-05-06` | ✅ | ✅ | ✅ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-flash-preview-05-20` | ✅ | ⚠️ | ✅ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-thinking-exp` | ⚠️ | ✅ | ✅ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-thinking-exp-1219` | ⚠️ | ✅ | ✅ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-thinking-exp-01-21` | ⚠️ | ✅ | ⚠️ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-flash-preview-04-17` | ⚠️ | ✅ | ⚠️ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-pro-preview-06-05` | ⚠️ | ✅ | ⚠️ | 1,048,576 | 2025-06-16 |
| `gemini-1.0-pro-vision-latest` | ❌ | ❌ | ❌ | 12,288 | 2025-06-16 |
| `gemini-1.5-flash` | ❌ | ❌ | ❌ | 1,000,000 | 2025-06-16 |
| `gemini-1.5-flash-002` | ❌ | ❌ | ❌ | 1,000,000 | 2025-06-16 |
| `gemini-1.5-flash-8b` | ❌ | ❌ | ❌ | 1,000,000 | 2025-06-16 |
| `gemini-1.5-flash-8b-001` | ❌ | ❌ | ❌ | 1,000,000 | 2025-06-16 |
| `gemini-1.5-flash-8b-latest` | ❌ | ❌ | ❌ | 1,000,000 | 2025-06-16 |
| `gemini-1.5-flash-latest` | ❌ | ❌ | ❌ | 1,000,000 | 2025-06-16 |
| `gemini-2.0-flash` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-001` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-exp` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-exp-image-generation` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-lite` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-lite-001` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-lite-preview` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-lite-preview-02-05` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-flash-preview-image-generation` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `gemini-2.0-pro-exp` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.0-pro-exp-02-05` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-flash-preview-tts` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `gemini-2.5-pro-exp-03-25` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-2.5-pro-preview-tts` | ❌ | ❌ | ❌ | 65,536 | 2025-06-16 |
| `gemini-exp-1206` | ❌ | ❌ | ❌ | 1,048,576 | 2025-06-16 |
| `gemini-pro-vision` | ❌ | ❌ | ❌ | 12,288 | 2025-06-16 |
---
## letta
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `letta-free` | ⚠️ | ❌ | ❌ | 8,192 | 2025-06-16 |
---
## together
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `Qwen/Qwen2.5-72B-Instruct-Turbo` | ✅ | ✅ | ⚠️ | 131,072 | 2025-06-16 |
| `arcee-ai/virtuoso-large` | ⚠️ | ✅ | ✅ | 131,072 | 2025-06-16 |
| `Qwen/QwQ-32B` | ⚠️ | ✅ | ⚠️ | 131,072 | 2025-06-16 |
| `Qwen/Qwen2.5-7B-Instruct-Turbo` | ⚠️ | ✅ | ⚠️ | 32,768 | 2025-06-16 |
| `Qwen/Qwen2.5-Coder-32B-Instruct` | ⚠️ | ✅ | ⚠️ | 16,384 | 2025-06-16 |
| `arcee-ai/coder-large` | ⚠️ | ✅ | ⚠️ | 32,768 | 2025-06-16 |
| `arcee_ai/arcee-spotlight` | ⚠️ | ✅ | ⚠️ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-3.2-3B-Instruct-Turbo` | ⚠️ | ✅ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-3.3-70B-Instruct-Turbo` | ⚠️ | ✅ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-3.3-70B-Instruct-Turbo-Free` | ⚠️ | ✅ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo` | ⚠️ | ✅ | ❌ | 130,815 | 2025-06-16 |
| `meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo` | ⚠️ | ✅ | ❌ | 131,072 | 2025-06-16 |
| `nvidia/Llama-3.1-Nemotron-70B-Instruct-HF` | ⚠️ | ✅ | ❌ | 32,768 | 2025-06-16 |
| `arcee-ai/virtuoso-medium-v2` | ⚠️ | ⚠️ | ✅ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8` | ⚠️ | ❌ | ✅ | 1,048,576 | 2025-06-16 |
| `Qwen/Qwen3-235B-A22B-fp8-tput` | ⚠️ | ⚠️ | ❌ | 40,960 | 2025-06-16 |
| `deepseek-ai/DeepSeek-V3` | ⚠️ | ⚠️ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-4-Scout-17B-16E-Instruct` | ⚠️ | ⚠️ | ❌ | 1,048,576 | 2025-06-16 |
| `meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo` | ⚠️ | ⚠️ | ❌ | 131,072 | 2025-06-16 |
| `mistralai/Mixtral-8x7B-Instruct-v0.1` | ⚠️ | ⚠️ | ❌ | 32,768 | 2025-06-16 |
| `arcee-ai/caller` | ❌ | ⚠️ | ❌ | 32,768 | 2025-06-16 |
| `mistralai/Mistral-Small-24B-Instruct-2501` | ❌ | ⚠️ | ❌ | 32,768 | 2025-06-16 |
| `NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `Qwen/Qwen2-72B-Instruct` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `Qwen/Qwen2-VL-72B-Instruct` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `Qwen/Qwen2.5-VL-72B-Instruct` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `arcee-ai/arcee-blitz` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `arcee-ai/maestro-reasoning` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `deepseek-ai/DeepSeek-R1` | ❌ | ❌ | ❌ | 163,840 | 2025-06-16 |
| `deepseek-ai/DeepSeek-R1-Distill-Llama-70B` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `deepseek-ai/DeepSeek-R1-Distill-Llama-70B-free` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `deepseek-ai/DeepSeek-R1-Distill-Qwen-14B` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `deepseek-ai/DeepSeek-V3-p-dp` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `google/gemma-2-27b-it` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `lgai/exaone-3-5-32b-instruct` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `lgai/exaone-deep-32b` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `marin-community/marin-8b-instruct` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-3-70b-chat-hf` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/Llama-3-8b-chat-hf` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Llama-Vision-Free` | ❌ | ❌ | ❌ | 131,072 | 2025-06-16 |
| `meta-llama/Meta-Llama-3-70B-Instruct-Turbo` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/Meta-Llama-3-8B-Instruct-Lite` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `mistralai/Mistral-7B-Instruct-v0.1` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `mistralai/Mistral-7B-Instruct-v0.2` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `mistralai/Mistral-7B-Instruct-v0.3` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `perplexity-ai/r1-1776` | ❌ | ❌ | ❌ | 163,840 | 2025-06-16 |
| `scb10x/scb10x-llama3-1-typhoon2-70b-instruct` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `scb10x/scb10x-typhoon-2-1-gemma3-12b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `togethercomputer/MoA-1` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `togethercomputer/MoA-1-Turbo` | ❌ | ❌ | ❌ | 32,768 | 2025-06-16 |
| `togethercomputer/Refuel-Llm-V2` | ❌ | ❌ | ❌ | 16,384 | 2025-06-16 |
| `togethercomputer/Refuel-Llm-V2-Small` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
---
## deepseek
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `deepseek-chat` | ❌ | ❌ | ❌ | 64,000 | 2025-06-16 |
| `deepseek-reasoner` | ❌ | ❌ | ❌ | 64,000 | 2025-06-16 |
---
## groq
| Model | Basic | Token Streaming | Multimodal | Context Window | Last Scanned |
|---------------------------------------------------|:---:|:-------------:|:--------:|:------------:|:----------:|
| `allam-2-7b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `compound-beta` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `compound-beta-mini` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `deepseek-r1-distill-llama-70b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `distil-whisper-large-v3-en` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `gemma2-9b-it` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `llama-3.1-8b-instant` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `llama-3.3-70b-versatile` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `llama-guard-3-8b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `llama3-70b-8192` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `llama3-8b-8192` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/llama-4-maverick-17b-128e-instruct` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/llama-4-scout-17b-16e-instruct` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/llama-guard-4-12b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/llama-prompt-guard-2-22m` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `meta-llama/llama-prompt-guard-2-86m` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `mistral-saba-24b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `playai-tts` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `playai-tts-arabic` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `qwen-qwq-32b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `qwen/qwen3-32b` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `whisper-large-v3` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
| `whisper-large-v3-turbo` | ❌ | ❌ | ❌ | 8,192 | 2025-06-16 |
---

4395
.github/scripts/model-sweep/test_output.mdx vendored
View File

File diff suppressed because it is too large Load Diff

4395
.github/scripts/model-sweep/test_output_with_links.mdx vendored
View File

File diff suppressed because it is too large Load Diff