f63419c78b
* updated local llm documentation * updated cli flags to be consistent with documentation * added preset documentation * update test to use new arg * update test to use new arg
1.5 KiB
1.5 KiB
MemGPT + LM Studio
!!! warning "Important LM Studio settings"
Make sure that "context length" is set (inside LM Studio's "Model Configuration" panel) to the max context length of the model you're using (e.g. 8000 for Mistral 7B variants).
If you see "Prompt Formatting" (inside LM Studio's "Server Options" panel), turn it **OFF**. Leaving it **ON** will break MemGPT.
- Download LM Studio and the model you want to test with
- Go to the "local inference server" tab, load the model and configure your settings (make sure to set the context length to something reasonable like 8k!)
- Click "Start server"
- Copy the IP address + port that your server is running on (in the example screenshot, the address is
http://localhost:1234)
In your terminal where you're running MemGPT, run memgpt configure to set the default backend for MemGPT to point at LM Studio:
# if you are running LM Studio locally, the default IP address + port will be http://localhost:1234
? Select LLM inference provider: local
? Select LLM backend (select 'openai' if you have an OpenAI compatible proxy): lmstudio
? Enter default endpoint: http://localhost:1234
...
If you have an existing agent that you want to move to the web UI backend, add extra flags to memgpt run:
memgpt run --agent your_agent --model-endpoint-type lmstudio --model-endpoint http://localhost:1234