From 7194e98669b1bb4d7a4c103cdb37336ca80783a4 Mon Sep 17 00:00:00 2001 From: Charles Packer Date: Sat, 28 Oct 2023 18:26:42 -0700 Subject: [PATCH] Update README.md --- memgpt/local_llm/README.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/memgpt/local_llm/README.md b/memgpt/local_llm/README.md index 28d59669..4a571125 100644 --- a/memgpt/local_llm/README.md +++ b/memgpt/local_llm/README.md @@ -31,6 +31,13 @@ For the purposes of this example, we're going to serve (host) the LLMs using [oo 3. Navigate to the web app (if local, this is probably [`http://127.0.0.1:7860`](http://localhost:7860)), select the model you want to use, adjust your GPU and CPU memory settings, and click "load" 4. If the model was loaded successfully, you should be able to access it via the API (if local, this is probably on port `5000`) 5. Assuming steps 1-4 went correctly, the LLM is now properly hosted on a port you can point MemGPT to! +6. Copy the IP address + port that your server is running on + +```sh +# if you are running web UI locally, the default port will be 5000 +export OPENAI_API_BASE=http://127.0.0.1:5000 +export BACKEND_TYPE=lmstudio +``` WebUI exposes a lot of parameters that can dramatically change LLM outputs, to change these you can modify the [WebUI settings file](/memgpt/local_llm/webui/settings.py). @@ -38,6 +45,25 @@ WebUI exposes a lot of parameters that can dramatically change LLM outputs, to c +
+

🖥️ Serving your LLM from a web server (LM Studio example)

 + +![image](https://github.com/cpacker/MemGPT/assets/5475622/abc8ce2d-4130-4c51-8169-83e682db625d) + +1. Download [LM STudio](https://lmstudio.ai/) and the model you want to test with +2. Go to the "local inference server" tab, load the model, and click "Start server" +3. Copy the IP address + port that your server is running on (in the example screenshot, the address is `http://localhost:1234`) + +In your terminal where you're running MemGPT, run: + +```sh +# if you used a different port in LM Studio, change 1234 to the actual port +export OPENAI_API_BASE=http://localhost:1234 +export BACKEND_TYPE=lmstudio +``` + +
+

🦙 Running MemGPT with your own LLM

 @@ -46,7 +72,7 @@ Once you have an LLM web server set up, all you need to do to connect it to MemG - `OPENAI_API_BASE` - set this to the IP address of your LLM API - for example, if you're using web UI on a local machine, this will look like `http://127.0.0.1:5000` - `BACKEND_TYPE` - - set this to `webui` + - set this to `webui` or `lmstudio` - this controls how MemGPT packages the HTTP request to the webserver, see [this code](https://github.com/cpacker/MemGPT/blob/main/memgpt/local_llm/webui/api.py) - currently this is set up to work with web UI, but it might work with other backends / web servers too! - if you'd like to use a different web server and you need a different style of HTTP request, let us know on the discussion page (https://github.com/cpacker/MemGPT/discussions/67) and we'll try to add it ASAP