-1
Daily
5,692
stars
385
forks
47
watchers
1,617
prs
180
issues
27
releases
README
Serge - LLaMA made easy 🦙
!License
Serge is a chat interface crafted with llama.cpp for running LLM models. No API keys, entirely self-hosted!
- 🌐 SvelteKit frontend
- 💾 Redis for storing chat history & parameters
- ⚙️ FastAPI + LangChain for the API, wrapping calls to llama.cpp using the python bindings
🎥 Demo:
demo.webm
⚡️ Quick start
🐳 Docker:
bash
docker run -d \
--name serge \
-v weights:/usr/src/app/weights \
-v datadb:/data/db/ \
-p 8008:8008 \
ghcr.io/serge-chat/serge:latest
🐙 Docker Compose:
yaml
services:
serge:
image: ghcr.io/serge-chat/serge:latest
container_name: serge
restart: unless-stopped
ports:
- 8008:8008
volumes:
- weights:/usr/src/app/weights
- datadb:/data/db/
volumes:
weights:
datadb:
Then, just visit http://localhost:8008, You can find the API documentation at http://localhost:8008/api/docs
🌍 Environment Variables
The following Environment Variables are available:
| Variable Name | Description | Default Value |
|-----------------------|---------------------------------------------------------|--------------------------------------|
| SERGE_DATABASE_URL | Database connection string | sqlite:////data/db/sql_app.db |
| SERGE_JWT_SECRET | Key for auth token encryption. Use a random string | uF7FGN5uzfGdFiPzR |
| SERGE_SESSION_EXPIRY| Duration in minutes before a user must reauthenticate | 60 |
| NODE_ENV | Node.js running environment | production |
🖥️ Windows
Ensure you have Docker Desktop installed, WSL2 configured, and enough free RAM to run models.
⚠️ Memory Usage
LLaMA will crash if you don't have enough available memory for the model
💬 Support
Need help? Join our Discord
🧾 License
Nathan Sarrazin and Contributors. Serge is free and open-source software licensed under the MIT License and Apache-2.0.
🤝 Contributing
If you discover a bug or have a feature idea, feel free to open an issue or PR.
To run Serge in development mode:
bash
git clone https://github.com/serge-chat/serge.git
cd serge/
docker compose -f docker-compose.dev.yml up --build
The solution will accept a python debugger session on port 5678. Example launch.json for VSCode:
json
{
"version": "0.2.0",
"configurations": [
{
"name": "Remote Debug",
"type": "python",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5678
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}/api",
"remoteRoot": "/usr/src/app/api/"
}
],
"justMyCode": false
}
]
}Top issues
#43[Enhancement] Add the Ability to Use NVIDIA for Docker46 comments · 2023-03-24#1144🐛 [Bug]: system reachable via ICMP and via Port 8008 but screen "navy blue" with no text whatsoever14 comments · 2024-02-18#1182🐛 [Bug]: New install - response keeps repeating the last line8 comments · 2024-03-05#17Add the possibility to upload PDFs in the front part of the application to read it5 comments · 2023-03-22#1061🐛 [Bug]: response text generated by a model sometimes disappears after computer/browser is woken up from a 'sleep'4 comments · 2024-01-15#1114bug: Allow loading .gguf and .bin files3 comments · 2024-02-05#475🐛 [Bug]: Number with open parentheses does not show in the UI3 comments · 2023-06-27#223🚀 [Feature]: Add support for Prompt Templates3 comments · 2023-04-25
Latest release
0.9.0 - Updates to dependencies and UI fixes 0.9.0 2024-02-14🚀 New Features: - Add support for IPv6 (1055) - Update llama-cpp-python to v0.2.41 (1133) - Add support for sidebar menu (1075) 🧠 Models - Update OpenChat model version (1046) - Add support for Phi 2 (1030) - Add Mixtral 2.7, MedicineLLM, MistralLite, and Notux (1026) 🧹 Updates: - Updates to UI components, cleanup of unused dependencies (1136) - Update llama-cpp-python to v0.2.41 (1133) - Update NodeJS dependencies (1132) - Update llama-cpp-python from 0.2.38 to 0.2.39 (1119) - Update README.md (1113) - Update llama-cpp-python to v0.2.38 (1062) - Bump pytest from 7.4