sgl-project /mini-sglang

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

★ 5,582⑂ 987👁 19⑃ 152● 33▤ 0
PythonMITCreated 2025-09-01
More from this author
+2
Daily
5,582
stars
987
forks
19
watchers
152
prs
33
issues
0
releases

README

Mini-SGLang

A lightweight yet high-performance inference framework for Large Language Models.

---

Mini-SGLang is a compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems. With a compact codebase of ~5,000 lines of Python, it serves as both a capable inference engine and a transparent reference for researchers and developers.

 ✨ Key Features

- High Performance: Achieves state-of-the-art throughput and latency with advanced optimizations.
- Lightweight & Readable: A clean, modular, and fully type-annotated codebase that is easy to understand and modify.
- Advanced Optimizations:
  - Radix Cache: Reuses KV cache for shared prefixes across requests.
  - Chunked Prefill: Reduces peak memory usage for long-context serving.
  - Overlap Scheduling: Hides CPU scheduling overhead with GPU computation.
  - Tensor Parallelism: Scales inference across multiple GPUs.
  - Optimized Kernels: Integrates FlashAttention and FlashInfer for maximum efficiency.
  - ...

 🚀 Quick Start

 ⚠️ Platform Support: Mini-SGLang currently supports Linux only (x86_64 and aarch64). Windows and macOS are not supported due to dependencies on Linux-specific CUDA kernels (sgl-kernel, flashinfer). We recommend using WSL2 on Windows or Docker for cross-platform compatibility.

 1. Environment Setup

We recommend using uv for a fast and reliable installation (note that uv does not conflict with conda).

bash
 Create a virtual environment (Python 3.10+ recommended)
uv venv --python=3.12
source .venv/bin/activate


Prerequisites: Mini-SGLang relies on CUDA kernels that are JIT-compiled. Ensure you have the NVIDIA CUDA Toolkit installed and that its version matches your driver's version. You can check your driver's CUDA capability with nvidia-smi.

 2. Installation

Install Mini-SGLang directly from the source:

bash
git clone https://github.com/sgl-project/mini-sglang.git
cd mini-sglang && uv venv --python=3.12 && source .venv/bin/activate
uv pip install -e .



💡 Installing on Windows (WSL2)

Since Mini-SGLang requires Linux-specific dependencies, Windows users should use WSL2:

1. Install WSL2 (if not already installed):
   powershell
    In PowerShell (as Administrator)
   wsl --install
   

2. Install CUDA on WSL2:
   - Follow NVIDIA's WSL2 CUDA guide
   - Ensure your Windows GPU drivers support WSL2

3. Install Mini-SGLang in WSL2:
   bash
    Inside WSL2 terminal
   git clone https://github.com/sgl-project/mini-sglang.git
   cd mini-sglang && uv venv --python=3.12 && source .venv/bin/activate
   uv pip install -e .
   

4. Access from Windows: The server will be accessible at http://localhost:8000 from Windows browsers and applications.




🐳 Running with Docker

Prerequisites:
- Docker
- NVIDIA Container Toolkit

1. Build the Docker image:
   bash
   docker build -t minisgl .
   

2. Run the server:
   bash
   docker run --gpus all -p 1919:1919 \
       minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0
   

3. Run in interactive she

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON