FMInference /FlexLLMGen
Running large language models on a single GPU for throughput-oriented scenarios.
★ 9,336⑂ 588👁 7⑃ 59● 83▤ 0
+0
Daily
9,336
stars
588
forks
7
watchers
59
prs
83
issues
0
releases
README
FlexLLMGen: High-throughput Generative Inference of Large Language Models with a Single GPU [paper] FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes. Motivation In recent years, large language models (LLMs) have shown great performance across a wide range of tasks. Increasingly, LLMs have been applied not only to interactive applications (such as chat), but also to many "back-of-house" tasks. These tasks include benchmarking, information extraction, data wrangling, and form processing. One key characteristic of these applications is that they are throughput-oriented: they require running LLM inferences over millions of tokens in batches, e.g., all the private documents in a company's corpus, or all the tasks in the HELM benchmark. These workloads are less sensitive to latency - the user starts up a job and lets it run overnight - but increasing throughput is critical for reducing costs. Throughput is a measure of tokens processed per second over the job's entire runtime (which can be hours). Throughput-oriented workloads provide opportunities to trade off latency for higher throughput, which makes it easier to take advantage of low-cost commodity GPUs. The goal of FlexLLMGen is to create a high-throughput system to enable new and exciting applications of foundation models to throughput-oriented tasks on low-cost hardware, such as a single commodity GPU instead of expensive systems. Check out the examples of what you can run on a single commodity GPU with FlexLLMGen, including benchmarking and data wrangling. ❌ Limitation. As an offloading-based system running on weak GPUs, FlexLLMGen also has its limitations. FlexLLMGen can be significantly slower than the case when you have enough powerful GPUs to hold the whole model, especially for small-batch cases. FlexLLMGen is mostly optimized for throughput-oriented batch processing settings (e.g., classifying or extracting information from many documents in batches), on single GPUs. ---------- This project was made possible thanks to a collaboration with ---------- Content - Installation - Usage and Examples - Get Started with a Single GPU - Run HELM Benchmark with FlexLLMGen - Run Data Wrangling Tasks with FlexLLMGen - Scaling to Distributed GPUs - API Example - Frequently Asked Questions - Performance Results - How It Works - Roadmap Installation Requirements: - PyTorch = 1.12 (Help) Method 1: With pip pip install flexllmgen Method 2: From source git clone https://github.com/FMInference/FlexLLMGen.git cd FlexLLMGen pip install -e . Usage and Examples Get Started with a Single GPU OPT-1.3B To get started, you can try a small model like OPT-1.3B first. It fits into a single GPU so n
Top issues
#60Is LLaMa supported?28 comments · 2023-02-24#40Add Erebus and GALACTICA support11 comments · 2023-02-22#75 AttributeError: 'OptLM' object has no attribute 'weight_home'9 comments · 2023-02-28#55Update documentation to explicitly describe compatability/performance with early Pascal cards6 comments · 2023-02-23#118how to install from source4 comments · 2023-05-28#46Mac4 comments · 2023-02-23#20Doesn't seem to obey --path argument, instead try to download to .cache again4 comments · 2023-02-21#137 File "setup.py" not found.3 comments · 2024-05-03