mlc-ai /mlc-llm

Universal LLM Deployment Engine with ML Compilation

★ 23,415⑂ 2,209👁 262⑃ 1,906● 1,661▤ 1
PythonApache-2.0Created 2023-04-29llm.mlc.ai/
More from this author
Download: v0.1.dev0v0.1.dev0 ↓02023-04-29
-2
Daily
+114
Weekly
+166
Monthly
23,415
stars
2,209
forks
262
watchers
1,906
prs
1,661
issues
1
releases

README

MLC LLM






Universal LLM Deployment Engine with ML Compilation

Get Started | Documentation | Blog



 About

MLC LLM is a machine learning compiler and high-performance deployment engine for large language models.  The mission of this project is to enable everyone to develop, optimize, and deploy AI models natively on everyone's platforms. 



  
    
       
      AMD GPU
      NVIDIA GPU
      Apple GPU
      Intel GPU
    
  
  
    
      Linux / Win
      ✅ Vulkan, ROCm
      ✅ Vulkan, CUDA
      N/A
      ✅ Vulkan
    
    
      macOS
      ✅ Metal (dGPU)
      N/A
      ✅ Metal
      ✅ Metal (iGPU)
    
    
      Web Browser
      ✅ WebGPU and WASM 
    
    
      iOS / iPadOS
      ✅ Metal on Apple A-series GPU
    
    
      Android
      ✅ OpenCL on Adreno GPU
      ✅ OpenCL on Mali GPU
    
  



MLC LLM compiles and runs code on MLCEngine -- a unified high-performance LLM inference engine across the above platforms. MLCEngine provides OpenAI-compatible API available through REST server, python, javascript, iOS, Android, all backed by the same engine and compiler that we keep improving with the community.

 Get Started

Please visit our documentation to get started with MLC LLM.
- Installation
- Quick start
- Introduction

 Citation

Please consider citing our project if you find it useful:

bibtex
@software{mlc-llm,
    author = {{MLC team}},
    title = {{MLC-LLM}},
    url = {https://github.com/mlc-ai/mlc-llm},
    year = {2023-2025}
}


The underlying techniques of MLC LLM include:


  References (Click to expand)

  bibtex
  @inproceedings{tensorir,
      author = {Feng, Siyuan and Hou, Bohan and Jin, Hongyi and Lin, Wuwei and Shao, Junru and Lai, Ruihang and Ye, Zihao and Zheng, Lianmin and Yu, Cody Hao and Yu, Yong and Chen, Tianqi},
      title = {TensorIR: An Abstraction for Automatic Tensorized Program Optimization},
      year = {2023},
      isbn = {9781450399166},
      publisher = {Association for Computing Machinery},
      address = {New York, NY, USA},
      url = {https://doi.org/10.1145/3575693.3576933},
      doi = {10.1145/3575693.3576933},
      booktitle = {Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},
      pages = {804–817},
      numpages = {14},
      keywords = {Tensor Computation, Machine Learning Compiler, Deep Neural Network},
      location = {Vancouver, BC, Canada},
      series = {ASPLOS 2023}
  }

  @inproceedings{metaschedule,
      author = {Shao, Junru and Zhou, Xiyou and Feng, Siyuan and Hou, Bohan and Lai, Ruihang and Jin, Hongyi and Lin, Wuwei and Masuda, Masahiro and Yu, Cody Hao and Chen, Tianqi},
      booktitle = {Advances in Neural Information Processing Systems},
      editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
      pages = {35783--35796},
      publisher = {Curran Associates, Inc.},
      title = {Tensor Program Optimization with Probabilistic 

Top issues

Latest release

v0.1.dev0 v0.1.dev0 Pre-release 2023-04-29

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON