alibaba /MNN
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
★ 16,534⑂ 2,534👁 237⑃ 1,362● 3,636▤ 64
+7
Daily
16,534
stars
2,534
forks
237
watchers
1,362
prs
3,636
issues
64
releases
README
!MNN --- News 🔥 - [2026/11/04] We released MNN-Pocket-0.3B, a 300M-parameter mobile-app-native language model for Chinese/English instruction following. The MNN team co-designed its architecture, training methods, and inference engine around smartphone apps' task requirements and resource budgets. It supports 32K context; the INT4 (W4A16) QAT MNN build occupies only 173 MB and achieves 350 tok/s decode throughput in tests on iPhone 18 Pro. Released under Apache 2.0. ModelScope | Hugging Face | Technical report History News - [2026/07/22] MNN 3.6.1 is released with a new 🚀 Hexagon backend, enabling accelerated model inference on Qualcomm Hexagon DSPs. Learn more. - [2026/03/05] Support Qwen3.5 Series. - [2026/02/13] MNN-Sana-Edit-V2 is now available at apps, offering cartoon-style photo editing based on Sana. - [2025/10/16] Support Qwen3-VL Series. - [2025/06/11] New App MNN TaoAvatar released, you can talk with 3DAvatar offline with LLM, ASR, TTS, A2BS and NNR models all run local on your device!! MNN TaoAvatar - [2025/05/12] android app support qwen2.5 omni 3b and 7b MNN Chat App. - [2025/04/30] android app support qwen3 and dark mode MNN Chat App. - [2025/02/18] iOS multimodal LLM App is released MNN LLM iOS. - [2025/02/11] android app support for deepseek r1 1.5b. - [2025/01/23] We released our full multimodal LLM Android App:MNN-LLM-Android. including text-to-text, image-to-text, audio-to-text, and text-to-image generation. Intro MNN is a highly efficient and lightweight deep learning framework. It supports inference and training of deep learning models and has industry-leading performance for inference and training on-device. At present, MNN has been integrated into more than 30 apps of Alibaba Inc, such as Taobao, Tmall, Youku, DingTalk, Xianyu, etc., covering more than 70 usage scenarios such as live broadcast, short video capture, search recommendation, product searching by image, interactive marketing, equity distribution, security risk control. In addition, MNN is also used on embedded devices, such as IoT. MNN-LLM is a large language model runtime solution developed based on the MNN engine. The mission of this project is to deploy LLM models locally on everyone's platforms(Mobile Phone/PC/IOT). It supports popular large language models such as Qianwen, Baichuan, Zhipu, LLAMA, and others. MNN-LLM User guide MNN-Diffusion is a stable diffusion model runtime solution developed based on the MNN engine. The mission of this project is to deploy stable diffusion models locally on everyone's platforms. MNN-Diffusion User guide !architecture Inside Alibaba, MNN works as the basic module of the compute container in the Walle System, the first end-to-end, general-purpose, and large-scale production system for device-cloud collaborative machine learning, which has been published in the top system conference OSDI’22. The key design principles of MNN an
Top issues
#4802Qwen2.5-Omni-3B/7B 在最新代码上opencl上推理崩溃14 comments · 2026-08-24#4883qwen3.5的mnn模型如何进行gpu推理12 comments · 2026-09-17#4641[bug]Gemma-4-E2B-it-MNN在识别图片时崩溃8 comments · 2026-07-18#4815Android 内存泄露6 comments · 2026-08-27#4811Android API inference crash: SIGABRT in MNN::DeferBufferAllocator::apply() / MNN::Session::resize()5 comments · 2026-08-26#5109模型生成基准测试字符串失败2 comments · 2026-11-07#5062[Bug] Qwen3-TTS 0.6B: generation never emits EOS and decodes to digital silence for some texts (int4/int8/int3/fp16)2 comments · 2026-10-29#5146MNN Chat 加载/推理 DeepSeek-R1(7B 与 1.5B)时稳定闪退,两规格均复现1 comments · 2026-11-15
Discussions
#50683.6.20 comments · 2026-10-29#46623.6.15 comments · 2026-07-23#45463.6.02 comments · 2026-06-16#4355MNN Supports Eagle30 comments · 2026-04-07#4354MNN Model Support: Qwen3.50 comments · 2026-04-07#4353How to Design High-Performance On-Device Tokenizers? MNN Refactoring Practice and Reflections0 comments · 2026-04-07#4352MNN Complex Task Skill Decomposition Experiment0 comments · 2026-04-07#4351TurboQuant: A Paper Crashes Storage Stock Prices? MNN Runs It on Mobile0 comments · 2026-04-07
Latest release
3.6.2 3.6.2 2026-10-29MNN 3.6.2 版本发布说明 发布日期: 2026年10月 📦 安装: pip install mnn==3.6.2 | PyPI 📌 版本概述 MNN 3.6.2 聚焦 端侧异构推理性能与投机解码加速。本版本以 Metal、Hexagon 和 DFlash 为重点,持续提升 Qwen3 / Qwen3.5 在 CPU、GPU 与 NPU 上的推理效率。 典型性能:Qwen3.5-2B 在 iPad Pro M5 的 GPU 上通过 Metal 后端运行,Prefill 达到 3542.91 tok/s;启用 DFlash 后,Decode 从 111.89 提升至 233.98 tok/s,达到常规解码的 2.09 倍。 🚀 版本亮点 Metal / Apple GPU Neural Accelerators 优化:通过 Metal Tensor API 利用 Apple GPU 内的神经网络加速单元,提升 Qwen3 / Qwen3.5 Dense 推理性能 Hexagon 推理优化:提升 NPU 上的 Prefill 效率,减少 Decode 阶段的计算与调度开销 DFlash 投机解码加速:加速草稿生成与验证,复用目标模型输出层权重,提高生成吞吐并减少内存占用 CPU / OpenCL 性能优化