intel /ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

★ 8,841⑂ 1,432👁 8⑃ 10,285● 3,024▤ 22
PythonApache-2.0Created 2016-08-29
More from this author
Download: IPEX-LLM release 2.2.0IPEX-LLM-2.2.0 ↓2.2.02025-04-07
-1
Daily
8,841
stars
1,432
forks
8
watchers
10,285
prs
3,024
issues
22
releases

README

THIS PROJECT IS ARCHIVED  
Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates.  
Patches to this project are no longer accepted by Intel.  
This project has been identified as having known security issues. 

  💫 Intel® LLM Library for PyTorch 

   | 中文 


IPEX-LLM is an LLM acceleration library for Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max), NPU and CPU [^1].
 [!NOTE]
 - IPEX-LLM provides seamless integration with llama.cpp, Ollama, vLLM, HuggingFace transformers, LangChain, LlamaIndex, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, ModeScope, etc. 
 - 70+ models have been optimized/verified on ipex-llm (e.g., Llama, Phi, Mistral, Mixtral, DeepSeek, Qwen, ChatGLM, MiniCPM, Qwen-VL, MiniCPM-V and more), with state-of-art LLM optimizations, XPU acceleration and low-bit (FP8/FP6/FP4/INT4) support; see the complete list here.

 Latest Update 🔥 
- [2025/05] You can now run DeepSeek V3/R1 671B and Qwen3MoE 235B models with just 1 or 2 Intel Arc GPU (such as A770 or B580) using FlashMoE in ipex-llm.
- [2025/04] We released ipex-llm 2.2.0, which includes Ollama Portable Zip and llama.cpp Portable Zip.
   ⚠️ Warning (for llama.cpp Portable Zip)  
   mmap-based model loading in llama.cpp may leak data via side-channels in multi-tenant or shared-host environments.  
   To disable mmap, add:  
   bash
   --no-mmap
   
- [2025/04] We added support of PyTorch 2.6 for Intel GPU.
- [2025/03] We added support for Gemma3 model in the latest llama.cpp Portable Zip.
- [2025/03] We can now run DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon using the latest llama.cpp Portable Zip.
- [2025/02] We added support of llama.cpp Portable Zip for Intel GPU (both Windows and Linux) and NPU (Windows only).
- [2025/02] We added support of Ollama Portable Zip to directly run Ollama on Intel GPU for both Windows and Linux (without the need of manual installations).
- [2025/02] We added support for running vLLM 0.6.6 on Intel Arc GPUs.
- [2025/01] We added the guide for running ipex-llm on Intel Arc B580 GPU.
- [2025/01] We added support for running Ollama 0.5.4 on Intel GPU.
- [2024/12] We added both Python and C++ support for Intel Core Ultra NPU (including 100H, 200V, 200K and 200H series).

More updates


- [2024/11] We added support for running vLLM 0.6.2 on Intel Arc GPUs.
- [2024/07] We added support for running Microsoft's GraphRAG using local LLM on Intel GPU; see the quickstart guide here.
- [2024/07] We added extensive support for Large Multimodal Models, including StableDiffusion, Phi-3-Vision, Qwen-VL, and more.
- [2024/07] We added FP6 support on Intel GPU. 
- [2024/06] We added experimental NPU support for Intel Core Ultra processors; see the examples here. 
- [2024/06] We added extensive support of pipeline parallel inference, which makes it easy to run large-sized 

Top issues

Discussions

Latest release

IPEX-LLM release 2.2.0 v2.2.0 2025-04-07
Highlights
Note: IPEX-LLM v2.2.0 has been updated to include functional and security updates. Users should update to the latest version.

Please go to https://github.com/ipex-llm/ipex-llm/releases/tag/v2.2.0 for the downloads.

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON