intel /ipex-llm
Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.
★ 8,841⑂ 1,432👁 8⑃ 10,285● 3,024▤ 22
More from this author
-1
Daily
8,841
stars
1,432
forks
8
watchers
10,285
prs
3,024
issues
22
releases
README
THIS PROJECT IS ARCHIVED Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. This project has been identified as having known security issues. 💫 Intel® LLM Library for PyTorch | 中文 IPEX-LLM is an LLM acceleration library for Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max), NPU and CPU [^1]. [!NOTE] - IPEX-LLM provides seamless integration with llama.cpp, Ollama, vLLM, HuggingFace transformers, LangChain, LlamaIndex, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, ModeScope, etc. - 70+ models have been optimized/verified on ipex-llm (e.g., Llama, Phi, Mistral, Mixtral, DeepSeek, Qwen, ChatGLM, MiniCPM, Qwen-VL, MiniCPM-V and more), with state-of-art LLM optimizations, XPU acceleration and low-bit (FP8/FP6/FP4/INT4) support; see the complete list here. Latest Update 🔥 - [2025/05] You can now run DeepSeek V3/R1 671B and Qwen3MoE 235B models with just 1 or 2 Intel Arc GPU (such as A770 or B580) using FlashMoE in ipex-llm. - [2025/04] We released ipex-llm 2.2.0, which includes Ollama Portable Zip and llama.cpp Portable Zip. ⚠️ Warning (for llama.cpp Portable Zip) mmap-based model loading in llama.cpp may leak data via side-channels in multi-tenant or shared-host environments. To disable mmap, add: bash --no-mmap - [2025/04] We added support of PyTorch 2.6 for Intel GPU. - [2025/03] We added support for Gemma3 model in the latest llama.cpp Portable Zip. - [2025/03] We can now run DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon using the latest llama.cpp Portable Zip. - [2025/02] We added support of llama.cpp Portable Zip for Intel GPU (both Windows and Linux) and NPU (Windows only). - [2025/02] We added support of Ollama Portable Zip to directly run Ollama on Intel GPU for both Windows and Linux (without the need of manual installations). - [2025/02] We added support for running vLLM 0.6.6 on Intel Arc GPUs. - [2025/01] We added the guide for running ipex-llm on Intel Arc B580 GPU. - [2025/01] We added support for running Ollama 0.5.4 on Intel GPU. - [2024/12] We added both Python and C++ support for Intel Core Ultra NPU (including 100H, 200V, 200K and 200H series). More updates - [2024/11] We added support for running vLLM 0.6.2 on Intel Arc GPUs. - [2024/07] We added support for running Microsoft's GraphRAG using local LLM on Intel GPU; see the quickstart guide here. - [2024/07] We added extensive support for Large Multimodal Models, including StableDiffusion, Phi-3-Vision, Qwen-VL, and more. - [2024/07] We added FP6 support on Intel GPU. - [2024/06] We added experimental NPU support for Intel Core Ultra processors; see the examples here. - [2024/06] We added extensive support of pipeline parallel inference, which makes it easy to run large-sized
Top issues
#12963Support for gemma3 from google62 comments · 2025-03-12#12844Ollama can't run on 9th gen intel CPU or older, please use a newer version of Ollama39 comments · 2025-02-18#13173A770 Performance Issue with INT437 comments · 2025-05-21#12248Brave Leo AI using Ollama and Intel GPU35 comments · 2024-10-23#12723non-stream is not supported33 comments · 2025-01-20#13010Unable to use GLM model30 comments · 2025-03-26#13245gemma3n unable to run ipex-llm26 comments · 2025-06-30#11400 Native API returns: -2 (PI_ERROR_DEVICE_NOT_AVAILABLE)26 comments · 2024-06-22
Discussions
#11138Support for max_loaded_maps and num_parallel variables/parameters1 comments · 2024-05-26#10258What are the relationship and differences between bigdl and openvino?0 comments · 2024-02-27
Latest release
IPEX-LLM release 2.2.0 v2.2.0 2025-04-07Highlights Note: IPEX-LLM v2.2.0 has been updated to include functional and security updates. Users should update to the latest version. Please go to https://github.com/ipex-llm/ipex-llm/releases/tag/v2.2.0 for the downloads.