kvcache-ai /ktransformers
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
★ 19,675⑂ 1,602👁 120⑃ 813● 1,317▤ 34
+4
Daily
+17
Weekly
+63
Monthly
19,675
stars
1,602
forks
120
watchers
813
prs
1,317
issues
34
releases
README
A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations 🎯 Overview | 🚀 Inference | 🎓 SFT | 🔥 Citation | 🚀 Roadmap(2026Q2) 🎯 Overview KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT. 🔥 Updates Oct 15, 2026: DeepSeek-V4-Flash LoRA fine-tuning supports native MXFP4 routed experts. quickstart. Oct 8, 2026: We release the author preprint of KTransformers-FineTune (KT-FT), presenting layout-aware and schedule-optimized CPU-GPU co-computation for MoE fine-tuning. Paper and citation · PDF. Sep 13, 2026: Kimi K2.5 / K2.6 LoRA fine-tuning is supported with native RAWINT4 routed experts. Aug 26, 2026: Added native support for GLM-5.3-flash, bringing 1M-token context and multimodal input to consumer GPUs. (Tutorial) Aug 25, 2026: Uploaded a new easy-to-use KTransformers × LlamaFactory MoE Fine-Tuning Cookbook, covering hardware checks, installation, BF16/FP8/INT8 recipes, LoRA and full fine-tuning, resource planning, and troubleshooting. Aug 17, 2026: LoRA fine-tuning now supports compatible AVX512 x86 CPUs, including AMD servers, without requiring AMX. (v0.7.0 Release Notes) Aug 16, 2026: DeepSeek-V4-Flash on a single Ascend NPU with CPU expert offload. (Tutorial) Aug 5, 2026: Introduced native block-FP8 LoRA fine-tuning (PR 2141) July 23, 2026: Added end-to-end BF16 full-parameter fine-tuning for MoE models, including complete checkpoint saving. (PR 2094) June 21, 2026: MiniMax-M3 Day0 Support! (Tutorial) June 17, 2026: GLM-5.2 Day0 Support! (Tutorial) May 6, 2026: KTransformers at GOSIM Paris 2026 — "Agentic AI on Edge" track. We'll present KT's inference performance on consumer hardware. May 02, 2026: DeepSeek-V4-Flash Support! (Tutorial) Apr 30, 2026: KTransformers v0.6.1 refreshes kt-kernel inference and SFT docs with separate Inference and SFT Quick Start entry points. Mar 26, 2026: Support AVX2-only CPU backend for KT-Kernel inference. (Tutorial) Feb 13, 2026: MiniMax-M2.5 Day0 Support! (Tutorial) Feb 12, 2026: GLM-5 Day0 Support! (Tutorial) Jan 27, 2026: Kimi-K2.5 Day0 Support! (Tutorial) (SFT Tutorial) Jan 22, 2026: Support CPU-GPU Expert Scheduling, Native BF16 and FP8 per channel Precision and AutoDL unified fine-tuning and inference Dec 24, 2025: Support Native MiniMax-M2.1 inference. (Tutorial) Dec 22, 2025: Support RL-DPO fine-tuning with LLaMA-Factory. (Tutorial) Dec 5, 2025: Support Native Kimi-K2-Thinking inference (Tutorial) Nov 6, 2025: Support Kimi-K2-Thinking inference (Tutorial) and fine-tune (Tutorial) Nov 4, 2025: KTransformers Fine-Tuning × LLaMA-Factory Integration. (Tutorial) Oct 27, 2025: Support Ascend NPU. (Tutorial) Oct 10, 2025: Integrating into SGLang. (Roadmap, Blog) Sept 11, 2025: Support Qwen3-Next. (Tutorial) Sept 05, 2025: Support Kimi-K2-0905. (Tut
Top issues
#1417[Bug] deepseek v3/r1 with full context with balance_serve backend74 comments · 2025-07-03#150再次提一下,能否支持2080ti44 comments · 2025-02-12#1785SGLang does not support kt-kernel37 comments · 2026-01-06#514这个模型跑起来,如何使用呀,如何和其他系统进行对接,比如dify 国产的maxkb上如何使用了31 comments · 2025-02-19#482复现了,但DeepSeek-R1-Q4_K_M跑起来速度非常慢,只有约1.5token/s,请问是我配置的原因么?30 comments · 2025-02-19#1598[Feature] Support Native K2 Thinking28 comments · 2025-11-12#1490is this project dead?28 comments · 2025-08-31#1730Performance Query: Is ~9.5 tokens/s normal for Kimi-K2-Thinking-Native on Xeon 6960P (MRDIMM 8800) + RTX 6000 Blackwell?23 comments · 2025-12-18
Latest release
KTransformers v0.7.2 v0.7.2 2026-10-15KTransformers v0.7.2: DeepSeek-V4-Flash LoRA Fine-Tuning KTransformers v0.7.2 adds LoRA fine-tuning for the April DeepSeek-V4-Flash checkpoint. Routed experts remain in native MXFP4 on CPU, while attention and other non-expert modules run on GPU. 🚀 Highlights - Native MXFP4 routed-expert LoRA forward and backward, without expanding the frozen expert weights to BF16. - LLaMA-Factory SFT integration with checkpoint saving and SGLang adapter inference. - Python 3.11 and 3.12 Linux x86-64 wheels. 📦 Installation For training: bash python -m pip install 'ktransformers[sft]==0.