modelscope /ms-swift

Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).

★ 16,105⑂ 1,803👁 55⑃ 4,364● 5,253▤ 108
PythonApache-2.0Created 2023-08-01swift.readthedocs.io/zh-cn/latest/
Download: v4.6.1v4.6.1 ↓4.6.12026-11-13
+1
Daily
16,105
stars
1,803
forks
55
watchers
4,364
prs
5,253
issues
108
releases

README

SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)


    
    
    


ModelScope Community Website

        中文 &nbsp | &nbsp English &nbsp

















        Paper &nbsp | English Documentation &nbsp | &nbsp 中文文档 &nbsp


 📖 Table of Contents
- Groups
- Introduction
- News
- Installation
- Quick Start
- Usage
- License
- Citation


 ☎ Groups

You can contact us and communicate with us by adding our group:


Discord Group              |  WeChat Group
:-------------------------:|:-------------------------:
  |  


 📝 Introduction
🍲 ms-swift is a large model and multimodal large model fine-tuning and deployment framework provided by the ModelScope community. It now supports training (pre-training, fine-tuning, human alignment), inference, evaluation, quantization, and deployment for 600+ text-only large models and 400+ multimodal large models. Large models include: Qwen3, Qwen3.5, InternLM3, GLM4.5, Mistral, DeepSeek-R1, Llama4, etc. Multimodal large models include: Qwen3-VL, Qwen3-Omni, Kimi-K3, Llava, InternVL3.5, MiniCPM-V-4, Ovis2.5, GLM4.5-V, DeepSeek-VL2, etc.

🍔 In addition, ms-swift integrates the latest training technologies, including Megatron parallelism techniques such as TP, PP, CP, EP to accelerate training, as well as numerous GRPO algorithm family reinforcement learning algorithms including: GRPO, DAPO, GSPO, SAPO, CISPO, RLOO, Reinforce++, etc. to enhance model intelligence. ms-swift supports a wide range of training tasks, including preference learning algorithms such as DPO, KTO, RM, CPO, SimPO, ORPO, as well as Embedding, Reranker, and sequence classification tasks. ms-swift provides full-pipeline support for large model training, including acceleration for inference, evaluation, and deployment modules using vLLM, SGLang, and LMDeploy, as well as model quantization using GPTQ, AWQ, BNB, and FP8 technologies.

Why Choose ms-swift?

- 🍎 Model Types: Supports 600+ text-only large models, 400+ multimodal large models, and All-to-All full modality models from training to deployment full pipeline, with Day-0 support for popular models.
- Dataset Types: Built-in 150+ datasets for pre-training, fine-tuning, human alignment, multimodal and various other tasks, with support for custom datasets. Users only need to prepare datasets for one-click training.
- Hardware Support: Supports NVIDIA GPUs (A100/H100/RTX series, etc., via CUDA), AMD GPUs (MI300 series, etc., via ROCm), domestic hardware such as Huawei Ascend NPU, MetaX GPU and Moore Threads GPU (via MUSA), as well as Apple MPS and CPU.
- Lightweight Training: Supports lightweight fine-tuning methods such as LoRA, QLoRA, DoRA, LoRA+, LLaMAPro, LongLoRA, LoRA-GA, ReFT, RS-LoRA, Adapter, LISA, etc.
- Quantized Training: Supports training on BNB, AWQ, GPTQ, AQLM, HQQ, EETQ quantized models, requiring only 9GB training resources for 7B models.
- Memory Optimization: GaLore, Q-Galore, UnSloth, Liger-Kernel, Flash-Attention 2/3, and Ulysses and Ring-Attention sequence parallelis

Top issues

Latest release

v4.6.1 v4.6.1 2026-11-13
v4.6.1

 New Models

- NVIDIA-Nemotron-3.5-Super-VL-120B-A12B(10377)
- NVIDIA-Nemotron-3.5-Transcribe-30B-A4B(10458)
- Unlimited-OCR-2(10448)
- MiniCPM-OCR-1.0(10414)
- Olmo-3-7B-Think(10371)
- flint(10482)
- EmbeddingGemma-2(10392)
- Decision / System-1 family: THX-02 (10398), kev-0.8b/4b/9b (10407), laya (10445) and StartLux-Decision-v1.1 4B/9B/27B-AutoThink (10479).

 Decision Task (typed-decision / JEV family)

- Four new decision heads with templates, alignment tests and examples/train/decision/.sh recipes: THX-02 (10398), jaredpalmer/kev- pointer heads (10407), Laya (10445

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON