modelscope /ms-swift
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).
★ 16,105⑂ 1,803👁 55⑃ 4,364● 5,253▤ 108
More from this author
+1
Daily
16,105
stars
1,803
forks
55
watchers
4,364
prs
5,253
issues
108
releases
README
SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)
ModelScope Community Website
中文   |   English  
Paper   | English Documentation   |   中文文档  
📖 Table of Contents
- Groups
- Introduction
- News
- Installation
- Quick Start
- Usage
- License
- Citation
☎ Groups
You can contact us and communicate with us by adding our group:
Discord Group | WeChat Group
:-------------------------:|:-------------------------:
|
📝 Introduction
🍲 ms-swift is a large model and multimodal large model fine-tuning and deployment framework provided by the ModelScope community. It now supports training (pre-training, fine-tuning, human alignment), inference, evaluation, quantization, and deployment for 600+ text-only large models and 400+ multimodal large models. Large models include: Qwen3, Qwen3.5, InternLM3, GLM4.5, Mistral, DeepSeek-R1, Llama4, etc. Multimodal large models include: Qwen3-VL, Qwen3-Omni, Kimi-K3, Llava, InternVL3.5, MiniCPM-V-4, Ovis2.5, GLM4.5-V, DeepSeek-VL2, etc.
🍔 In addition, ms-swift integrates the latest training technologies, including Megatron parallelism techniques such as TP, PP, CP, EP to accelerate training, as well as numerous GRPO algorithm family reinforcement learning algorithms including: GRPO, DAPO, GSPO, SAPO, CISPO, RLOO, Reinforce++, etc. to enhance model intelligence. ms-swift supports a wide range of training tasks, including preference learning algorithms such as DPO, KTO, RM, CPO, SimPO, ORPO, as well as Embedding, Reranker, and sequence classification tasks. ms-swift provides full-pipeline support for large model training, including acceleration for inference, evaluation, and deployment modules using vLLM, SGLang, and LMDeploy, as well as model quantization using GPTQ, AWQ, BNB, and FP8 technologies.
Why Choose ms-swift?
- 🍎 Model Types: Supports 600+ text-only large models, 400+ multimodal large models, and All-to-All full modality models from training to deployment full pipeline, with Day-0 support for popular models.
- Dataset Types: Built-in 150+ datasets for pre-training, fine-tuning, human alignment, multimodal and various other tasks, with support for custom datasets. Users only need to prepare datasets for one-click training.
- Hardware Support: Supports NVIDIA GPUs (A100/H100/RTX series, etc., via CUDA), AMD GPUs (MI300 series, etc., via ROCm), domestic hardware such as Huawei Ascend NPU, MetaX GPU and Moore Threads GPU (via MUSA), as well as Apple MPS and CPU.
- Lightweight Training: Supports lightweight fine-tuning methods such as LoRA, QLoRA, DoRA, LoRA+, LLaMAPro, LongLoRA, LoRA-GA, ReFT, RS-LoRA, Adapter, LISA, etc.
- Quantized Training: Supports training on BNB, AWQ, GPTQ, AQLM, HQQ, EETQ quantized models, requiring only 9GB training resources for 7B models.
- Memory Optimization: GaLore, Q-Galore, UnSloth, Liger-Kernel, Flash-Attention 2/3, and Ulysses and Ring-Attention sequence parallelisTop issues
#4030🚀 Best Practices for Training Qwen3/Qwen3-MoE146 comments · 2025-04-28#7250👋Welcome ms-swift v462 comments · 2025-12-30#4561🍭[Roadmap] ms-swift3.6-3.848 comments · 2025-06-11#5507qwen3 sft的时候的think33 comments · 2025-08-23#6476qwen3-235b megatron pipeline并行报错25 comments · 2025-11-07#8053grpo训练超时 rollout成功拉起 但是grpo训练报错 XXX:51216 timeour21 comments · 2026-02-15#6364GME模型分别用vLLM部署和swift部署(使用vllm backend )的服务,返回的Emb结果不一致21 comments · 2025-10-30#8289Best-Practice of Qwen3.5 GRPO training by Megatron-Swift20 comments · 2026-03-11
Latest release
v4.6.1 v4.6.1 2026-11-13v4.6.1 New Models - NVIDIA-Nemotron-3.5-Super-VL-120B-A12B(10377) - NVIDIA-Nemotron-3.5-Transcribe-30B-A4B(10458) - Unlimited-OCR-2(10448) - MiniCPM-OCR-1.0(10414) - Olmo-3-7B-Think(10371) - flint(10482) - EmbeddingGemma-2(10392) - Decision / System-1 family: THX-02 (10398), kev-0.8b/4b/9b (10407), laya (10445) and StartLux-Decision-v1.1 4B/9B/27B-AutoThink (10479). Decision Task (typed-decision / JEV family) - Four new decision heads with templates, alignment tests and examples/train/decision/.sh recipes: THX-02 (10398), jaredpalmer/kev- pointer heads (10407), Laya (10445