hiyouga /LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
★ 76,051⑂ 9,375👁 342⑃ 1,660● 8,056▤ 36
+9
Daily
+97
Weekly
76,051
stars
9,375
forks
342
watchers
1,660
prs
8,056
issues
36
releases
README
! LlamaFactory Used by Amazon, NVIDIA, Aliyun, etc. ---- Check our new open-source project —🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically! Follow us and give us a star ⭐: https://github.com/Prism-Shadow/penguin-harness https://github.com/user-attachments/assets/9b7033e8-f08a-4c3f-bd33-547896664e6e ---- Supporters ❤️ | Warp, the agentic terminal for developersAvailable for MacOS, Linux, & Windows | | | ---- | ---- | ---- Easily fine-tune 100+ large language models with zero-code CLI and Web UI !GitHub Trend 👋 Join our WeChat and NPU user groups. \ English | [中文 \] Fine-tuning a large language model can be easy as... https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e Start local training: - Please refer to usage Start cloud training: - Colab (free): https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing - PAI-DSW (free trial): https://gallery.pai-ml.com//preview/deepLearning/nlp/llama_factory - AMD GPU Cloud (free credits): https://github.com/AMD-AIM/AMD_Developers_Notebooks/blob/main/en/AMD_developer_LLaMAFactory_note_en.md Read technical notes: - Documentation (WIP): https://llamafactory.readthedocs.io/en/latest/ - Documentation (AMD GPU): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html - Documentation (ASCEND NPU): https://llamafactory.readthedocs.io/en/latest/multibackend/npu/index.html - Official Blog: https://blog.llamafactory.net/en/ [!NOTE] Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them. Table of Contents - Features - Blogs - Changelog - Supported Models - Supported Training Approaches - Provided Datasets - Requirement - Getting Started - Installation - Data Preparation - Quickstart - Fine-Tuning with LLaMA Board GUI - Build Docker - Deploy with OpenAI-style API and vLLM - Download from ModelScope Hub - Download from Modelers Hub - Use W&B Logger - Use SwanLab Logger - Projects using LlamaFactory - License - Citation - Acknowledgement Features - Various models: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi, etc. - Integrated methods: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc. - Scalable resources: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ. - Advanced algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA. - Practical tricks: FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA. - Wide tasks: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc. - Experiment monitors: LlamaBoard, TensorBoard
Top issues
#6792结合 GRPO 支持 DeepSeek-R1 等推理模型的复现,达到 huggingface open-r1 的类似效果60 comments · 2025-02-02#10221qwen3.5-27b 全参微调训练耗时很久43 comments · 2026-02-27#7767Hanging problem when LoRA finte-tune Qwen2.5Omni with multi-turn video-audio samples with deepspeedz341 comments · 2025-04-19#4388关于npu训练模型总结以及疑问35 comments · 2024-06-20#7396Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1580, OpType=ALLREDUCE, NumelIn=466119168, NumelOut=466119168, Timeout(ms)=600000) ran for 600004 milliseconds before timing out.30 comments · 2025-03-21#8600OOM and slow tokenization after upgrade LLaMA-Factory27 comments · 2025-07-10#9474Qwen3omni-30B-A3B-Instruct Lora微调训练一直卡住25 comments · 2025-11-12#8041[question] Dose Muon Optimizer support deepspeed ?25 comments · 2025-05-13
Discussions
#10969loraspectra: Audit spectral rank, detect dead layers, compress, and orthogonally merge LLaMA-Factory LoRA/DoRA .safetensors in Rust0 comments · 2026-11-12#10886Can anyone help me train a NodeJS / JavaScript model ?1 comments · 2026-10-10#10848Which abliterated variants are actually worth hosting? (founder, hosting them)0 comments · 2026-09-25#10843请问webui上面有办法实现多机多卡微调吗1 comments · 2026-09-20#10836Training the Qwen3.5-9B model with only_text data, how to avoid multimodal placeholders1 comments · 2026-09-15#10768Qwen3-30B-A3B-Base SFT-LORA+DeepSpeed ZERO3微调OOM问题2 comments · 2026-08-21#10750VOKO 自荐:做 AI Agent 之间的"通信协议层"0 comments · 2026-08-13#10631How should I balance learning rate and data sampling during CPT on multiple datasets?3 comments · 2026-07-06
Latest release
v0.9.5: Qwen3.5/3.6, Gemma 4, Transformers v5 v0.9.5 2026-05-30Added primary support for Qwen3.5/Qwen3.6/Gemma4 models and compatibility with Transformers v5. What's Changed [misc] set dev version by @hiyouga in https://github.com/hiyouga/LlamaFactory/pull/9703 fix(fp8): add Transformer Engine backend support by @sbhavani in https://github.com/hiyouga/LlamaFactory/pull/9705 [misc] Compatible with an empty architectures field in config.json by @tangefly in https://github.com/hiyouga/LlamaFactory/pull/9709 [model] support Youtu-LLM-2B by @isLinXu in https://github.com/hiyouga/LlamaFactory/pull/9707 [misc] lint by @hiyouga in https://github.c