huggingface /alignment-handbook
Robust recipes to align language models with human and AI preferences
★ 5,715⑂ 496👁 103⑃ 92● 147▤ 0
+2
Daily
5,715
stars
496
forks
103
watchers
92
prs
147
issues
0
releases
README
🤗 Models & Datasets | 📃 Technical Report The Alignment Handbook Robust recipes to continue pretraining and to align language models with human and AI preferences. What is this? Just one year ago, chatbots were out of fashion and most people hadn't heard about techniques like Reinforcement Learning from Human Feedback (RLHF) to align language models with human preferences. Then, OpenAI broke the internet with ChatGPT and Meta followed suit by releasing the Llama series of language models which enabled the ML community to build their very own capable chatbots. This has led to a rich ecosystem of datasets and models that have mostly focused on teaching language models to follow instructions through supervised fine-tuning (SFT). However, we know from the InstructGPT and Llama2 papers that significant gains in helpfulness and safety can be had by augmenting SFT with human (or AI) preferences. At the same time, aligning language models to a set of preferences is a fairly novel idea and there are few public resources available on how to train these models, what data to collect, and what metrics to measure for best downstream performance. The Alignment Handbook aims to fill that gap by providing the community with a series of robust training recipes that span the whole pipeline. News 🗞️ July 24, 2025: We release the full post-training recipe behind SmolLM3-3B: a state-of-the-art hybrid reasoning model 💭 November 21, 2024: We release the recipe for fine-tuning SmolLM2-Instruct. August 18, 2024: We release SmolLM-Instruct v0.2, along with the recipe to fine-tuning small LLMs 💻 April 12, 2024: We release Zephyr 141B (A35B), in collaboration with Argilla and Kaist AI, along with the recipe to fine-tune Mixtral 8x22B with ORPO 🪁 March 12, 2024: We release StarChat2 15B, along with the recipe to train capable coding assistants 🌟 March 1, 2024: We release Zephyr 7B Gemma, which is a new recipe to align Gemma 7B with RLAIF 🔥 February 1, 2024: We release a recipe to align open LLMs with Constitutional AI 📜! See the recipe and the blog post for details. January 18, 2024: We release a suite of evaluations of DPO vs KTO vs IPO, see the recipe and the blog post for details. November 10, 2023: We release all the training code to replicate Zephyr-7b-β 🪁! We also release No Robots, a brand new dataset of 10,000 instructions and demonstrations written entirely by skilled human annotators. Links 🔗 Zephyr 7B models, datasets, and demos How to navigate this project 🧭 This project is simple by design and mostly consists of: scripts to train and evaluate models. Four steps are included: continued pretraining, supervised-finetuning (SFT) for chat, preference alignment with DPO, and supervised-finetuning with preference alignment with ORPO. Each script supports distributed training of the full model weights with DeepSpeed ZeRO-3, or LoRA/QLoRA for parameter-efficient fine-tuning. recipes to reproduce models like Zephyr 7B. Each recipe takes the f
Top issues
#45Reproducing of Lora Model Result on MT-Bench27 comments · 2023-11-22#22How to perform full parameter finetuning without A100 GPUs13 comments · 2023-11-14#38DPO loss8 comments · 2023-11-17#125Major bug: Chat template is not actually applied in run_sft.py and run_dpo.py7 comments · 2024-02-23#59Get this error on run_sft.py when calling "trainer.push_to_hub": [Rank 0] Watchdog caught collective operation timeout7 comments · 2023-12-01#57"RuntimeError: The size of tensor a (0) must match the size of tensor b (4096) at non-singleton dimension 1" (DPO + LoRA)7 comments · 2023-11-28#44Global batch size question7 comments · 2023-11-21#159FSDP + QDoRA Support6 comments · 2024-04-24