+3
Daily
5,909
stars
1,080
forks
58
watchers
5,018
prs
896
issues
6
releases
README
A PyTorch-native platform for pretraining and post-training generative AI models at scale Quick start · Models · Features · RL · Docs · Contributing torchtitan is a clean-room, PyTorch-native implementation of large-scale training. One readable codebase takes a model from pretraining through supervised fine-tuning to reinforcement learning, composing FSDP, tensor, pipeline, context, and expert parallelism without rewriting the model. - Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP, and EP stack into N-D layouts through DeviceMesh and DTensor. - Built for research. Minimal, hackable code; every run is a Python recipe you can read, copy, and modify. - Pretraining to RL in one stack. TitanRL runs the same model definitions and kernels in the trainer and in vLLM generation, with a bitwise-reproducible batch-invariant mode. - Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more. [!NOTE] torchtitan is under active development. To use the latest features, use a recent PyTorch nightly. Latest news - [2026/10] Zephon, a data loader by DatologyAI, released its TorchTitan integration. Zephon keeps the same training batches in the same order even when your GPU count changes (elastic determinism). Check out the blog post on pytorch.org. - [2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode. - [2025/11] AMD released an optimized fork of torchtitan for AMD GPUs. - [2025/10] We released torchtitan v0.2.0. - [2025/10] SkyPilot now supports torchtitan! See the tutorial. Older news - [2025/07] We published instructions on how to add a model to torchtitan. - [2025/04] Our paper was accepted by ICLR 2025. - [2024/12] GPU MODE lecture on torchtitan. - [2024/07] Presentation at PyTorch Conference 2024. Quick start 1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build. bash git clone https://github.com/pytorch/torchtitan && cd torchtitan pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132 pip install -r requirements.txt 2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed. bash NGPU=1 ./run_train.sh Llama 3 debug model, 10 steps 3. Train a real model. Download the tokenizer (Llama weights require access on Hugging Face), then select a recipe by module and function name: bash python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=... MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh 8 GPUs 4. Make it yours. A recipe is a Python function returning a complete config, so changing a run means writing a function, not passing flags: python my_recipes.py from torchtitan_recipes.models.llama3 import
Top issues
#4272Kimi K3 issue tracking20 comments · 2026-08-21#5019NCCL all-reduce OOM despite reclaimable PyTorch cache during Kimi K3 training15 comments · 2026-10-02#431Question about Pipeline parallelism13 comments · 2024-06-27#3336Investigate mismatch between cpu received tokens and cuda received tokens in EP11 comments · 2026-05-13#3368[question] Why is the load balancing disabled for multi-mask models10 comments · 2026-05-15#3123[SimpleFSDP] SimpleFSDP silently ignores `enable_fsdp_float8_all_gather=True`8 comments · 2026-04-28#5692[RFC] Support additional training diagnostics in TorchTitan7 comments · 2026-10-29#4349[RFC] Support Anticipatory Routing to mitigate MoE training loss spikes6 comments · 2026-08-27
Discussions
#3911Validation Loss Plateaus only after 2000 steps Pretraining Regardless of LR Schedule (Follow-up to #3868)3 comments · 2026-07-13#3882Can I use 'torchtitan' for any model architecture?1 comments · 2026-07-07#3868How should I balance learning rate and data sampling during CPT on multiple datasets?6 comments · 2026-07-06#3589Requesting some insight on time spent during recovery when training with torchFT enabled1 comments · 2026-06-09#2918Usage of nn.Identity1 comments · 2026-04-09#2893MFU calculation for Quantized Workloads1 comments · 2026-04-08#2796torchtitan is featured on Shypd — 84K+ AI tools0 comments · 2026-04-02#2790RoPE config: enforcing `dim`1 comments · 2026-04-02
Latest release
v0.4.0 v0.4.0 2026-10-28TorchTitan 0.4.0 TorchTitan 0.4.0 aligns with PyTorch 2.15, expands model coverage, improves large-scale MoE training, and unifies core training and TitanRL on a shared engine. These highlights cover changes since 0.3.0. For broader updates and benchmarks, see TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL. New model implementations - Muse Spark Open: Adds the 816B-A42B model, including text and vision/audio configurations. Multimodal CUDA graphs remain unsupported. (5420) - Kimi K3: Hybrid KDA/MLA attention, LatentMoE, and vision support, with tensor/sequence