NVIDIA /Megatron-LM
Ongoing research training transformer models at scale
★ 18,406⑂ 4,735👁 162⑃ 7,276● 1,992▤ 55
+12
Daily
+85
Weekly
+283
Monthly
18,406
stars
4,735
forks
162
watchers
7,276
prs
1,992
issues
55
releases
README
Megatron-LM and Megatron Core ============================= GPU-optimized library for training transformer models at scale About This repository contains two components: Megatron-LM and Megatron Core. Megatron-LM is a reference example that includes Megatron Core plus pre-configured training scripts, ideal for research teams, learning distributed training, and quick experimentation. Megatron Core is a composable library with GPU-optimized building blocks for custom training frameworks. It provides transformer building blocks, advanced parallelism strategies (TP, PP, DP, EP, and CP), mixed precision support (FP16, BF16, FP8, and FP4), and model architectures, ideal for framework developers and ML engineers building custom training pipelines. Megatron Bridge provides bidirectional Hugging Face ↔ Megatron checkpoint conversion with production-ready recipes. Getting Started Compatibility: Python 3.12+ and PyTorch 2.6.0+ recommended. For source development, the tested NGC PyTorch base images are pinned in docker/.ngc_version.dev and docker/.ngc_version.lts. Install from PyPI: bash uv pip install megatron-core Or clone and install from source: bash git clone https://github.com/NVIDIA/Megatron-LM.git cd Megatron-LM uv pip install -e . Note: Building from source can use a lot of memory. If the build runs out of memory, limit parallel compilation jobs by setting MAX_JOBS (for example, MAX_JOBS=4 uv pip install -e .). For NVIDIA GPU Cloud (NGC) container setup and all installation options, review the Installation Guide. - Your First Training Run - End-to-end training examples with data preparation - Parallelism Strategies - Scale training across GPUs with TP, PP, DP, EP, and CP - Contribution Guide - How to contribute to Megatron Core - Style Guide - Python style overrides Latest News - [2026/10] Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core - A guide to reproducing training runs, validating checkpoint resume, and reducing determinism overhead at scale with Megatron Core. - [2026/08] Megatron Core 0.19 - Introduces HybridModel, quantile-based MoE router balancing, packed-sequence MoE dispatch, and CUDA Graph-compatible activation offloading. - [2026/07] DeepSeek-V3 pretraining on GB300 NVL72 - Megatron Core achieves 1,648 TFLOPs/s per GPU for DeepSeek-V3 671B on 256 GPUs. - [2026/05] DeepSeek-V4 initial support - Megatron Core's dev branch includes the initial DeepSeek-V4 implementation; Megatron Bridge provides conversion, inference, and pretraining recipes. - [2026/04] Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron - Muon and other emerging optimizers are now supported in Megatron Core via the new Emerging-Optimizers library. Previous News Project Structure Megatron-LM/ ├── megatron/ │ ├── core/ Megatron Core (kernels, parallelism, building blocks) │ │ ├── models/ Transformer, hybrid, and multimodal models │ │ ├── transformer
Top issues
#5660[MCore][TE][NVFP4] Persist activation global input_scale/input_amax for offline FP4 inference26 comments · 2026-07-06#3945[BUG] "Spike no more" feature was removed18 comments · 2026-03-19#7452Silent numerical-correctness bugs in Megatron Core parallelism composition16 comments · 2026-09-17#6714MFSDP v2: overlap DP-inner and DP-outer communication16 comments · 2026-08-20#4088Hyperparameter Transfer beyond MuP16 comments · 2026-04-01#4091Add MLA support for Muon optimizer15 comments · 2026-04-01#1445[BUG]loss error when using MLA15 comments · 2025-03-03#2797Feature Request: Dual Chunk Attention for Long Context15 comments · 2026-01-02
Discussions
#7497Scaling Bitwise-Deterministic Pretraining for a Trillion-Parameter Nemotron Model with NVIDIA Megatron Core0 comments · 2026-09-18#5002What If Megatron-LM Had Actual Physics?0 comments · 2026-05-26#4061Megatron LM is featured on Shypd — 80K+ AI tools0 comments · 2026-03-30#3915Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether some language capability may eventually move outside the main dense training path0 comments · 2026-03-18#2438Why doesn't theoretical memory take into account the expert parallelism and the partitioning of experts?0 comments · 2025-12-01#1640How to set hierarchical CP group sizes to enable cp_comm_type="a2a+p2p"1 comments · 2025-06-20#1577[QUESTION]About HybridDeviceOptimizer CPU Offloading Behavior1 comments · 2025-05-14#1558About the issue of interleaved pipeline scheduling0 comments · 2025-04-29
Latest release
NVIDIA Megatron Core 0.20.0 core_v0.20.0 2026-10-29MoE DeepEP v2 dispatcher: adds an experimental DeepEP v2 backend for the flex token dispatcher (\5153). NCCL EP MXFP8 communication (experimental): adds MXFP8 dispatch-forward and combine-backward communication to the NCCL EP backend (\6524). NCCL EP eager mode: now NCCL EP supports both eager mode and full-iteration graph mode (\6229). ShortcutMoE: adds shortcut-connected routing in HybridStack and overlaps expert all-to-all communication with paired attention or Mamba computation; CUDA graphs are not supported (\6959). Model Architecture and Support DeepSeek V4: fully