deepspeedai /DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

★ 43,326⑂ 5,072👁 350⑃ 4,614● 3,518▤ 120
PythonApache-2.0Created 2020-01-23www.deepspeed.ai/
Download: v0.20.0v0.20.0 ↓0.20.02026-11-12
+1
Daily
+13
Weekly
43,326
stars
5,072
forks
350
watchers
4,614
prs
3,518
issues
120
releases

README

English · 简体中文



 Office Hours

DeepSpeed hosts regular office hours on the last Tuesday of each month at 12:00 America/New_York to discuss development plans, features, etc.  This meeting is public for anyone to join and ask questions.
The meeting is hosted on Zoom and can be joined here.

 Latest News

 [2026/05] Using Muon Optimizer with DeepSpeed

 [2026/05] System DMA (SDMA) for ZeRO-3: offload collectives off compute units on AMD GPUs for better overlap

 [2026/03] DeepSpeed Team gave a tutorial at ASPLOS 2026 titled "Building Efficient Large-Scale Model Systems with DeepSpeed: From Open-Source Foundations to Emerging Research" 

 [2026/03] Our SuperOffload work received an Honorable Mention for the ASPLOS 2026 Best Paper Award

 [2025/12] DeepSpeed Core API updates: PyTorch-style backward and low-precision master states

 [2025/11] DeepSpeed ZeRO++ powers large-scale distillation training of LLMs for Recommendation Systems at LinkedIn

 [2025/10] We hosted the Ray x DeepSpeed Meetup at Anyscale. We shared our most recent work on SuperOffload, ZenFlow, Muon Optimizer Support, Arctic Long Sequence Training and DeepCompile. Please find the meetup slides here.

 [2025/10] SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips

 [2025/10] Study of ZenFlow and ZeRO offload performance with DeepSpeed CPU core binding

 [2025/08] ZenFlow: Stall-Free Offloading Engine for LLM Training

 [2025/06] Arctic Long Sequence Training (ALST) with DeepSpeed: Scalable And Efficient Training For Multi-Million Token Sequences

 [2025/06] DeepNVMe: Affordable I/O scaling for Deep Learning Applications





 More news
 

   [2025/04] DeepCompile: Unlocking Compiler Optimization for Distributed Training

   [2025/03] DeepSpeed AutoTP: Automatic Tensor Parallel Training of Hugging Face models

[2024/12] Ulysses-Offload: Democratizing Long Context LLM Training

 


---

 Extreme Speed and Scale for DL Training

DeepSpeed enabled the world's most powerful language models (at the time of this writing) such as MT-530B and BLOOM. DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations include ZeRO, ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism, DeepSpeed-MoE, etc.

---

 DeepSpeed Adoption

DeepSpeed was an important part of Microsoft’s
AI at Scale
initiative to enable next-generation AI capabilities at scale, where you can find more
information here.

DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of (if you'd like to include your model please submit a PR):

   Megatron-Turing NLG (530B)
   Jurassic-1 (178B)
   BLOOM (176B)
   GLM (130B)
   xTrimoPGLM (100B)
   YaLM (100B)
   GPT-NeoX (20B)
   AlexaTM (20B)
   Turing NLG (17B)
   METRO-LM (5.4B)

DeepSpeed has been integrated wi

Top issues

Discussions

Latest release

v0.20.0 v0.20.0 2026-11-12
What's Changed
 Sum GPTBigCode's shared key/value gradient across tensor-parallel ranks by @Achyuthan-S in https://github.com/deepspeedai/DeepSpeed/pull/8928
 [AffineIR] Storage maps of AutoEP experts under EDP and ZeRO by @jinyouzhi in https://github.com/deepspeedai/DeepSpeed/pull/8941
 Add klxpu (KUNLUNXIN XPU) self-hosted CI for the v1 unit suite by @GuCarpenter in https://github.com/deepspeedai/DeepSpeed/pull/8923
 ZeRO-3: copy offloaded master weights back to the device through pinned buffer by @pengdurice in https://github.com/deepspeedai/DeepSpeed/pull/8840
 ci: pin torch<2.15 for 

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON