huggingface /accelerate
🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
★ 9,947⑂ 1,607👁 94⑃ 2,463● 2,060▤ 73
+3
Daily
9,947
stars
1,607
forks
94
watchers
2,463
prs
2,060
issues
73
releases
README
--
Run your raw PyTorch training script on any kind of device
Easy to integrate
🤗 Accelerate was created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boilerplate code needed to use multi-GPUs/TPU/fp16.
🤗 Accelerate abstracts exactly and only the boilerplate code related to multi-GPUs/TPU/fp16 and leaves the rest of your code unchanged.
Here is an example:
diff
import torch
import torch.nn.functional as F
from datasets import load_dataset
+ from accelerate import Accelerator
+ accelerator = Accelerator()
- device = 'cpu'
+ device = accelerator.device
model = torch.nn.Transformer().to(device)
optimizer = torch.optim.Adam(model.parameters())
dataset = load_dataset('my_dataset')
data = torch.utils.data.DataLoader(dataset, shuffle=True)
+ model, optimizer, data = accelerator.prepare(model, optimizer, data)
model.train()
for epoch in range(10):
for source, targets in data:
source = source.to(device)
targets = targets.to(device)
optimizer.zero_grad()
output = model(source)
loss = F.cross_entropy(output, targets)
- loss.backward()
+ accelerator.backward(loss)
optimizer.step()
As you can see in this example, by adding 5-lines to any standard PyTorch training script you can now run on any kind of single or distributed node setting (single CPU, single GPU, multi-GPUs and TPUs) as well as with or without mixed precision (fp8, fp16, bf16).
In particular, the same code can then be run without modification on your local machine for debugging or your training environment.
🤗 Accelerate even handles the device placement for you (which requires a few more changes to your code, but is safer in general), so you can even simplify your training loop further:
diff
import torch
import torch.nn.functional as F
from datasets import load_dataset
+ from accelerate import Accelerator
- device = 'cpu'
+ accelerator = Accelerator()
- model = torch.nn.Transformer().to(device)
+ model = torch.nn.Transformer()
optimizer = torch.optim.Adam(model.parameters())
dataset = load_dataset('my_dataset')
data = torch.utils.data.DataLoader(dataset, shuffle=True)
+ model, optimizer, data = accelerator.prepare(model, optimizer, data)
model.train()
for epoch in range(10):
for source, targets in data:
- source = source.to(device)
- targets = targets.to(device)
optimizer.zero_grad()
output = model(source)
loss = F.cross_entropy(output, targets)
- loss.backward()
+ accelerator.backward(loss)
optimizer.step()
Want to learn more? Check out the documentation or have a look at our examples.
Launching script
🤗 Accelerate also provides an optional CLI tool that allows you to quickly configure and test your training environment before launching the scripts. No need to remember hoTop issues
#1089how to cleanly exit when using `accelerate launch`25 comments · 2023-02-16#4555Sequence parallelism: let an SP-aware model opt out of the linear-attention refusal, and make `disable_in_eval` configurable2 comments · 2026-11-13#4525CVE-2026-69112 / GHSA-4j2p-28q2-5m79: advisory says <= 1.14.0, but 1.15.0 has the same code. Please clarify the project's position.2 comments · 2026-11-11#4553FSDP2 `cpu_ram_efficient_loading`: ignored modules get sharded, and persistent buffers crash the load0 comments · 2026-11-13
Latest release
v1.15.0: FSDP2 activation memory, dtensor improvements v1.15.0 2026-09-09v1.15.0: FSDP2 activation memory, dtensor improvements FSDP2 A large batch of FSDP2 work this release: two fixes that cut activation memory at long sequence lengths, tied-embedding support on torch = 2.13, and a round of checkpointing correctness and scale fixes. Activation checkpointing was wrapping each child of the matched layer (self_attn, mlp, the norms) instead of the layer itself, so every inter-child activation stayed saved for backward. It now wraps the layer. There's also a new FSDP2-only activation_checkpointing_offload, which moves the remaining per-layer checkpoint i