+6
Daily
+54
Weekly
+152
Monthly
25,299
stars
3,231
forks
144
watchers
1,240
prs
1,860
issues
131
releases
README
FlashAttention
This repository provides the official implementation of FlashAttention and
FlashAttention-2 from the
following papers.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
Paper: https://arxiv.org/abs/2205.14135
IEEE Spectrum article about our submission to the MLPerf 2.0 benchmark using FlashAttention.
!FlashAttention
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Tri Dao
Paper: https://tridao.me/publications/flash2/flash2.pdf
!FlashAttention-2
Usage
We've been very happy to see FlashAttention being widely adopted in such a short
time after its release. This page
contains a partial list of places where FlashAttention is being used.
FlashAttention and FlashAttention-2 are free to use and modify (see LICENSE).
Please cite and credit FlashAttention if you use it.
FlashAttention-3 beta release
FlashAttention-3 is optimized for Hopper GPUs (e.g. H100).
Blogpost: https://tridao.me/blog/2024/flash3/
Paper: https://tridao.me/publications/flash3/flash3.pdf
!FlashAttention-3 speedup on H100 80GB SXM5 with FP16
This is a beta release for testing / benchmarking before we integrate that with
the rest of the repo.
Currently released:
- FP16 / BF16 forward and backward, FP8 forward
Requirements: H100 / H800 GPU, CUDA = 12.3.
We highly recommend CUDA 12.8 for best performance.
To install:
sh
cd hopper
python setup.py install
To run the test:
sh
export PYTHONPATH=$PWD
pytest -q -s test_flash_attn.py
Once the package is installed, you can import it as follows:
python
from flash_attn_3 import flash_attn_interface
flash_attn_interface.flash_attn_func()
To install using uv, in your pyproject.toml:
toml
[project]
dependencies = [
"flash-attn-3"
]
[tool.uv]
no-build-isolation = true
[tool.uv.sources]
flash-attn-3 = { git = "https://github.com/Dao-AILab/flash-attention", subdirectory = "hopper" }
FlashAttention-4 (CuTeDSL)
FlashAttention-4 is written in CuTeDSL and optimized for Hopper and Blackwell GPUs (e.g. H100, B200).
To install:
sh
pip install flash-attn-4
If you're on CUDA 13, we recommend installing with the cu13 extra for best performance:
sh
pip install "flash-attn-4[cu13]"
Once installed, you can use it as follows:
python
from flash_attn.cute import flash_attn_func
out = flash_attn_func(q, k, v, causal=True)
Installation and features
Requirements:
- CUDA toolkit or ROCm toolkit
- PyTorch 2.2 and above.
- packaging Python package (pip install packaging)
- psutil Python package (pip install psutil)
- ninja Python package (pip install ninja)
- Linux. Might work for Windows starting v2.3.2 (we've seen a few positive reports) but Windows compilation still requires more testing. If you have ideas on how to set up prebuilt CUDA wheels for Windows, please reach out via Github issue.
\ Make sure that ninja is installed and that it works correctly (e.g. ninja
--version then echo Top issues
#595Might be a solution to get built/compiles Flash Attention 2 on Windows87 comments · 2023-10-08#1038Build flash-attn takes a lot of time 77 comments · 2024-07-10#246No Module Named 'torch'77 comments · 2023-05-28#1464Add support for Cuda 12.8 and B200 GPUs55 comments · 2025-01-25#1708ImportError: /lib64/libc.so.6: version `GLIBC_2.32' not found44 comments · 2025-06-11#945[bug] build is verrrrrrrrrrrrrrrrrrrry slow42 comments · 2024-05-11#358Not able to install 2.033 comments · 2023-07-20#879Has anyone successfully compiled this on ARM Linux (aarch64)?31 comments · 2024-03-07
Discussions
#2767What should long-context support prove beyond maximum length?0 comments · 2026-08-07#2693Does FA4 support split-kv flashdecode with cudagraph?1 comments · 2026-07-02#2644文章寻求0 comments · 2026-06-11#2612Source commit for FlashAttention 3.0.0b11 comments · 2026-06-01#2240Routed Attention: Learning when to use O(N) conv vs O(N²) attention (75-99% compute savings)0 comments · 2026-02-07#2114Why does FA3 use persistent blocks while FA2 doesn't?1 comments · 2025-12-31#2112When can I use the logsumexp I get from return_attn_probs?1 comments · 2025-12-30#2012Clarification: baseline GLIBC version for 2.8.x releases1 comments · 2025-11-14
Latest release
v2.8.3.post2 v2.8.3.post2 2026-10-13Maintenance release of FlashAttention 2 on top of v2.8.3.post1. No kernel feature changes. - Backport 2622: torch.load(..., weights_only=True) in flash_attn/utils/distributed.py and flash_attn/models (CVE-2026-31253 / GHSA-7g5w-pq96-8c5w). 2974 - Backport 2702: safe tarfile extraction in hopper/setup.py (CVE-2026-62239 / GHSA-w9r7-4gwr-j959). 2974 - Backport 2260: fix int32 overflow in the FA2 backward dq_accum offsets, which caused an illegal memory access or NaN gradients when batch round128(seqlen_q) nheads headdim exceeded 2^31 (2987). 2994 - setup.py: request cu13 prebuilt wheels for