FlashAttention

FlashAttention, from Tri Dao and collaborators, is the reference implementation of IO-aware exact attention: it computes standard attention exactly but tiles the computation to minimize traffic to GPU high-bandwidth memory, cutting memory use from quadratic to linear in sequence length and running 2 to 4 times faster than naive PyTorch attention, with roughly 20x memory savings at 4K context. FlashAttention-2 is the stable release, FlashAttention-3 targets Hopper GPUs such as the H100 with FP16, BF16, and FP8 support, and a CuTeDSL-based version 4 covers Hopper and Blackwell. Features include causal and sliding-window masking, multi-query and grouped-query attention, KV-cache inference paths, rotary embeddings, ALiBi, and softcapping. It supports NVIDIA GPUs from Ampere onward with CUDA 12 and AMD GPUs via ROCm, and it integrates with PyTorch and the Hugging Face ecosystem. BSD-3-Clause licensed, it has become a standard building block for training and serving large language models.