DeepSpeed

DeepSpeed is Microsoft's deep learning optimization library for training and serving very large models. Its centerpiece is the ZeRO family of optimizers, which shards model states across GPUs and, with ZeRO-Infinity, offloads to CPU memory and NVMe so models far larger than device memory can train. It combines data, tensor, and pipeline parallelism into 3D parallelism, adds sequence parallelism for long contexts, supports mixture-of-experts training, and includes quantization-based compression through ZeroQuant. The library has been used to train systems such as Megatron-Turing NLG at 530B parameters and BLOOM at 176B. It integrates with PyTorch, Hugging Face Transformers, Accelerate, and Lightning, compiles its C++ and CUDA kernels just in time, and runs on NVIDIA, AMD, and Intel accelerators with partial Windows support. Released under Apache 2.0, it is a standard tool for researchers and engineers doing multi-GPU and multi-node training of large language models.