Megatron-LM

Megatron-LM by NVIDIA is a GPU-optimized framework for training transformer models at very large scale. It ships Megatron Core, a composable library of transformer building blocks with tensor, pipeline, data, expert, and context parallelism plus mixed precision, alongside reference training scripts. It has been used to train many of the largest open models. Released under a custom NVIDIA open-source license.