Hugging Face Datasets

Hugging Face Datasets is a library for loading, processing, and sharing datasets across NLP, vision, audio, and multimodal tasks. Storage is backed by Apache Arrow and Parquet files with memory-mapped reads, so most datasets stream and process out of core. It integrates with the Hugging Face Hub for download and upload, supports streaming mode for datasets larger than disk, and ships a unified load_dataset() entry point.