RedPajama

An open reproduction effort from Together AI, RedPajama set out to recreate the training data behind LLaMA-class models and publish every step. RedPajama-V1 released a roughly 1 trillion token corpus following the original LLaMA recipe, along with the RedPajama-INCITE models trained on it through a fully open pipeline. RedPajama-V2 scales far beyond that: over 100 billion raw documents from 84 CommonCrawl snapshots processed with the CCNet pipeline, yielding about 30 trillion tokens across English, German, French, Spanish, and Italian, with 30 billion documents carrying quality signals covering natural language metrics, repetitiveness, toxicity, and deduplication. Rather than shipping one filtered dataset, V2 exposes these annotations so teams can apply their own filtering recipes. The code is Apache 2.0 licensed while the data follows Common Crawl terms of use, and the corpora have served as pretraining material for hundreds of open models.