TinyLlama

TinyLlama set out to answer a simple question: how capable can a 1.1 billion parameter model become if trained far past standard scaling recipes? The project pretrained a compact model on 3 trillion tokens drawn from SlimPajama and StarCoder data in a roughly 7 to 3 natural language to code ratio, using 16 A100-40G GPUs over about 90 days. Because it adopts exactly the same architecture and tokenizer as Llama 2, with grouped query attention, Flash Attention 2, and fused kernels for efficiency, TinyLlama drops directly into most Llama-based tooling. The team released intermediate checkpoints throughout training alongside chat variants fine-tuned on OpenAssistant data, all under the Apache 2.0 license. Its small footprint makes it a practical base for edge and mobile deployment, a draft model for speculative decoding, lightweight applications like game dialogue, and pretraining research, and it became a common reference point for anyone studying small language models trained on very large token budgets.