Hugging Face Tokenizers
Ultra-fast text tokenization library in Rust with Python bindings.
About
Hugging Face Tokenizers is a tokenization library with a Rust core and bindings for Python and other languages, built for speed and versatility. It implements Byte-Pair Encoding, WordPiece, and Unigram models, trains custom tokenizers in a couple of lines, and handles configurable pre- and post-processing, tokenizing about a gigabyte of text in under twenty seconds on a server CPU. Released under the Apache 2.0 license.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- AI Frameworks & Libraries
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Easy (2/5)
- License
- Apache-2.0
- Added
- Apr 3, 2026
Related Tools
Unified system for large-scale distributed training and inference.
Guidance language for controlling LLM generation with constraints, grammars, and JSON schemas.
Library for easily accessing and processing ML datasets.
JAX neural network library with PyTorch-like module system.
High-level deep learning API supporting JAX, TensorFlow, and PyTorch backends.
Gradient boosting library by Yandex with native categorical feature handling.