Hugging Face Tokenizers
Ultra-fast text tokenization library in Rust with Python bindings.
Open SourceSelf HostedOffline Capable
0.0 (0)
About
Hugging Face Tokenizers is an ultra-fast tokenization library written in Rust. Provides BPE, WordPiece, and Unigram tokenizers. Trains custom tokenizers and handles pre/post-processing. Used by Transformers library. Apache 2.0 license.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- AI Frameworks & Libraries
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Easy (2/5)
- License
- Apache-2.0
- Added
- Apr 3, 2026
Similar Tools
Featured
Open-source machine learning framework by Meta with dynamic computation graphs.
Open SourceSelf HostedOffline
Intermediate
0.0 (0)
Featured
End-to-end open-source ML platform by Google for training and deployment.
Open SourceSelf HostedOffline
Intermediate
0.0 (0)
High-performance numerical computing library by Google with auto-differentiation.
Open SourceSelf HostedOffline
Advanced
0.0 (0)