tokenizers
Fast NLP tokenizer library by Hugging Face for text processing.
Used in agent projects
32
0.3% of 12,697 public repos built with AI coding agents
Rank in prompts, structured output and evals
#3 of 4
by use in those projects
Downloads
43.7M
PyPI, last 7 days, tokenizers
GitHub stars
11.2K
last push Oct 9, 2026
How tokenizers is used
tokenizers is a dependency in 32 of the 12,697 public GitHub projects set up for AI coding agents that we scanned (0.3%), #3 of 4 in prompts, structured output and evals. The most used alternative there is Ultracite. Projects that use it most often also use NumPy (100.0%), PyYAML (93.8%), Hugging Face Hub (90.6%).
Often used with tokenizers
| NumPy | 100.0% of tokenizers projects · 26.1× more than average |
|---|---|
| PyYAML | 93.8% of tokenizers projects · 30.9× more than average |
| Hugging Face Hub | 90.6% of tokenizers projects · 195.0× more than average |
| Pydantic | 87.5% of tokenizers projects · 17.1× more than average |
| tqdm | 87.5% of tokenizers projects · 78.8× more than average |
| charset-normalizer | 84.4% of tokenizers projects · 67.8× more than average |
| packaging | 84.4% of tokenizers projects · 74.4× more than average |
| Requests | 84.4% of tokenizers projects · 15.9× more than average |
By coding agent
| Claude Code | 0.5% of projects set up for it |
|---|---|
| GitHub Copilot | 0.5% of projects set up for it |
| Cursor | 0.5% of projects set up for it |
Share of public repos configured for each agent that list tokenizers as a dependency.
Install
| Package | pip install tokenizers |
|---|---|
| Main stacks | Python (28), Next.js (3), Other (1) |
| Example projects | Llaminar/llaminar, ManojINaik/manimAnimationAgent, nihilistau/CosySim, brandoz2255/Harvis |
Alternatives to tokenizers
| Tool | Used in agent projects | Downloads / wk | GitHub stars | License |
|---|---|---|---|---|
| Ultracite npm AI-ready code formatter that helps write and generate code faster. |
69 (0.5%) | 1.2M | 3,317 | MIT |
| tiktoken pypi Fast BPE tokenizer for OpenAI models enabling accurate token counting and text splitting. |
60 (0.5%) | 46.1M | 19.4K | — |
| LangSmith pypi Platform client for monitoring, evaluating and debugging LLM applications. |
23 (0.2%) | 19.3M | 1,072 | MIT |