tiktoken
Fast BPE tokenizer for OpenAI models enabling accurate token counting and text splitting.
Used in agent projects
60
0.5% of 12,697 public repos built with AI coding agents
Rank in prompts, structured output and evals
#2 of 4
by use in those projects
Downloads
46.1M
PyPI, last 7 days, tiktoken
GitHub stars
19.4K
last push Aug 17, 2026
How tiktoken is used
tiktoken is a dependency in 60 of the 12,697 public GitHub projects set up for AI coding agents that we scanned (0.5%), #2 of 4 in prompts, structured output and evals. The most used alternative there is Ultracite. Projects that use it most often also use Pydantic (75.0%), python-dotenv (75.0%), OpenAI Python Library (73.3%).
Often used with tiktoken
| Pydantic | 75.0% of tiktoken projects · 14.7× more than average |
|---|---|
| python-dotenv | 75.0% of tiktoken projects · 11.5× more than average |
| OpenAI Python Library | 73.3% of tiktoken projects · 38.2× more than average |
| NumPy | 68.3% of tiktoken projects · 17.8× more than average |
| PyYAML | 61.7% of tiktoken projects · 20.3× more than average |
| FastAPI | 58.3% of tiktoken projects · 13.8× more than average |
| Uvicorn | 58.3% of tiktoken projects · 13.6× more than average |
| Requests | 56.7% of tiktoken projects · 10.7× more than average |
By coding agent
| GitHub Copilot | 0.9% of projects set up for it |
|---|---|
| Claude Code | 0.7% of projects set up for it |
| Cursor | 0.8% of projects set up for it |
| Gemini CLI | 1.6% of projects set up for it |
Share of public repos configured for each agent that list tiktoken as a dependency.
Install
| Package | pip install tiktoken |
|---|---|
| Main stacks | Python (56), Next.js (2), Other (1), .NET (1) |
| Example projects | Josh-XT/AGiXT, prompt-security/ps-fuzz, owid/etl, sepehrbayat/SEPCC, leonardsellem/readwise-vector-db |
Alternatives to tiktoken
| Tool | Used in agent projects | Downloads / wk | GitHub stars | License |
|---|---|---|---|---|
| Ultracite npm AI-ready code formatter that helps write and generate code faster. |
69 (0.5%) | 1.2M | 3,317 | MIT |
| tokenizers pypi Fast NLP tokenizer library by Hugging Face for text processing. |
32 (0.3%) | 43.7M | 11.2K | — |
| LangSmith pypi Platform client for monitoring, evaluating and debugging LLM applications. |
23 (0.2%) | 19.3M | 1,072 | MIT |