3.5% of 1,622 Python repos built with AI coding agents
Gid Score
80 /100
#63 of 3,794 tools, #1 in prompts, structured output and evals · ranking
Rank in prompts, structured output and evals
#2 of 72
by use in those projects
Downloads
47.9M
PyPI, last 7 days, tiktoken
GitHub stars
19.5K
1,653 forks · last push Aug 17, 2026
How tiktoken (Python) is used
tiktoken (Python) is a dependency in 57 of the 6,208 public GitHub projects set up for AI coding agents that we scanned (3.5% of the 1,622 Python ones), #2 of 72 in prompts, structured output and evals. Counting the projects our scan samples by dependency (Solana 2,661, Ethereum and EVM 1,496, 3D and games 2,818, agent frameworks and SDKs 10,983), it appears in 1,081; those are left out of every share on this page. The most used alternative in our scan is tokenizers (25 projects). Projects that use it most often also use Pydantic (75.4%), OpenAI Python Library (70.2%), python-dotenv (64.9%). It is more common in projects set up for Roo Code (4.5%) than for Cursor (0.8%).
Share of public repos configured for each agent that list tiktoken (Python) as a dependency.
What tiktoken (Python) projects add to it
Share of the 57 public projects built with AI coding agents that use tiktoken (Python) and also depend on each package. From our scan of 6,208 repos, updated Oct 20, 2026.
## tiktoken
- To match a specific OpenAI API model, use `tiktoken.encoding_for_model("gpt-4o")`. Use `tiktoken.get_encoding("o200k_base")` when you know the encoding name.
- If you only need a token count, `enc.encode_to_numpy(text).shape[0]` is an option. Its docstring says it avoids copying tokens into a Python list, and one blogger writes it "barely takes any memory or CPU". It is not in the docs.
- When you change the set of special tokens, build a new `tiktoken.Encoding` with a different `name` that makes the behaviour clear.
- In production, load `Encoding` arguments directly instead of reading private attributes like `_mergeable_ranks`. See `openai_public.py` for examples.
- Prefer passing your `Encoding` object around. Use the `tiktoken_ext` plugin only when `tiktoken.get_encoding` must find it by name.
- For a plugin, create a namespace package with no `tiktoken_ext/__init__.py` and expose `ENCODING_CONSTRUCTORS`. Do not install it in editable mode.
- For API details, read `tiktoken/core.py`. Post questions to the GitHub issue tracker.
Minimal code example: encode, decode and pick a tokeniser by model
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"
# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
tiktoken is a byte pair encoding (BPE) tokeniser that OpenAI publishes "for use with OpenAI's models". The README says to install it from PyPI with pip install tiktoken. The GitHub repository lists the MIT license and was created on 2022-12-01.
The README explains the idea briefly. Language models read sequences of numbers called tokens, and BPE turns text into them. The README lists four properties of BPE. It is reversible and lossless. It works on arbitrary text. It compresses text, with "each token corresponds to about 4 bytes" on average. It lets the model see common subwords. In tiktoken you load an encoding by name with get_encoding("o200k_base") or by model with encoding_for_model("gpt-4o"), then call encode and decode.
Beyond that core, the README covers two other uses. Its "Extending tiktoken" section shows how to add encodings, such as a cl100k_base variant with <|im_start|> and <|im_end|> special tokens. Its educational submodule is for readers who want to see the BPE procedure step by step. [1][2]
Why teams pick it
Look up the tokeniser by model name.tiktoken.encoding_for_model("gpt-4o") returns the tokeniser for a given OpenAI API model. You do not need to know the encoding name first. [1]
Speed claim, with stated conditions. The maintainers write that tiktoken is "between 3-6x faster than a comparable open source tokeniser". The measurement used 1GB of text and the GPT-2 tokeniser, compared against GPT2TokenizerFast. [1]
New encodings as plain objects. You can construct a tiktoken.Encoding from an existing encoding's parts, add special tokens, and pass the object around. The README calls this the preferred option. The tiktoken_ext plugin is for cases where get_encoding must find your encoding by name. [1]
A teaching submodule for BPE.tiktoken._educational can train a small BPE tokeniser on a little text. It can also show how the GPT-4 encoder (cl100k_base) splits a string. [1]
A lighter call when you only need a count. Dhaval Singh points to encode_to_numpy, which returns a numpy array. To get a count, he takes .shape[0] of that array. He writes that this approach "barely takes any memory or CPU". [4]
Know before you start
The API reference is a source file. For API documentation, the README points to tiktoken/core.py. For example code, it points to an OpenAI Cookbook notebook. Questions go to the issue tracker. [1]
Useful methods can go unnoticed. Dhaval Singh writes that encode_to_numpy "is not mentioned in the docs". He found it by reading its docstring in the code. [4]
Large strings can spike memory. Dhaval Singh writes that for large strings like 10MB, memory "can spike a lot". With the 4o encoding, he gives peak memory of up to ~80MB and CPU wall time of up to ~1.3s. [4]
The benchmark is from an early version. The 3-6x speed figure was measured with tiktoken==0.2.0 on the GPT-2 tokeniser. These are the only figures the README cites. [1]
Plugin packaging has strict rules. A tiktoken_ext plugin must be a namespace package with no tiktoken_ext/__init__.py. It needs a module exposing ENCODING_CONSTRUCTORS, and it must not be installed in editable mode. [1]
Built with tiktoken (Python)
Real uses, from the builders' own posts
Dhaval Singh
Reworking a team token-counting function
The question
He was optimizing a token counting function that his team uses to chunk data for embedding models and to cut off older text. He writes that one problem he ran into was memory spiking on large strings like 10MB. He adds that this "can be a potential problem when scale is high".
What they built
The post describes two ways to deal with it. One is approximating with <code>len(text) // 4</code> where a rough count is enough. The other is counting with tiktoken's <code>model.encode_to_numpy(text).shape[0]</code>.
Result
He writes that the <code>encode_to_numpy</code> approach "barely takes any memory or CPU and is much faster too!"
Tools that do the same job, so a project picks one or the other. Add-ons and libraries used alongside tiktoken (Python) are not listed here; they are in "Often used with".
Description written by AgentGid from these official pages, checked Oct 13, 2026. Usage numbers come from our own scan. Claims by the maintainers are quoted, not tested by us.
