JavaScript BPE tokenizer with pricing and token counts for OpenAI models.
GPT Tokenizer is a dependency in 14 of the 6,208 public GitHub projects set up for AI coding agents that we scanned, too few to describe it from them alone. Our scan also samples projects by dependency (Solana 2,661, Ethereum and EVM 1,496, 3D and games 2,818, agent frameworks and SDKs 10,983); counting those, it appears in 39 projects, and what it is used with below comes from all of them. The most used alternative in our scan is tiktoken (JavaScript) (10 projects). Projects that use it most often also use OpenAI (46.2%), Anthropic SDK (43.6%), Husky (43.6%).
Gid Score 44: downloads 65, use in agent projects 37, GitHub stars 9, downloads trend 66 (each 0-100). How it is calculated.
| OpenAI | 46.2% of GPT Tokenizer projects · 10.1× more than average |
|---|---|
| Anthropic SDK | 43.6% of GPT Tokenizer projects · 11.7× more than average |
| Husky | 43.6% of GPT Tokenizer projects · 4.3× more than average |
| Google Generative AI | 35.9% of GPT Tokenizer projects · 13.2× more than average |
| lint-staged | 35.9% of GPT Tokenizer projects · 4.6× more than average |
| uuid | 35.9% of GPT Tokenizer projects · 7.6× more than average |
| tsx | 46.2% of GPT Tokenizer projects · 3.1× more than average |
| LangChain.js | 33.3% of GPT Tokenizer projects · 55.9× more than average |
From all 39 projects in our scan that use GPT Tokenizer, including the dependency samples.
| Claude Code | 0.3% of projects set up for it |
|---|---|
| Cursor | 0.4% of projects set up for it |
| GitHub Copilot | 0.3% of projects set up for it |
Share of public repos configured for each agent that list GPT Tokenizer as a dependency.
Share of the 14 public projects built with AI coding agents that use GPT Tokenizer and also depend on each package. From our scan of 6,208 repos, updated Oct 20, 2026.
| JavaScript linter eslint | 69.2% | |
|---|---|---|
| Schema validation zod | 53.8% | |
| OpenAI SDK openai | 46.2% | |
| Git hooks husky | 43.6% | |
| Lint staged files lint-staged | 35.9% | |
| Vercel AI SDK ai | 30.8% |
Our summary of the project's own guidance. [1][3] Which file does your agent read?
## GPT Tokenizer
- A bare `import ... from 'gpt-tokenizer'` uses `o200k_base` (GPT-5, GPT-4.1, GPT-4o, o-series). For other models import `gpt-tokenizer/model/<model>`; for an unlisted model with a known encoding import `gpt-tokenizer/encoding/<encoding>`.
- For limit checks use `isWithinTokenLimit(input, limit)`, not `encode(text).length`. It returns `false` when over the limit, otherwise the token count.
- When calling `encodeChat` from the default import, pass the model name. Otherwise import the model-specific module.
- Special tokens are disallowed by default. Pass `{ allowedSpecial: ALL_SPECIAL_TOKENS }` or a custom `Set` only when the input is meant to contain them.
- Use `countChatCompletionTokens` for requests with function definitions, only on models that support `function_calling`.
- `estimateCost` needs a model spec, from a model import or as the second argument. Treat fields like `main?.input` as possibly missing.
- In the browser, import the ES module (`https://unpkg.com/gpt-tokenizer/esm/main.js`) and pin a version. UMD bundles were removed in 4.0.0.
import {
countChatCompletionTokens,
type ChatCompletionRequest,
} from 'gpt-tokenizer/model/gpt-4o'
const request: ChatCompletionRequest = {
messages: [
{ role: 'system', content: 'You are a helpful assistant.' },
{ role: 'user', content: 'Find the weather for San Francisco.' },
],
functions: [
{
name: 'get_weather',
description: 'Look up the weather for a city.',
parameters: {
type: 'object',
required: ['city'],
properties: {
city: { type: 'string' },
unit: { type: 'string', enum: ['celsius', 'fahrenheit'] },
},
},
},
],
}
const promptTokenEstimate = countChatCompletionTokens(request)Code: MIT, © niieani, from the project README [1] · Open the playground
gpt-tokenizer is a TypeScript Byte Pair Encoding encoder and decoder for OpenAI models, published on npm as gpt-tokenizer under the MIT license. The repository describes it as a "Port of OpenAI's tiktoken with additional features". It was created on 2023-03-22. The latest release is 4.0.0, published 2026-08-16.
It turns text into token integers and back. A plain import uses o200k_base, the encoding the README gives for GPT-5, GPT-4.1, GPT-4o and the o-series. Other models get their own import path, such as gpt-tokenizer/model/gpt-3.5-turbo, and any encoding can be loaded directly, such as gpt-tokenizer/encoding/cl100k_base. Beyond encode and decode, there are helpers for chats, token limits, function-calling requests, streaming generators and cost estimates.
It is for JavaScript and TypeScript developers who need token counts for OpenAI models in Node or in the browser. The builders cited below used it for browser-based token counters and for an n8n node that counts tokens and cost before calling OpenAI. [1][2]
isWithinTokenLimit takes a string or a chat and returns false once the limit is exceeded, or the token count if it fits. The README says it does this without encoding the entire input. [1]countChatCompletionTokens takes a whole chat completion request and counts message overhead, function definitions and pinned function calls. It is only available on models that support function_calling. [1]estimateCost returns USD figures for the main and batch APIs, including cached tokens where data exists. [1]decodeAsyncGenerator for async token streams. [1]EncodeOptions. The README also notes that not every model supports every special token. [1]encodeChat from the default import needs the model name passed in. Otherwise you must import the model-specific module. [1]estimateCost needs a model spec, either from a model import or as an argument. Even then, the README warns that pricing may not exist for every model. [1]Real uses, from the builders' own posts
Koala (koala.sh)
geckse (Marcel), n8n community
LaunchBoosts
Mahmoud Zalt (zalt.me)
Official in-browser playground for trying the tokenizer.
playground browser Source
Official live demos by the maintainers; they open on the project's site and run WebGL in your browser. [1]
Microsoft (Teams, GenAIScript) Elastic (Kibana) Effect TS CodeRabbit Rivet by Ironclad
As listed by the source; we have not checked each one. [1]
| Latest version | 4.0.0, August 16, 2026 [3] |
|---|---|
| On GitHub since | March 2023 [2] |
| License | MIT [2] |
On AgentGid: All prompts, structured output and evals Husky Anthropic SDK Testing Library
| Package | npm install gpt-tokenizer |
|---|---|
| Main stacks | TypeScript / JavaScript (22), Next.js (15), Python (1), Rust (1) |
| Example projects | yamadashy/repomix, HoangNguyen0403/agent-skills-standard, Doist/todoist-mcp, n-WN/SubgroupX, vual/lobe-chat-pro |
Tools that do the same job, so a project picks one or the other. Add-ons and libraries used alongside GPT Tokenizer are not listed here; they are in "Often used with".
| Tool | Gid Score | Used in agent projects | Downloads / wk | GitHub stars | License |
|---|---|---|---|---|---|
| tiktoken (JavaScript) npm JavaScript/WASM bindings for OpenAI's tokenizer for token counting. |
43 | 10 (0.2%) | 806K | 1,073 | MIT |
| js-tiktoken npm JavaScript implementation of OpenAI's tiktoken for counting and encoding tokens. |
51 | 5 (0.1%) | 5.7M | 1,073 | MIT |
| tokenx npm · new in the catalog Fast token count estimates for LLM prompts in a 2 kB bundle. |
37 | 2 (0.0%) | 1.5M | 185 | MIT |
More in Prompts, structured output and evals: Ultracite tiktoken (Python) tokenizers LangSmith Langfuse (Python) TokenLens @graphql-codegen/typescript-operations partial-json
Description written by AgentGid from these official pages, checked Oct 13, 2026. Usage numbers come from our own scan. Claims by the maintainers are quoted, not tested by us.