TokenLens
Registry of LLM model information including context sizes for AI applications.
How TokenLens is used
TokenLens is a dependency in 12 of the 6,208 public GitHub projects set up for AI coding agents that we scanned, too few to describe it from them alone. Our scan also samples projects by dependency (Solana 2,661, Ethereum and EVM 1,496, 3D and games 2,818, agent frameworks and SDKs 10,983); counting those, it appears in 41 projects, and what it is used with below comes from all of them. The most used alternative in our scan is GPT Tokenizer (14 projects). Projects that use it most often also use AI SDK (100.0%), Class Variance Authority (100.0%), clsx (100.0%).
TokenLens popularity over time
Gid Score 39: downloads 58, use in agent projects 38, GitHub stars 0, downloads trend 62 (each 0-100). How it is calculated.
Often used with TokenLens
| AI SDK | 100.0% of TokenLens projects · 22.7× more than average |
|---|---|
| Class Variance Authority | 100.0% of TokenLens projects · 6.1× more than average |
| clsx | 100.0% of TokenLens projects · 5.1× more than average |
| Lucide React | 100.0% of TokenLens projects · 5.0× more than average |
| Streamdown | 100.0% of TokenLens projects · 194.0× more than average |
| Radix UI | 97.6% of TokenLens projects · 7.5× more than average |
| Nanoid (JavaScript) | 97.6% of TokenLens projects · 59.4× more than average |
| Tailwind Merge | 97.6% of TokenLens projects · 5.2× more than average |
From all 41 projects in our scan that use TokenLens, including the dependency samples.
By coding agent
| Cursor | 0.4% of projects set up for it |
|---|---|
| Claude Code | 0.2% of projects set up for it |
| GitHub Copilot | 0.2% of projects set up for it |
Share of public repos configured for each agent that list TokenLens as a dependency.
Install
| Package | npm install tokenlens |
|---|---|
| Main stacks | Next.js (34), TypeScript / JavaScript (7) |
| Example projects | mastra-ai/ui-dojo, LiorVainer/data-israel, agents-ui/agents-kit, ssdeanx/AgentStack, nsoybean/nira-ai |
Alternatives to TokenLens
Tools that do the same job, so a project picks one or the other. Add-ons and libraries used alongside TokenLens are not listed here; they are in "Often used with".
| Tool | Gid Score | Used in agent projects | Downloads / wk | GitHub stars | License |
|---|---|---|---|---|---|
| GPT Tokenizer npm JavaScript BPE tokenizer with pricing and token counts for OpenAI models. |
44 | 14 (0.3%) | 1.2M | 855 | MIT |
| tiktoken (JavaScript) npm JavaScript/WASM bindings for OpenAI's tokenizer for token counting. |
43 | 10 (0.2%) | 806K | 1,073 | MIT |
| js-tiktoken npm JavaScript implementation of OpenAI's tiktoken for counting and encoding tokens. |
51 | 5 (0.1%) | 5.7M | 1,073 | MIT |
More in Prompts, structured output and evals: Ultracite tiktoken (Python) tokenizers LangSmith Langfuse (Python) @graphql-codegen/typescript-operations partial-json TOON