#tokenizer
GitHub repositories that have self-applied the topic "tokenizer" — a creator-tagged metadata that surfaces how AI projects describe themselves.
REPOS Repos for #tokenizer (top 8 by stars)
Arabic Costs Double on AI — why every Arabic-script language (Darija, MSA, Persian, Urdu) pays a 2x token tax on ChatGPT, Claude & Gemini.
Dragon-01-you/xh-forensicsAI 中转站 API 验真取证工具:模型验真 · 系统提示词提取 · 上下游供应链溯源 | LLM relay API forensics: verify models, extract system prompts, trace upstream supply chains
zch6a/build-your-own-aiBuild your own AI from scratch: tokenizer, autograd, GPT, inference, vector database, RAG, agent and MCP. Each a small, tested, readable module that runs on a laptop CPU with no API key.
KeigoShimadaCC/llm-from-scratchMac-local LLM-from-scratch lab for building and evaluating a small decoder-only language model on Apple Silicon.
lanxingyun/ctx-autopsyWhere your agent's context went — and why it died. Local forensics for LLM request bodies and agent session logs.
s3bastiankoch/tiktoken-fastDrop-in replacement for OpenAI's tiktoken in Python, JavaScript and Go, plus a Rust crate. Same token ids, 3 to 11x faster.
mahirfatih/context-shrink-mcpMCP server that estimates, trims and compacts context to keep any OpenAI-compatible endpoint (oMLX, Ollama, llama.cpp, LM Studio, vLLM, OpenAI) under its context window.
ATOM00blue/tokenmeterEstimate LLM token counts and $ cost for any text, file, or directory across OpenAI, Anthropic, and Google models.
RELATED Other topics · full topics ranking →
#claude-code
3,787#ai-agents
3,201#llm
2,090#mcp
1,866#claude
1,854#python
1,678#developer-tools
1,530#codex
1,376#ai
1,181Aggregated by case-insensitive match against topics_json of each repo's latest content snapshot. methodology