AI Dev Impact Lab JA
← Topics ranking · 2026-11
GITHUB TOPIC

#tokenizer

GitHub repositories that have self-applied the topic "tokenizer" — a creator-tagged metadata that surfaces how AI projects describe themselves.

8
tagged repos
4
top 8 stars
1
with tool sigs
8
shown

REPOS Repos for #tokenizer (top 8 by stars)

miloudbelarebia/arabic-tax

Arabic Costs Double on AI — why every Arabic-script language (Darija, MSA, Persian, Urdu) pays a 2x token tax on ChatGPT, Claude & Gemini.

Jupyter Notebook ★ 2 AI 70
Dragon-01-you/xh-forensics

AI 中转站 API 验真取证工具:模型验真 · 系统提示词提取 · 上下游供应链溯源 | LLM relay API forensics: verify models, extract system prompts, trace upstream supply chains

Python ★ 2 AI 70
zch6a/build-your-own-ai

Build your own AI from scratch: tokenizer, autograd, GPT, inference, vector database, RAG, agent and MCP. Each a small, tested, readable module that runs on a laptop CPU with no API key.

Python ★ 0 AI 100
KeigoShimadaCC/llm-from-scratch

Mac-local LLM-from-scratch lab for building and evaluating a small decoder-only language model on Apple Silicon.

Python ★ 0 AI 90 2 sig
lanxingyun/ctx-autopsy

Where your agent's context went — and why it died. Local forensics for LLM request bodies and agent session logs.

JavaScript ★ 0 AI 70
s3bastiankoch/tiktoken-fast

Drop-in replacement for OpenAI's tiktoken in Python, JavaScript and Go, plus a Rust crate. Same token ids, 3 to 11x faster.

Rust ★ 0 AI 60
mahirfatih/context-shrink-mcp

MCP server that estimates, trims and compacts context to keep any OpenAI-compatible endpoint (oMLX, Ollama, llama.cpp, LM Studio, vLLM, OpenAI) under its context window.

Python ★ 0 AI 60
ATOM00blue/tokenmeter

Estimate LLM token counts and $ cost for any text, file, or directory across OpenAI, Anthropic, and Google models.

Python ★ 0 AI 40

RELATED Other topics · full topics ranking →

#claude-code

3,787

#ai-agents

3,201

#llm

2,090

#mcp

1,866

#claude

1,854

#python

1,678

#developer-tools

1,530

#codex

1,376

#ai

1,181

Aggregated by case-insensitive match against topics_json of each repo's latest content snapshot. methodology