AI開発影響研究所 EN
← Topicランキング · 2026-11
GitHub TOPIC

#tokenizer

GitHub Topic「tokenizer」がついているAI関連リポジトリの集計。Topicはリポジトリ作者が自己申告するメタタグで、AI関連の文脈で何が「tokenizer」と分類されているかを可視化します。

8
タグ付Repo
4
TOP8合計★
1
AIツール痕跡あり
8
TOP表示数

REPOS #tokenizer のRepo (TOP 8 / Stars降順)

miloudbelarebia/arabic-tax

Arabic Costs Double on AI — why every Arabic-script language (Darija, MSA, Persian, Urdu) pays a 2x token tax on ChatGPT, Claude & Gemini.

Jupyter Notebook ★ 2 AI 70
Dragon-01-you/xh-forensics

AI 中转站 API 验真取证工具:模型验真 · 系统提示词提取 · 上下游供应链溯源 | LLM relay API forensics: verify models, extract system prompts, trace upstream supply chains

Python ★ 2 AI 70
zch6a/build-your-own-ai

Build your own AI from scratch: tokenizer, autograd, GPT, inference, vector database, RAG, agent and MCP. Each a small, tested, readable module that runs on a laptop CPU with no API key.

Python ★ 0 AI 100
KeigoShimadaCC/llm-from-scratch

Mac-local LLM-from-scratch lab for building and evaluating a small decoder-only language model on Apple Silicon.

Python ★ 0 AI 90 2 sig
lanxingyun/ctx-autopsy

Where your agent's context went — and why it died. Local forensics for LLM request bodies and agent session logs.

JavaScript ★ 0 AI 70
s3bastiankoch/tiktoken-fast

Drop-in replacement for OpenAI's tiktoken in Python, JavaScript and Go, plus a Rust crate. Same token ids, 3 to 11x faster.

Rust ★ 0 AI 60
mahirfatih/context-shrink-mcp

MCP server that estimates, trims and compacts context to keep any OpenAI-compatible endpoint (oMLX, Ollama, llama.cpp, LM Studio, vLLM, OpenAI) under its context window.

Python ★ 0 AI 60
ATOM00blue/tokenmeter

Estimate LLM token counts and $ cost for any text, file, or directory across OpenAI, Anthropic, and Google models.

Python ★ 0 AI 40

RELATED 他のTopicも見る · 全Topicランキング →

#claude-code

3,787

#ai-agents

3,201

#llm

2,090

#mcp

1,866

#claude

1,854

#python

1,678

#developer-tools

1,530

#codex

1,376

#ai

1,181

集計対象: 各Repoの最新contentスナップショットの topics_json に小文字一致でマッチしたもの。 算出方法