#tokenizer

GitHub repositories that have self-applied the topic "tokenizer" — a creator-tagged metadata that surfaces how AI projects describe themselves.

tagged repos

top 3 stars

with tool sigs

shown

REPOS Repos for #tokenizer (top 3 by stars)

Arabic Costs Double on AI — why every Arabic-script language (Darija, MSA, Persian, Urdu) pays a 2x token tax on ChatGPT, Claude & Gemini.

Jupyter Notebook ★ 2 AI 70

Mac-local LLM-from-scratch lab for building and evaluating a small decoder-only language model on Apple Silicon.

Python ★ 0 AI 90 2 sig

Estimate LLM token counts and $ cost for any text, file, or directory across OpenAI, Anthropic, and Google models.

Python ★ 0 AI 40