#mixture-of-experts

Faithful from-scratch PyTorch reproduction of OpenAI's GPT-OSS architecture (sliding/full attention alternation, learned attention sinks, YaRN 128K, top-2-of-8 MoE), scaled to Chinchilla-optimal 502M total / 247M active training on a single A100 80GB

A 1.32B-parameter Mixture-of-Experts LLM trained from scratch in PyTorch - custom tokenizer, MLA attention, 18B-token pretraining, SFT and DPO. No pretrained weights, no HuggingFace Trainer.

Production-oriented Kimi K3 inference control plane with checkpoint release gates, MoE capacity planning, OpenAI gateway, NVL72 deployment, benchmarks, and observability.

Local-first agentic coding harness — Go + Svelte + llama.cpp, engineered for small local MoE models on consumer hardware. cerveau.sh