#llm-inference

An agent harness that compiles a model into one provably-correct, self-retargeting CUDA megakernel and self-tunes it past cuBLAS at batch-1 LLM decode, paper: https://arxiv.org/abs/2606.09682

A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

Agentic Runtime

ROCmFPX llama.cpp fork for Windows 🏆 — native build, headless OpenAI-compatible server & benchmarks. Tested on AMD Strix Halo (gfx1151), runs on other GPUs too.

基于 llama.cpp 的跨平台本地大模型客户端：一键调优 GGUF 模型，多模型共享一个 OpenAI 兼容端点，内置模型下载、本地聊天与实时监控。Windows / Linux / Android。| A friendly cross-platform local-LLM client powered by llama.cpp — one-click GGUF auto-tuning, all your models behind one OpenAI-compatible endpoint, with built-in downloads, local chat and live monitoring. Windows / Linux / Android.

caix — native Apple Core AI inference server for Apple silicon (beta): OpenAI/Anthropic API, dashboard, streaming chat with tools/skills/MCP, MTP speculative decoding.

Kvasir — control plane for large GGUF models across GPUs/machines; expert-sharded MoE swarm, non-custodial KVR wallet + mobile nodes, OpenAI/Anthropic gateways.

Agent Skills and an MCP server for GPU performance profiling, benchmarking, optimization, and reporting, with an inference focus.

🏆 Ultralight AI Coding Agent 2026 — Blazing CLI, Tiny Codebase, Full Power Free

Docker Compose stack for serving a local, OpenAI-compatible LLM (vLLM on Intel XPU) on an Intel Arc Pro B60 GPU — reproducible config with operator and developer docs.

Run Qwen3.8-27B fully offline on an Apple Silicon Mac (MLX, 4/5/8-bit) and use it as the local backend for Claude Code. Uncensored/abliterated build supported. No API key, no network. Built by yempik.

Demonstrating an attribution gap in LLM inference telemetry: per-request tracing records the symptom of KV-cache contention and not the cause

后悔承诺门 · 试验台（探针项目，非产品）：Transformer 缺一条「写通道」。第一步已实现——零后训练 / 零厂商协作 / 不碰 KV 的 harness，测四项外部强制机制（参数级风险分级、尾部强制自审、独立审核 Agent、射程锁上下文编辑）及 A–H 组合的离线效果；第二步已规划——协同进化：KV 局部编辑、推理轨迹纠错、后训练。每一步都给出可操作结论，包括证伪。

Production-oriented Kimi K3 inference control plane with checkpoint release gates, MoE capacity planning, OpenAI gateway, NVL72 deployment, benchmarks, and observability.

Run 35B mixture-of-experts LLMs on a 16 GB AMD Radeon GPU. A Rust inference engine for ROCm on Windows: reads the GGUF, solves weight placement across VRAM and system RAM, tunes by measurement, compacts context, and serves an OpenAI-compatible API with a web console. No Ollama, no daemon.

A tool that transforms audio or video files into text transcripts and generates concise meeting minutes. Stay organized and efficient in your meetings, and get ready for Phase 2 where we'll be open for contributions to enable real-time meeting transcription!

Not a leaderboard — a way to pick your daily driver. Benchmark model + quant + serving flags + thinking mode on your own hardware, then install the config that won. Code-gen and agentic tool-calling suites, hidden executable tests, mermaid reports.

Educational implementation and visualization of autoregressive, DFlash-inspired, and DFlash2-inspired LLM decoding with a tiny Transformer benchmark.

LLM inference

From-scratch LLM inference engine — continuous batching, paged KV cache, prefix caching, hand-written Triton paged attention, CUDA graphs, OpenAI-compatible API. Benchmarked against vLLM on an A100.

Serve DeepSeek and Qwen, run an on-demand model library, and fine-tune with Unsloth QLoRA on 2x NVIDIA DGX Spark. TP=2 vLLM lanes behind one OpenAI-compatible endpoint for OpenCode, Cursor, and Hermes. Honest benchmarks, committed artifacts.

JavaScript/TypeScript SDK for VeloxQuant-MLX: hardware-aware memory optimization and OpenAI-compatible local inference on Apple Silicon.

Zero-config orchestrator for distributed LLM inference on your LAN — splits a model across nodes by layer (pipeline parallelism) behind one OpenAI-compatible endpoint. Open-core (MIT).