#inference

Predictive GPU memory framework for AI inference workflows

Local MLX tuned models through OpenAI compatible LLM, image generation and audio inference on Apple Silicon — Rust + PyO3 + MLX

OpenAI- and Anthropic-compatible AI gateway. Capability-based routing. Streaming. BYOK passthrough. No key custody.

Production-oriented Kimi K3 inference control plane with checkpoint release gates, MoE capacity planning, OpenAI gateway, NVL72 deployment, benchmarks, and observability.

GPU profiling and optimization SKILLS with bundled MCP server

Load test LLM inference servers with k6. TTFT, ITL, TPOT, goodput, cost, and energy metrics for any OpenAI-compatible server. Ships to Prometheus and Grafana.

MCP server for the GONKA network: run cheap LLM inference through the server (free trial or your own key), get multi-model second opinions, and compare live prices — for any AI agent.

Production-grade distributed ML inference platform — FastAPI gateway, Redis-backed worker pool with dynamic batching, WebSocket result streaming, and a live ops dashboard. Serves YOLO, Whisper, RAG and text models, with an MCP agent server and streaming chat.

Provider-neutral access & supply layer for open AI: run open models locally (SIN) and route paid, private, verifiable inference across decentralized providers (SIP-AI). DecentralizeAI hackathon entry.

An LLM inference engine written from scratch in Python and C++/Metal — tokenizer, forward pass, KV cache, and INT4 quantization implemented by hand. No PyTorch, no llama.cpp.