#evaluation

Release repository for agent benchmark evidence-reporting artifacts and reproduction workflows.

Controlled coding-agent harness ablations with auditable traces, reproducible evidence bundles, and SWE-bench interoperability.

A reference implementation for learning and building AI evaluation systems.

Open research on local-first, model-agnostic long-term memory for LLM agents

Software-only embodied AI evaluation stack for benchmark trajectories, ROS-compatible logs, replay, failure analysis, and comparison reports.

Claude Code plugin marketplace for agent discipline, reusable skills, executable gates and evaluation.

Long-horizon reliability benchmark for industrial edge agents - do self-improving agents get better or corrupt over 1000+ episodes on constrained hardware?

Provider-agnostic coding-agent harness + DeepEval evaluation suite

Case study: unified observability + evaluation for automation and conversational AI agents. Versioned error catalog, multi-signal confidence, cost meters.

Fine-tuning an open 7B LLM into a GST question-answering assistant, with a reproducible eval harness measuring the gain over the base model.

Open-source multilingual evaluation toolkit for conversational AI agents.

LLM tool-calling data quality agent with read-only PostgreSQL, evidence guardrails, traces, evals, FastAPI, Docker, and CI-verified reports.

Author, validate, preview, calibrate, diff, and export criterion-level AI evaluation rubrics.

v0.8 preview · 2,288 multi-step agent trajectories · 513 hand-authored gold + 1,775 provenance-flagged augmented. A per-step benchmark that scores whether a verifier catches drift inside an agent's trajectory — not whether a prompt is harmful.

Java RAG knowledge base with controlled Multi-Agent repair: Tool Calling, HITL approval, versioned rollback, pgvector retrieval and reproducible evals.

An AI leaderboard scoring chatbots on how much they bullshit vs. give honest answers to unanswerable questions. Works with paid APIs and free/local models.

Claim-level provenance auditing for multi-step LLM agents: deterministic, post-hoc, and honest about what it cannot see.

A terminal AI agent that works on a codebase, container, browser or desktop, uses what it built, and judges its work against criteria written before it started. One agent loop, rubric-first evaluator, tiered memory, four-vendor routing, six benchmark harnesses.

CI/CD evaluation infrastructure for tool-using, multi-step AI agents

Evidence-first evaluation orchestration for models, RAG pipelines, agents, APIs, and complete AI systems.

Quantization, sparsity, and behavioral robustness of Qwen LLMs under edge hardware constraints

Supplementary materials for Reversibility as a Safety Gate for Agentic Actions

Qwen3 prompt baselines for tool selection and evaluation in the Huawei LLM Agent Competition.

How to know if your AI is actually working

Evaluation harness for coding agents - repeated runs, pass@k, bootstrap/McNemar significance, tamper detection, cost. Single vs Planner-Coder-Reviewer multi-agent on CLOVA.

Enterprise AI Copilot Platform — modular, model-agnostic enterprise AI copilots with RAG, multi-agent orchestration, MCP, and evaluation-gated CI/CD. Module 1: ERP Sync Reconciliation Copilot.

Read-only first-pass incident triage on the Claude Agent SDK + MCP: classifies an alert, retrieves the matching runbook, and proposes a cited first response — or abstains to a human. Measured over four runs: 100% abstention with 0 missed escalations, ~$0.014 per incident.

Production-grade evaluation framework for LLM systems: accuracy, hallucination, latency, consistency and cost metrics with async batch execution, WebSocket progress streaming and A/B comparison reports.

Multi-step agent safety compliance benchmarking - finds critical depth D* where LLM agents fail safety policies

MaixCAM 开发助手 —— 一个 RAG × Harness 教学项目。同一份语料与检索能力，用三种方式交付：从零仿写 agent（教学）／基于 DeepSeek Harness 做应用（落地）／Skill 包（普及）。

Five SKILL.md skills for AI coding agents — not how to build things, but how not to ship something broken while you build them. Each is backed by an A/B measurement against a current model. Five more were written and removed, because the same measurement found no effect.

Local-first, open-source platform for AI agents and business workflows with typed, calibrated decisions. Ready-to-run flows for finance, sales, IT and customer service; visual builder, human approvals, evaluations, RAG, MCP, REST API, SDK and CLI. One command on your own computer; Docker Compose for servers.

Project-local Codex skill for agent tournaments and validated, reusable playbooks.

Receipt extraction agent that validates its own output and knows when it might be wrong. Model-agnostic (Claude, Ollama, SGLang, vLLM), with guardrails and a full evaluation.

A CLI developer tool for running custom evaluation suites to OpenAI-compatible APIs (MiMo, Claude, GPT, etc.), outputting Markdown research reports + JSON traces. Useful for developers who want to benchmark models before production. Suitable for Data/Research + Dev tools, and a natural fit using the MiMo API + Hermes Agent workflow.

Mac-local LLM-from-scratch lab for building and evaluating a small decoder-only language model on Apple Silicon.

Provider-neutral evaluation runner for command-line AI agents.

Keyword-based eval runner with latency tracking

LLM red-team evaluation harness — prompt injection, refusal, leakage and staleness, with cross-judge validation and attack-corpus audits.

Open, always-on leaderboard + CI gate for autonomous coding agents — every patch sandboxed, every run traced, every regression fails the build. $0 stack.