#evaluation

Release repository for agent benchmark evidence-reporting artifacts and reproduction workflows.

Provider-agnostic coding-agent harness + DeepEval evaluation suite

Long-horizon reliability benchmark for industrial edge agents - do self-improving agents get better or corrupt over 1000+ episodes on constrained hardware?

Software-only embodied AI evaluation stack for benchmark trajectories, ROS-compatible logs, replay, failure analysis, and comparison reports.

Case study: unified observability + evaluation for automation and conversational AI agents. Versioned error catalog, multi-signal confidence, cost meters.

Open-source multilingual evaluation toolkit for conversational AI agents.

Fine-tuning an open 7B LLM into a GST question-answering assistant, with a reproducible eval harness measuring the gain over the base model.

Author, validate, preview, calibrate, diff, and export criterion-level AI evaluation rubrics.

An AI leaderboard scoring chatbots on how much they bullshit vs. give honest answers to unanswerable questions. Works with paid APIs and free/local models.

AI knowledge base & RAG QA system with gold-label retrieval evaluation (Spring Boot 3 + Vue 3): hybrid keyword/vector search, RRF fusion, rerank, four-way Hit@3/MRR comparison

Supplementary materials for Reversibility as a Safety Gate for Agentic Actions

Evaluation harness for coding agents - repeated runs, pass@k, bootstrap/McNemar significance, tamper detection, cost. Single vs Planner-Coder-Reviewer multi-agent on CLOVA.

Read-only first-pass incident triage on the Claude Agent SDK + MCP: classifies an alert, retrieves the matching runbook, and proposes a cited first response — or abstains to a human. Measured over four runs: 100% abstention with 0 missed escalations, ~$0.014 per incident.

Enterprise AI Copilot Platform — modular, model-agnostic enterprise AI copilots with RAG, multi-agent orchestration, MCP, and evaluation-gated CI/CD. Module 1: ERP Sync Reconciliation Copilot.

How to know if your AI is actually working

Mac-local LLM-from-scratch lab for building and evaluating a small decoder-only language model on Apple Silicon.

A CLI developer tool for running custom evaluation suites to OpenAI-compatible APIs (MiMo, Claude, GPT, etc.), outputting Markdown research reports + JSON traces. Useful for developers who want to benchmark models before production. Suitable for Data/Research + Dev tools, and a natural fit using the MiMo API + Hermes Agent workflow.

Autonomous UI/game-testing agent: vision-language reasoning, browser control, action planning, failure recovery, and evaluation.

Evaluate Claude Code skills with cost-weighted A/B performance testing for Claude Code skills — one skill, or several combined. It measures real token cost and blind-judged output quality from live runs.

Open-source evaluation framework for AI customer support agents. 4-dimension LLM-as-judge (tone, accuracy, compliance, resolution), provider-agnostic, configurable rubric. Dogfooded on GlamShelf Twin.

Multilingual, RAGAS-compatible RAG evaluation with language-native judge prompts and Claude as the judge.

A thin, CI-gated evaluation gate for RAG & LLM systems — golden set, band-based pass/warn/fail gates, LLM-judge or heuristic scorers. pip install raggate

What: local LLM bakeoff harness (coding/vision/tools/agentic). For: objective model comparisons. How: ./setup.sh then ZWELL_BASE=… python bench_zwell.py --tag …

Open, always-on leaderboard + CI gate for autonomous coding agents — every patch sandboxed, every run traced, every regression fails the build. $0 stack.

An MCP server that serves PostHog's JS/Web SDK docs as agent-optimized context, with an eval harness that measures it against a naive baseline.

LLM red-team evaluation harness — prompt injection, refusal, leakage and staleness, with cross-judge validation and attack-corpus audits.