#llm-evaluation

A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

EvalBot — local-first chatbot security & quality evaluation (FastAPI + Next.js). Evaluate chatbot answers against your own docs & guidelines with ML/NLP + AI-judge scoring. Apache-2.0.

An open-source debugger for voice agent workflows

Open-source Agent Skill for Claude Code that makes Claude reason with explicit probabilities, base rates, and falsifiers. Distilled from 51 books; first in 18/18 blind passes vs baseline Claude and GPT-5.5, and benchmarked against 14 rival skills and prompt techniques with full cost telemetry (round 5).

Case study: unified observability + evaluation for automation and conversational AI agents. Versioned error catalog, multi-signal confidence, cost meters.

Public, reproducible benchmark of CLI coding agents (Claude Code, Codex, Aider) on SWE-bench Verified. Live leaderboard: https://ttxs69.github.io/coding-agent-eval/

Tests exact agent interventions—skills, prompts, models, tools, and workflows—and publishes what changed, what failed, and what decision follows.

An AI leaderboard scoring chatbots on how much they bullshit vs. give honest answers to unanswerable questions. Works with paid APIs and free/local models.

Production-style RAG operations console with grounded retrieval, document ingestion, evaluation, cost analysis, and bias-aware LLM judging.

A privacy-aware multi-agent career management system with structured source governance, human approval workflows, and LLM output evaluation.

Audit-grade multi-agent orchestration. Fabrication-resistance enforced as a validated contract, not a hope.

RF front-end analysis engine and a 60-problem LLM benchmark. Grounded small model ≈ ungrounded frontier model.

Evidence-first, reproducible quality benchmarks for AI agents.

Estimand-indexed audit of LLM synthetic respondents against CGSS benchmarks

Integrity scenarios for agent runtimes, distilled from real fleet incidents — deterministic fault-injection checks against real SDKs

Empirical study: do LLM-generated features improve tabular ML models? 10 LLMs, 3 datasets, 3 algorithms, 219 experiments.

Hierarchical Bayesian evaluation of LLM/VLM outputs — posterior credible intervals & partial pooling over any eval's item-level scores (Bambi/PyMC).

Same agent action, safe or dangerous by context: email team vs investor; delete scratch vs prod config. A safety layer routes each action: auto, confirm, escalate, block. A reading model auto-executes 0% of unsafe actions vs a baseline's 38% (McNemar p<0.001). Plus a runtime loop: an agent earns or loses autonomy with each counterparty over time.

Regression-diff eval harness for Anthropic tool-use agents that surfaces LLM-judge reasoning drift, not just pass/fail flips

Backend-agnostic black-box hallucination & RAG-faithfulness detection for LLMs — Probabilistic Certainty & Consistency (arXiv:2601.02574). Works on MLX / OpenAI / vLLM via token logprobs; no model internals, no training.

Framework open source para evaluar agentes LLM con function calling / tool use. Scoring por dimensión (tool accuracy, response quality, safety) + CLI, API y dashboard.

⚡ Chatbot Arena — battle LLMs side-by-side. Elo leaderboard, LLM-as-judge, PII redaction, guardrails, routing & distillation. 100% browser-based.

Controlled coding-agent harness ablations with auditable traces, reproducible evidence bundles, and SWE-bench interoperability.

Can your AI act as a fiduciary inside a regulated bank? A deployment-readiness benchmark that drops LLMs into a synthetic regulated organization as an employee and audits the behavior.

MCP server exposing multivon-eval + pdfhell as agent-callable tools. Drop into Claude Desktop, Cursor, Cline.

Evaluate Claude Code skills with cost-weighted A/B performance testing for Claude Code skills — one skill, or several combined. It measures real token cost and blind-judged output quality from live runs.

Do your agent pipeline's own checks catch planted faults? Measured on LangGraph, CrewAI and AutoGen: median 16.7%. One prompt-level change takes it to 55.0%. Pre-registered spec, Apache-2.0 harness, every raw trace published.

Open, always-on leaderboard + CI gate for autonomous coding agents — every patch sandboxed, every run traced, every regression fails the build. $0 stack.

Regression testing and automated optimization for LLM prompts. Run a YAML test suite, score outputs with an LLM judge, and auto-improve across 4 strategies. Works with any OpenAI-compatible API — including local models.

Evaluator-grounded autonomous prompt evolution. Genetic algorithms + runtime validation + meta-evolution to evolve prompts that generate better AI agent code. 150 generations, 862/1000 best score.

Orquestación de IA multiagente: un registro de agentes autogenerado + una evaluación a ciegas que demuestra que el enrutado acierta el agente correcto (27/27, sin regresiones)