#llm-evaluation

A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

CI for your coding-agent setup (CLAUDE.md, skills, hooks): pinned baselines, a canary on every Claude Code release, model and version matrix, context-cost drift, self-diagnosing reports, repair. Claude Code, Codex, Gemini.

Open-source Agent Skill for Claude Code that makes Claude reason with explicit probabilities, base rates, and falsifiers. Distilled from 51 books; first in 18/18 blind passes vs baseline Claude and GPT-5.5, and benchmarked against 14 rival skills and prompt techniques with full cost telemetry (round 5).

An open-source debugger for voice agent workflows

EvalBot — local-first chatbot security & quality evaluation (FastAPI + Next.js). Evaluate chatbot answers against your own docs & guidelines with ML/NLP + AI-judge scoring. Apache-2.0.

Practical agent skills and examples for building with Jev. API setup, routing, ranking, and evidence checks.

Public, reproducible benchmark of CLI coding agents (Claude Code, Codex, Aider) on SWE-bench Verified. Live leaderboard: https://ttxs69.github.io/coding-agent-eval/

Agentic evaluation of misalignment, scheming and evaluation awareness in LLM agents. Scenario text encoded at rest to resist training-data contamination.

Case study: unified observability + evaluation for automation and conversational AI agents. Versioned error catalog, multi-signal confidence, cost meters.

Integrity scenarios for agent runtimes, distilled from real fleet incidents — deterministic fault-injection checks against real SDKs

Cost-aware agentic RAG with iterative multi-hop retrieval, benchmarked against 4 baselines using an independent LLM judge on MultiHop-RAG.

Production-shaped, self-evolving AI agent operations platform: immutable agents, bounded loops and graphs, HarnessDev evolution, evaluations, observability, approvals, guardrails, and Cloudflare D1 deployment.

Production-inspired AI observability lab: a traced and evaluated fictional banking RAG assistant with OpenTelemetry, Phoenix, prompt experiments, guardrails, and optional Dynatrace fan-out.

Agent skill that writes clear, unambiguous docs, READMEs, error messages, and prompts in ASD-STE100 Simplified Technical English. Works with Claude Code, Codex, Cursor, and any SKILL.md agent.

An AI leaderboard scoring chatbots on how much they bullshit vs. give honest answers to unanswerable questions. Works with paid APIs and free/local models.

Tests exact agent interventions—skills, prompts, models, tools, and workflows—and publishes what changed, what failed, and what decision follows.

Evidence-first evaluation orchestration for models, RAG pipelines, agents, APIs, and complete AI systems.

How often do you correct your coding agent, and on what kind of work? Measures it from your Claude Code transcripts, drafts CLAUDE.md rules from your recurring corrections, and exports them as preference pairs.

Operations agent for a letting agency's maintenance desk: reads each case history, decides the next step, escalates with a brief, and works by voice (ElevenLabs) and SMS.

Audit-grade multi-agent orchestration. Fabrication-resistance enforced as a validated contract, not a hope.

Empirical study: do LLM-generated features improve tabular ML models? 10 LLMs, 3 datasets, 3 algorithms, 219 experiments.

Same agent action, safe or dangerous by context: email team vs investor; delete scratch vs prod config. A safety layer routes each action: auto, confirm, escalate, block. A reading model auto-executes 0% of unsafe actions vs a baseline's 38% (McNemar p<0.001). Plus a runtime loop: an agent earns or loses autonomy with each counterparty over time.

Hierarchical Bayesian evaluation of LLM/VLM outputs — posterior credible intervals & partial pooling over any eval's item-level scores (Bambi/PyMC).

Estimand-indexed audit of LLM synthetic respondents against CGSS benchmarks

Evidence-first, reproducible quality benchmarks for AI agents.

RF front-end analysis engine and a 60-problem LLM benchmark. Grounded small model ≈ ungrounded frontier model.

A privacy-aware multi-agent career management system with structured source governance, human approval workflows, and LLM output evaluation.

Production-style RAG operations console with grounded retrieval, document ingestion, evaluation, cost analysis, and bias-aware LLM judging.

Platform for versioning, evaluating, comparing, and governing AI agent skills with deterministic and model-based evals.

Three AI-engineering services in Java 21: tool calling, hybrid RAG with verified citations, and an offline-first eval harness. Built with Claude, with the prompt workflow documented.

Offline evaluation and CI regression gating for tool-calling LLM agents

Reproducible Ukrainian-language evaluation of language models and agents, with optional reference tools.

Protocol, prompt, rubric, data and session-log scripts for a six-agent, one-feature coding eval (Claude Code, ZCode, Codex CLI)

Self-healing AI reliability platform: LangGraph multi-agents + MCP tool execution + RAG-grounded Gemini evidence to detect, diagnose, and safely recover from LLM/RAG/agent failures — with human-in-the-loop approval and SLA-verified recovery.

A governed text-to-SQL analyst agent on BigQuery: Claude Agent SDK, warehouse-enforced permissions, dbt governance, lineage, OpenTelemetry, and a leak-focused eval suite.

A provable ceiling on LLM routing, voting and mixture-of-agents, measured on 67 models: code, audit and one-command reproduction

Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.

Engineering deterministic, production-grade systems around non-deterministic LLMs — FSM, durable execution, retries, DAGs, agent runtimes, model routing, edge inference, RAG, memory, multi-agent orchestration, security, and observability. 14 runnable proof-of-concept phases.

Reproducible retail inventory and pricing simulation using Kaggle data and RetailBench, with shock experiments, notebooks, validity tests, and baseline results.

How I validate LLM agents in regulated financial services: from regulation, the NIST AI RMF and its Generative AI Profile to research-based test design, with illustrations on simulated outcomes.