#benchmark

A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

Lightweight agent that pulls and runs LLM benchmarks from Protorikis Bench

Live, open-source benchmark for comparing AI coding agents on real GitHub issues

Release repository for agent benchmark evidence-reporting artifacts and reproduction workflows.

Pre-registered benchmark: HTML dialect + whole-tree rewrite (barkup) vs JSON, granular mutation tools, and two patch dialects, for LLM agents editing typed trees. 9,600 scored runs across 4 models. Findings in REPORT.md.

Long-horizon reliability benchmark for industrial edge agents - do self-improving agents get better or corrupt over 1000+ episodes on constrained hardware?

Benchmark for AI agents proving multi-tenant SaaS authorization bugs

An evidence-hostile, container-isolated benchmark and behavior analysis platform for long-horizon AI software agents.

Catch visible-but-unbound coding-agent handoffs: CLI + GitHub Action with proof, policy gates, SARIF, HTML, and benchmark cards.

Reproducible token-throughput benchmark for OpenAI-compatible LLM servers, tuned for NVIDIA Spark and GB10 inference.

Public, reproducible benchmark of CLI coding agents (Claude Code, Codex, Aider) on SWE-bench Verified. Live leaderboard: https://ttxs69.github.io/coding-agent-eval/

An AI leaderboard scoring chatbots on how much they bullshit vs. give honest answers to unanswerable questions. Works with paid APIs and free/local models.

Supplementary materials for Reversibility as a Safety Gate for Agentic Actions

Trustworthy AI evaluation tool for testing chatbot safety, reliability, hallucination behavior, privacy risk, and instruction-following quality.

RF front-end analysis engine and a 60-problem LLM benchmark. Grounded small model ≈ ungrounded frontier model.

Integrity scenarios for agent runtimes, distilled from real fleet incidents — deterministic fault-injection checks against real SDKs

Measuring and verifying LLM decision agents under orbital dynamics, delay, and intermittency: a reproducible testbed, failure-mode taxonomy, and in-chain verification.

Synthetic benchmark for role-limited multi-agent deliberation in reservoir construction decision support

Evaluation harness for coding agents - repeated runs, pass@k, bootstrap/McNemar significance, tamper detection, cost. Single vs Planner-Coder-Reviewer multi-agent on CLOVA.

Production-ready Transformer implementation from scratch with multi-scale benchmarking and Arabic language support

Professional multi-agent benchmark library for evaluating LLMs in 23 strategy games — grid, board, social deduction, cards & game theory

Benchmark local coding LLMs on an OpenAI-compatible endpoint, then keep the serving config that won. Hidden executable tests, reference-validated tasks, mermaid reports.

A CLI developer tool for running custom evaluation suites to OpenAI-compatible APIs (MiMo, Claude, GPT, etc.), outputting Markdown research reports + JSON traces. Useful for developers who want to benchmark models before production. Suitable for Data/Research + Dev tools, and a natural fit using the MiMo API + Hermes Agent workflow.

Reproducible OpenAI-compatible AI API gateway benchmark with API NODE defaults, latency metrics, success rate, and GitHub Actions results.

Benchmarking schema-valid false tool observations and defense baselines for tool-using LLM agents.

Can your AI act as a fiduciary inside a regulated bank? A deployment-readiness benchmark that drops LLMs into a synthetic regulated organization as an employee and audits the behavior.

Do your agent pipeline's own checks catch planted faults? Measured on LangGraph, CrewAI and AutoGen: median 16.7%. One prompt-level change takes it to 55.0%. Pre-registered spec, Apache-2.0 harness, every raw trace published.

What: local LLM bakeoff harness (coding/vision/tools/agentic). For: objective model comparisons. How: ./setup.sh then ZWELL_BASE=… python bench_zwell.py --tag …

Reproducible benchmark for LLM-based structured extraction from documents. Compare Claude / GPT / Gemini / open-weight on the same task with cost + latency tracking.

A latency-fair LLM-vs-LLM fighting game benchmark. The engine blocks until an agent responds, so inference speed can't win a match — what's scarce is a per-match token bank that drains as a model thinks.

Fully-local terminal AI agent for Ollama. 34 tools, MCP, local RAG, deterministic hallucination checks. No cloud, no API keys, nothing leaves your machine.

LLM red-team evaluation harness — prompt injection, refusal, leakage and staleness, with cross-judge validation and attack-corpus audits.

Does following a negated instruction come free with the positive one? A reproducible eval — two pre-registered nulls for frontier Claude.

A Kubernetes lab that reproduces real production incidents on a live, instrumented microservice stack — for testing root-cause-analysis tools and agents.