A measurement instrument for multi-step research agents.

Existing agent evals — MMLU, SWE-bench, GAIA, AgentBench — measure autonomous task completion on synthetic, single-turn, closed-form tasks. That answers "can the model finish this problem on its own?", which is a fine question, but it is not the question a research, coding, or support agent has to answer in deployment. In deployment the agent runs a multi-step workflow with a real person at the other end, and the interesting failure modes are step-level, not output-level.

A research workflow is five to nine steps. Looping during canvas creation, regenerating notes that drift from their sources, refusing to push back when the cited papers disagree with the user's thesis — these failures live across steps. A score on the final output doesn't see them.

And tool selection — the most honest diagnostic there is — is almost never measured. Whether the agent reached for academic_search or invented a citation, read_pdf or paraphrased from a hallucinated abstract, says more than whether the prose reads well. eval-kit makes per-step expected-tools-versus-actual-calls a first-class assertion, with a strict, subset, or any matching mode per step.

LLM-as-judge shares the blind spots of the agent it grades — both were trained against similar objectives, so the judge rationalizes the failures the agent makes for the same reasons the agent makes them: calibration drift, agency erosion, fabricated grounding. eval-kit's answer is a structured human rubric: every reviewed step gets a 0–3 golden-truth score plus 0–3 on the dimensions in scope — explainability, agency preservation, long-term capability, calibration, and collaborative performance — the same axes the Assist-Not-Complete paper argues for.

The LLM is allowed exactly one role: optional pre-fill that a human accepts or overrides, flagged with pre_filled: true on every score it touches. Any human edit flips the flag. If LLM-judge ever became the default scorer, the framework would lose its reason to exist.

Distractor tasks invert the usual assumption entirely. Suite tasks marked is_distraction — future-dated papers, unverifiable claims, out-of-scope asks — are pass-when-the-agent-pushes-back, not pass-when-it-tries. The framework scores the refusal, because in deployment, catching the trap is the competent behavior.

You describe a multi-step workflow as YAML — Zod schemas are the source of truth for every shape — and run any agent against it through a small adapter contract (anthropic with tool-use and prompt caching, openai function-calling, a generic http adapter, or a deterministic mock). Tier-1 auto-scoring runs at trace time: tool-match and distraction-caught, computed mechanically. Tier-3 active triage then ranks the review inbox by where human attention pays off — low-confidence drafts and auto-score disagreements float up.

The reviewer sits in a keyboard-first local dashboard — built on @hitl-kit/react primitives, dogfooding the sibling kit — scores step-by-step against golden truths, and the scored run becomes a JSON file on disk. A deterministic replay harness diffs runs across model versions; a CI gate exits non-zero on auto-scored regressions; an exporter turns approved scores into SFT or DPO training JSONL.

Everything is file-based, single-user, and local by design — no auth, no multi-tenancy, no cloud. And aggregate scores are internal signal, not leaderboard fodder: every task in the seed suites is ported from observed real usage, and the framework's argument is that step-level human judgment is the eval signal.