The paper
perspective paper·the argument this feed measures
The AI failure crisis is at root a measurement crisis. The paper synthesizes benchmark science, cognitive neuroscience, uncertainty quantification, and enterprise deployment data to argue for Assist-Not-Complete: evaluate AI on whether it assists humans without displacing them, not on whether it can finish the task alone.
2026-08-31·essay
№ 005 argued that no current benchmark can tell a rubber stamp from a genuine review. eval-kit's own gate schema could not either: `resolution: "approved"` is exactly the field that collapses the two. This note is about closing that gap without letting the instrument invent the thing it is measuring, and about two divergences between the published contract and the reference implementation that were recorded rather than resolved.
eval-kitgatesoversightEU AI Actprovenanceschemarelease notes
2026-08-11·essay
eval-kit's pitch narrowed from evaluating research agents to measuring authorization. The useful part was not the new sentence. It was discovering which parts of the codebase the sentence contradicted. The reframe broke six things, and each break marked a place where the old claim had been carrying weight the new one will not.
eval-kitgatesauthorizationrelease notesdesign systemsaccessibilityshadcnHITL Kit
2026-08-06·essay
We built four CI gates to check that a published package is trustworthy, then published a package whose npm page reads 'ERROR: No README data found!', with every gate green. The cause generalises past this bug: a checker that filters its own input list by file existence converts a missing artifact into an empty work-list, and an empty work-list into a pass. Shipping the fix then failed on a 404 that means four different things, which is the adjacent lesson.
testingcredibility stacknpm packagingdocumentationtag-kiteval-kitHITL Kitrelease engineering
2026-07-26·essay
The companion to the gate thesis: what happens when you build it into a domain where being wrong has a victim either way. Content review turns out to produce both gate kinds without being asked to, and the architecture that results has one property worth stating plainly. The machine may resolve a case alone only when the resolution is to do nothing.
gateshuman-in-the-loopcontent reviewauditHITL KitinertialAssist-Not-Complete
2026-07-21·essay
A reframing of what the akaOSS measurement family actually measures: not task completion, but the gate, the moment an agent hands control back to a human. Gates come in two kinds that must never share a metric, autonomous benchmarks structurally penalize both, and a gate over unverifiable output is oversight theater. Regulation now mandates effective human oversight; nothing measures whether oversight is effective. That gap is the research program.
gateshuman-in-the-loopeval-kitHITL KitEU AI ActmeasurementAssist-Not-Complete
2026-07-18·essay
Before anyone reads your test suite, they run an implicit checklist: does the published package install, do the README examples compile, does the API change without warning. We built those checks as CI harnesses across three of our packages and ran them on ourselves. First run: one package shipped no type declarations at all, and three README examples had never compiled.
testingpublish smokeREADME-runsAPI surfaceeval-kitHITL Kittag-kit
2026-07-18·essay
When an AI coding agent became a real part of a Blender → glTF game pipeline: writing the export scripts, running them, rearranging the shipped files. The speed-up came with a blind spot: no picture of what it was actually doing. Hologram is the pattern that closed the gap: one append-only event log, a live dashboard beside the assets, and the same pipeline handed back to the agent as a small non-destructive MCP surface.
HologramMCPBlenderglTFobservabilityread-only by design
2026-07-18·essay
LLM-as-judge shares the blind spots of the agent it grades. Both were trained against similar objectives, so the judge rationalizes the same failures. eval-kit's answer is a measurement instrument built around step-level human judgment: per-step tool-match assertions, distractor tasks that score the refusal, and a five-dimension 0–3 rubric the judge never touches by default.
eval-kitLLM-as-judge5-dim rubricdistractor taskstool_matchAssist-Not-Complete
2026-07-18·finding
Findings here are produced by running eval-kit suites, displayed with HITL Kit components, and scored by humans on a five-dimension rubric, never by an LLM judge. Every entry links its experiment folder and checked-in run JSON, and none of it is leaderboard material.
Assist-Not-Complete5-dim rubricdistractor taskseval-kitHITL Kittag-kit