The paper
perspective paper·the argument this feed measures
The AI failure crisis is at root a measurement crisis. The paper synthesizes benchmark science, cognitive neuroscience, uncertainty quantification, and enterprise deployment data to argue for Assist-Not-Complete: evaluate AI on whether it assists humans without displacing them, not on whether it can finish the task alone.
2026-07-21·essay
A reframing of what the akaOSS measurement family actually measures: not task completion, but the gate, the moment an agent hands control back to a human. Gates come in two kinds that must never share a metric, autonomous benchmarks structurally penalize both, and a gate over unverifiable output is oversight theater. Regulation now mandates effective human oversight; nothing measures whether oversight is effective. That gap is the research program.
gateshuman-in-the-loopeval-kitHITL KitEU AI ActmeasurementAssist-Not-Complete
2026-07-18·essay
Before anyone reads your test suite, they run an implicit checklist: does the published package install, do the README examples compile, does the API change without warning. We built those checks as CI harnesses across three of our packages and ran them on ourselves. First run: one package shipped no type declarations at all, and three README examples had never compiled.
testingpublish smokeREADME-runsAPI surfaceeval-kitHITL Kittag-kit
2026-07-18·essay
When an AI coding agent became a real part of a Blender → glTF game pipeline — writing the export scripts, running them, rearranging the shipped files — the speed-up came with a blind spot: no picture of what it was actually doing. Hologram is the pattern that closed the gap: one append-only event log, a live dashboard beside the assets, and the same pipeline handed back to the agent as a small non-destructive MCP surface.
HologramMCPBlenderglTFobservabilityread-only by design
2026-07-18·essay
LLM-as-judge shares the blind spots of the agent it grades — both were trained against similar objectives, so the judge rationalizes the same failures. eval-kit's answer is a measurement instrument built around step-level human judgment: per-step tool-match assertions, distractor tasks that score the refusal, and a five-dimension 0–3 rubric the judge never touches by default.
eval-kitLLM-as-judge5-dim rubricdistractor taskstool_matchAssist-Not-Complete
2026-07-18·finding
Findings here are produced by running eval-kit suites, displayed with HITL Kit components, and scored by humans on a five-dimension rubric — never by an LLM judge. Every entry links its experiment folder and checked-in run JSON, and none of it is leaderboard material.
Assist-Not-Complete5-dim rubricdistractor taskseval-kitHITL Kittag-kit