HITL Kit

95% of enterprise AI pilots fail, not because the models are bad, but because we measure the wrong thing. Despite $30–40 billion in investment, 95% of organizations achieve zero measurable return from generative AI initiatives (Challapally et al., 2025). The paper argues this failure crisis is at root a measurement crisis.

Current benchmarks ask one question: can the model complete this task autonomously? Deployment asks another: does it respect the user's authority, preserve their agency, and make them better over time? The benchmark-to-deployment gap is the distance between those two questions, and every component in this kit exists because something falls into that gap.

The alternative is Assist-Not-Complete: evaluate AI on whether it assists humans without displacing them, not on whether it can finish the task alone. The paper synthesizes benchmark-saturation research (Ott et al., 2022, a large fraction of 3,765 mapped benchmarks quickly trending toward near-saturation), cognitive-debt findings from AI-assisted learning (Kosmyna et al., 2025), scaffolding theory (Dhillon et al., 2024), and uncertainty quantification (Liu et al., 2025) into that single reframing.

The component library is not a grab-bag of agent UI widgets. Each primitive is the physical embodiment of a specific claim from the paper. The MiniTrace instantiates the supporting-facts requirement from HotpotQA (Yang et al., 2018): an answer without its evidence trail is unverifiable, so the trace rides along. The AI Generation Scale operationalises Dhillon et al.'s scaffolding principle, their CHI study found sentence-level suggestions reduced writing quality while paragraph-level scaffolds improved it, so the scale makes the level of AI involvement a first-class, visible control. The Interrupt Card is the agency-preservation boundary: the moment the system pauses and the human decides.

That traceability is the design discipline for the whole kit: if a primitive can't be tied to a claim the paper defends, it doesn't ship. The paper is the spec.

The architecture is a protocol with adapters. @hitl-kit/core defines Zod event schemas, a framework-agnostic wire format for human-in-the-loop moments (approvals, interrupts, traces, generation-scale changes). @hitl-kit/react ships HitlEventRenderer, a typed dispatcher that maps validated events to UI primitives. @hitl-kit/gates adds composable decision gates: confidence, cost, scope, approval-chain, rate-limit, pure functions that wrap any adapter's emit point and decide allow, deny, or escalate.

Three adapters carry the protocol into real agent stacks: LangGraph (emit events from interrupt() nodes, resume with typed Commands), the Vercel AI SDK (typed tool() wrappers), and MCP (an MCP server exposing the primitive event kinds as tools, so any MCP-aware client can emit schema-validated HITL events). The paper becomes the protocol; the protocol becomes the platform.

The UI primitives live in one package, @hitl-kit/ui, and distribute two ways from that one source: as the package, or the shadcn way, copy, paste, own. The registry JSON the CLI resolves into your codebase is generated from the package source and drift-checked, so the copy you install and the copy the site demonstrates cannot disagree. Every primitive takes its core event as props and reports the human's decision through one typed onAction, whose vocabulary is shared: approve, reject, can't tell, undo. The kit was originally extracted from Agatha, a research-agent workspace. The primitives earned their shapes in a real product before they were generalized.