AI assistants no longer just answer questions. Agentic assistants read and write files, call other software and act across many steps — and that autonomy multiplies the cost of a single invented fact. A fabricated figure in a chat is an annoyance. The same figure written into a document and sent out is a business quality and product quality incident, with serious implications.
AI models state things confidently that are not backed by any source — a hallucination — and on long, multi-source work they do it in ways their own checks do not catch: a court ruling cited for a holding it does not contain, a paraphrase presented as a verbatim quote, a significance test attributed to the wrong measure. The desktop platform has no built-in way to enforce checks at project level — found in our diagnostic of early 2026 — so the discipline has to travel inside the instructions the AI follows, and the decisive check has to come from outside the author: an independent review in a fresh context, with the sources, that signs for what it read.
A set of written procedures and automated checks, installed alongside the AI as one versioned bundle. Some change how it works before a fact is written: cite a source or say "I do not know". Others verify afterwards: a claim-by-claim check, an independent review that signs a hash-bound receipt, and tests that catch what no instruction can — such as code that imports software packages that do not exist. Many of the checks are built on established open-source and vendor tools rather than written from scratch; the system's own contribution is the procedures, the integration and the gates that keep them honest.
The system is a running process, not a one-off release. Every week it is compared against the latest developments in the field and its recommendations are re-verified before they become a plan. Every release passes automated gates, and a weak check, once found, becomes a new gate.
The work began as in-house research into how hallucinations and errors arise in an AI assistant’s work, written up in early 2026 as a playbook on hallucination control. It catalogued the failure modes: fabricated facts and citations; code importing software packages that do not exist; stale facts presented as current; and agentic hallucinations, where the assistant claims to have completed a step it never performed.
Two findings turned the playbook into a system. The cheapest controls the vendor itself recommends — allow "I don’t know", quote before answering, verify with citations, use only the supplied documents — are under-used in practice and had no packaged, reusable form. And a diagnostic in early 2026 established that the platform had no project-level hook layer to enforce anything, which is still the case at the present time. So the discipline has to live in instructions and procedures the assistant actually follows.
The work has moved with the field. Early measurements on short test suites showed large gaps between an unmanaged assistant and a managed one, and figures from those suites were published here. They are superseded: they measured short, single-answer questions, where the most recent run, with live web search enabled for auto-correction and validation by the baseline agent, found no difference in correctness to measure, and which is not the regime this work is for. The projects this system is built for — and the projects on this site — are long, multi-faceted and at the edge of what the tools can do, and there the failure mode remains highly evident. The system is built for that regime and applied there, on our own work, with the receipts to show for it.
Four layers, in the order they act. Procedures and assurance change how work is produced — they require every claim sourced and every unverified point marked. Control verifies what was produced, from outside the author. Improvement keeps all of it current.
Rules the AI follows before it writes a fact.
Written procedures the assistant is held to before a claim is produced. Every non-trivial claim carries its source and a verbatim quote. "I do not know" is preferred to a guess. Anything that cannot be verified is marked [UNVERIFIED] rather than smoothed over. A question to a human beats an assumption.
CITE OR ABSTAIN · QUOTE BEFORE ANSWERING · ASK, DO NOT GUESS
Making the rules apply even when nobody invokes them.
The process change itself — rules that apply before a fact is written, rather than checks run afterwards. It needs no external tooling, which is why it travels: the same discipline is carried as a versioned instruction block by every project, so a session that never calls a procedure by name still works under the defaults. A gate reports which project copies have fallen behind.
DEFAULTS THAT APPLY WHEN NOBODY REMEMBERS THEM
Checking the work after it is done — from outside the author, and checking the checks.
Three kinds of check. A claim-by-claim pass lists every assertion in a draft, writes one verification question per claim, answers each independently from the sources and reconciles. An independent reviewer works in a fresh, isolated context that never saw the work, judges it on eight dimensions and signs a receipt bound to the document’s hash — so a review that never ran, or a document edited after review, is detectable. Deterministic checks catch what no instruction can: that every imported package exists in its registry and is not listed as malicious; that a fact marked unverified stays marked in every document repeating it; that the assistant’s picture of the files matches the disk. The review step is itself calibrated against a document with planted errors, and its own early runs showed that a reviewer can be confidently wrong — so every finding is a candidate to adjudicate against the primary source, never a verdict, and the calibration record is kept with the system.
DETERMINISTIC CHECKS CATCH WHAT PROMPTING CANNOT
Keeping the system honest against a fast-moving field — and against itself.
Outward: a scheduled task runs every Monday and compares the system against the latest developments — the vendor’s releases and security advisories, the dependency stack, published research, and standards work at NIST and ISO/IEC — grading each finding adopt, trial, watch or note and tying it to the component it affects. Inward: each report is re-verified before it enters a plan, because carried-forward premises go stale; every recommendation is either carried or declined with a reason; and a weak check, once found, becomes a release gate. Those gates now check every release.
A RECOMMENDATION MAY BE DECLINED. IT MAY NOT BE LOST.
In addition to our own original code and tools, the system integrates established tools rather than re-inventing them — some run inside it, others it instructs the assistant to use — and the procedures that govern them pass the same release gates.
Maintainer and licence as stated on each project's own page, October 2026. These projects are independent of us and do not endorse this system.
Every checker reports on the same four-value scale, and "could not fully check" is kept strictly distinct from a pass. A check that silently degrades to a weaker check and reports success is worse than one that fails outright. Five separate false cleans have been found in this system’s own tooling, and every one of them looked like a 0. The project’s own summary: a check which cannot fail is not a check.
Every system is built for the project it governs. The first one was built for our own work, as one installable bundle of eighteen procedures and checks for an agentic AI assistant, built with the tools listed above — twenty-eight documented releases so far.
In its first form, on our own work, and developed release by release since.
Compared against new developments in the field. Each finding is graded adopt, trial, watch or note.
Automated gates check each release. A weak check, once found, becomes a new gate.
Each one is either carried out or declined with a stated reason, and a later audit compares report, plan and release.
Reported as practised. Reports are not produced for every scheduled week, and some gates depend on manual steps — when one of those is behind, it shows as a failed gate, and we report it.
Tell us what the AI is being asked to produce, and who relies on the result.
