AI in government systems · OMB M-25-21
The question is not whether the model is impressive.
It is what happens when it is wrong.
Agencies are being asked to adopt AI and to manage its risk at the same time. The hard part is not the model. It is the seam between a probabilistic system and a system of record that has to be right, auditable and attributable.
We have been building on that seam since 2023, on our own production systems, with our own money at stake. Six patterns carry the weight.
- 01
Verbatim evidence or nothing
When a model extracts a fact from a document, it must return the quote. Code then checks that the quote is literally present in the source. A model can invent a citation; it cannot invent a quote that survives a string search. Fields that fail are dropped, not flagged and forgotten.
- 02
Confidence bands that route to people
High-confidence matches proceed. The uncertain band always goes to a human, because we measured what lives there: wrong people, confidently matched.
- 03
Read-only by construction
Where an agent needs data, it gets a client whose write methods throw. Not a policy, not a prompt. The capability does not exist.
- 04
Deny by default on irreversible actions
Mass notifications and production releases are blocked unless a human sets an explicit second key. We wrote these guards after our own near-misses, including production deployments that silently reverted work.
- 05
Named attribution on every decision
The decision record cannot exist without a named human attached. Enforced by the database, not by convention.
- 06
Cost and scope ceilings in code
Token accounting on every call since 2023. Hard spend limits that halt an autonomous run. Per-session budgets that return an error rather than an invoice.
Policy caught up with the practice
OMB M-25-21 asks agencies for human oversight, testing and evaluation, and risk management on high-impact AI. The NIST AI Risk Management Framework asks the same in its Manage function.
We can show you the code that implements it, on systems that were carrying money before the memo was written.