Petrov’s Loop · AISA CyberCon 2026 companion
Before you sign.
The working artifacts from the talk, in one place: the three questions that open the work, the four that sign it off, the three interventions you can require starting Monday, and the matrix that decides how much autonomy a system has earned.
Four questions that sign it off From the slide
Before you put your name on an AI-assisted security system. If you can answer all four, you can sign. If you can’t, what you’re being asked to approve isn’t ready — no matter how good the demo was.
Those three questions open the work. These four sign it off. And every one of them is the same question wearing different clothes: where, in this system, does a human being return to the question?
Has anyone tried to break this on purpose — and can you show me the evidence?
Has anyone actually tried to break this on purpose before it shipped — and can you show me the evidence?
Evidence, not assurance. The adversary this system will meet moves at machine speed; “we tested it” is not the same claim as “we attacked it, and here is what happened.”
The evidence.
Mine: 30 attempts across three attack types. Zero fabricated IDs after the fix. Bounded below 10%.
Which of our controls assumed a human was the one acting?
Which of our existing controls were designed assuming a human was acting, and have we actually rechecked whether they still apply now that a machine is?
Most controls were written for a human actor. They do not fail loudly when a non-human one shows up — the gate is still standing, and it simply has nothing to say.
Who is accountable — and could they actually have stopped it?
Who is the named person accountable when this fails, and could they actually have stopped it, or only absorbed the blame afterward?
A named person, with the information, the time and the authority to decide differently. Responsibility without authority is not accountability; it is a liability sponge.
Measure it: refusal rate — how many times this quarter has that named person declined to approve something?
Mine: fifty reviewer verdicts, six refusals. Twelve per cent. A figure, not a target.
It is already in your logs and costs nothing to collect, which is precisely why nobody collects it. A refusal rate of exactly zero, across every decision this quarter, is not evidence of a good system. It is evidence of an inert control.
What turns it off, who has that authority, and does it match the blast radius?
What would have to happen for us to turn this off, who has the authority to do it, and does that authority match how much damage this thing could actually do?
A stop that needs three approvals is not a stop. Match the authority to the blast radius, and decide it before go-live — not during the incident.
These four are an entry gate. They are not maintenance. Mine sit beside a debt ledger of five entries — each closed only by evidence, never by intention. Passing them once tells you the system was defensible on the day it shipped and nothing about the day it fails. A gate you only walk through once is a door you left open.
So they get paired with a governance debt ledger — every accepted shortcut, written down, with the name of whoever accepted it — and both get re-run on a cadence, not at launch.
Three questions that open the work
Ask these before anyone asks “how”.
- What are we actually building for?
- What does success look like?
- Would we even do this if it weren't for AI?
Governance is a control system
Five parts. Which one is your AI policy?
- ReferenceWhat you want. The policy.
- SensorWhat tells you what is actually happening.
- ControllerWhat compares the two and decides.
- ActuatorWhat can change it, or stop it.
- PlantThe thing being governed. Often bigger than the model.
A policy document is a reference. An organisation with an excellent AI policy, a quarterly risk register and no ability to stop a deployment has one part out of five: the reference, the part that does nothing on its own.
FM1 · Rational compliance blindness · a sensor failure
Every step within policy. The sensor cannot see it.
FM2 · Result-laundering · a controller failure
It checks the answer, not the process. It read like a review. It was not one.
FM3 · Complexity as camouflage · a plant failure
You governed the model, and the plant was bigger than the model.
That is the structural definition of governance theatre: rich in references, poor in actuators.
Three interventions you can require starting Monday
1 Build the loop in
Where a human return to the question is structurally required, not optional and not merely logged.
- RecommendThe system proposes. A human decides.Log it — and record whether the human ever disagrees.
- Execute, reversibleThe system acts. It can be undone.Require a named owner and an alert.
- Execute, irreversibleThe system acts. Nothing brings it back.A human return to the question is structurally required.
Two different failures, one identical hole: the gate that existed was not the gate that was needed.
2 Audit knowing-how
AI lets someone perform at the last stage without ever passing through the earlier ones. The output looks like expert judgement. Nothing underneath it is.
Ask the person who signed it what they ruled out, and why.
3 The three-engineer decision map
Give three engineers a week. Separately. Ask each to draw how the system actually decides: inputs, model, rules, where a human sits, where the action fires. Then put the three maps side by side, and look first at whether the human is in the same place on all three.
Three maps that agree: you can govern it. Three maps that don’t: you have a system you are hoping about.
The strongest argument against all of it
Adding a human to the loop is not automatically a control. A preregistered meta-analysis of 106 studies found human–AI combinations did, on average, worse than whichever of the two was better on its own, and the losses were concentrated in decision-making tasks. The same paper’s second finding is the design rule: they did outperform where the human was already better than the AI at that task.
Approve what you have no independent basis for judging and you have not built a control. You have built a signature.
Vaccaro, Almaatouq & Malone, “When combinations of humans and AI are useful: A systematic review and meta-analysis”, Nature Human Behaviour, 2024.
The autonomy matrix
Match the degree of autonomy to two things: how confident the signal is, and how big the blast radius is.
| Low blast radius | High blast radius | |
|---|---|---|
| High confidence | Full automation. Get the human out of the way — you’re wasting time you don’t have. | Pre-stage everything. One human clicks go. Target 60 seconds. |
| Low confidence | Automate and monitor. Review the patterns, not the instances. | Petrov’s chair. The loop is non-negotiable. This is where the named human lives. |
Automate the reversible. Human-gate the irreversible. And reversible means cheap to reverse and harmless if reversed wrongly — a credential you can restore in ten minutes, revoked in the middle of your payment batch, was never reversible.
The clock you cannot see
10 December 2026
New APP 1.7 commences. Where an organisation has arranged for a computer program to make, or do a thing substantially and directly related to making, a decision that could reasonably be expected to significantly affect the rights or interests of an individual, its privacy policy has to say so.
Privacy and Other Legislation Amendment Act 2024 (Cth). CPS 234 and CPS 230 are already live. Described, not advised: ask your own counsel before December.
The book
The long version of this talk
Decide Who Decides is almost ready. Signed copies post from Melbourne in early November. Register your interest, tell me you were in the room, and I’ll email you myself with an early-reader code. Orders close 31 October.
Register your interestRan these against a system and got stuck?
Tell me — that conversation is exactly why I gave the talk. Find me on LinkedIn.
