Tools
Control Mesh
Writing
Book
About

Zero to Production · AISA CyberCon 2026 companion

Go break something.

Everything from the talk, in one place: the four layers, the four ways they broke, the question that would have caught each one, and the tools, live and free. The most useful thing you can do with them is try to break them, then tell me how.

The architecture From the slide

Four boxes. Three of them break somewhere specific. The fourth is the fix.

  1. 01The model

    Four models, Llama 3.1 8B and 3.3 70B, in production to March 2026. LoRA on Apple Silicon, 2,084 training examples.

    Breaks at: the training data

  2. 02The wrapper

    Auth, session isolation, a scope guard, output validation, version pins.

    Breaks at: the boundary

  3. 03Retrieval

    Three passes over a verified control table. A control ID never originates in the model.

    This is the fix. Not the model.

  4. 04The evaluation

    Golden datasets, regression detection, an append-only ledger of 8,693 entries.

    Breaks at: result-laundering

The model is just the part that talks. The product is the constraints around it.

Four ways it broke, and the question that catches each

Run the four questions against any AI feature in your organisation. If you can’t get a confident answer to all four, you have an AI feature in production that your SDLC didn’t secure.

  1. FM1 · The training data

    Who reviewed the training data as a security artifact?

    Not the code — the data. Whatever entered the fine-tuning set is what the model believes about security. Against the live tool: one reply carried six real control IDs with six wrong names. Not a crash, not a flag: one quiet, plausible, well-formatted wrong belief.

    A false equivalence looks right. Watch the table, not just the code.

  2. FM2 · The wrapper under pressure

    Has anyone actually tried to break the wrapper?

    The model enforces nothing. Auth, session isolation, rate limiting and output validation all live in the wrapper, and the wrapper is what breaks first. Neither boundary failure was an intelligence problem.

    Both were authority problems. Separate what reads untrusted input from what holds authority.

  3. FM3 · Result-laundering

    Does your evaluation prove the process, or only that the output looked right?

    Right output, wrong process: an undetected future failure wearing a green tick. A model will write 1,642 distinct notes naming sources it never opened. Evaluate the path, not just the destination.

    An eval that passed once is a compressor that worked last year. Prove the guardrail fires on a schedule.

  4. FM4 · Vibe-coding hides the assumptions

    Did anyone test the assumptions, or just the result?

    AI-assisted coding accelerates the build and obscures the assumptions baked into it. Write down what the generated code assumes about its inputs. If that list can’t be written in fifteen minutes, it isn’t ready.

    The tool hands the expert and the novice the same output. Only one of them can be suspicious of it. Make sure that one is in the loop.

And underneath all four: find the actual constraint before you spend your budget on the wrong variable.

Retrieval in three passes

How the hallucinated control ID stopped. Sometimes retrieval failure is a representational limit, not a tuning problem.

  1. Pass 1Scope

    Which standard, which version. A deterministic router. No model.

    Rules out: the wrong framework

  2. Pass 2Candidates

    Structured lookup against the verified table. Unsure returns nothing.

    Rules out: an invented ID

  3. Pass 3Verify

    Every ID, and every control name, checked against the table before it leaves.

    Rules out: an invented or misnamed ID

The model writes the language. It never selects, invents or edits a control ID. Generation is a language problem; recall is a lookup. They are not the same job.

Zero in thirty is not zero

0 / 30 bounds the rate below 10%

Zero failures in thirty adversarial trials bounds the true rate at ten per cent, at 95% confidence. It is not nothing, and it is a very long way from what the word “zero” sounds like.

Rule of three: the 95% upper bound is about 3 ÷ n.

The shelf, live

Five tools from the shelf. Any that calls a model says so on the card. No account, no catch.

AI framework scopingEight questions about how you actually use AI, and three live meters showing whether ISO 42001, the NIST AI RMF or the AU Voluntary AI Safety Standard would earn its place.The Control MeshNine frameworks, 597 controls and 794 cross-framework mappings, each one carrying the obligation it shares, the evidence that satisfies both, and the objection that breaks it.The SkeletonThe decisions a policy has to make, how they are usually got wrong, and a worked extract. Organised by the policy you actually write rather than by control, because nobody writes 597 policies.Where the mapping breaksExamine where control mappings between frameworks actually break down in practice, based on hand-curated notes. No percentages, just structural gaps.AI NavigatorUses a modelA chat assistant you talk to in plain language, back and forth. Launching it asks you to choose ISO 27001:2022 or NIST CSF 2.0 first, then answers on that framework only: controls, requirements and implementation guidance. It opens in a window over this page and nothing you type is kept.

A hint, since you’re here to break it: control IDs never come from the model. The interesting seams are elsewhere.

Decide Who Decides, by Vik Soni. Book cover.

The book

The long version of this talk

Decide Who Decides is almost ready. Signed copies post from Melbourne in early November. Register your interest, tell me you were in the room, and I’ll email you myself with an early-reader code. Orders close 31 October.

Register your interest

Broke something? Found a gap?

Tell me. That’s a conversation I’ll take any day, not just conference week. Find me on LinkedIn.