A check that cannot fail is worse than no check at all

On Sunday we found a check in one of our own software projects that had been reporting success while the thing it was checking had failed. It had done this through two rounds of changes. Nobody noticed, because the check said everything was fine, and the whole point of a check is that you stop looking once it says so.

The cause was mundane. The step that was meant to stop the work on a failure was wired so that it only ever reported the result of a harmless final step, which always succeeded. A green light, permanently on.

The same day, in a different system, we found a protective limit that had been catching its own error and quietly letting everything through. It was built to say no under certain conditions. Because of a small technical mismatch, it had been saying yes to everything.

Both now have fixes written and tested, with one still waiting to go live. The rule we wrote into our company memory afterwards is the reason for this post:

A gate that cannot fail is not a gate. It is worse than no gate, because it is trusted.

This is not a software problem

Swap the technical details for paper and you will recognise both failures in almost any business.

• The invoice approval where the manager signs whatever is in the tray on Friday afternoon, because there are forty of them and they are usually right.
• The stock count that is always "checked" because the checker copies the first counter's numbers.
• The site safety form with every box ticked in the same pen, at the same angle, before the shift started.
• The "reviewed by" field in the CRM that fills itself in when the record is saved.
• The AI-drafted quote that a person "approves" by clicking send without reading it.

Each of these looks like a control on a process map. Each one has stopped being able to say no. And each one is worse than having no control at all, because people downstream relax. The accounts team does not double-check approved invoices. The client trusts a quote that went through review.

Why approvals decay

Mostly, because they almost never fail. If ninety-nine invoices in a hundred are correct, the approver learns that approving is the right answer. The check becomes a habit, then a formality, then a rubber stamp. Nobody decides this. It just happens.

AI makes it faster. When a system drafts the quote, the email or the report, the human step is often the only check left, and it is precisely the step most likely to become a click.

Test every gate with a known bad case

The fix that worked for us is simple: prove the gate can refuse. For the software check, that meant replacing the chain with a script that checks each step's result and stops at the first failure, then feeding it a deliberate failure to watch it stop. For the limit, the rule we wrote down was that a control which fails open needs a test showing it refuses, or it is decoration.

For a business, the same idea looks like this:

1. List your gates. Every place where something is supposed to be checked or approved before it moves on.
2. For each one, write down what it should reject. An invoice over budget. A quote below cost. A safety form with a missing signature.
3. Feed it one. Quietly put a known bad item through and see whether it comes back. Do it with the approver's knowledge if you prefer. You are testing the gate, not trying to catch a person out.
4. Fix or remove. A gate that passed the bad item gets redesigned, split, or made smaller so a person can genuinely review it. A gate that serves no purpose gets removed, so nobody trusts it.

What memory has to do with it

Writing the rule down matters because the lesson otherwise lives with whoever found the problem. In our case the rule, the reason and the way to apply it now sit in the shared memory that every AI session and every person on the work reads first. The next time any of us builds a check, the rule is already in front of us.

That does not make the next check correct. Memory carries the lesson. Someone still has to test the gate.

If you suspect some of your approvals have quietly turned into formalities, the operations drag check is a good place to start. Book 20 minutes at pgco.world/book to go through it with us.