The boring truth about AI risk

When people imagine AI going wrong, they picture dramatic, sci-fi failures. In practice, the harm that actually shows up in deployed systems is mundane: a screening tool that quietly disadvantages a group, a chatbot that confidently gives wrong information, a system that no one can explain when a customer challenges its decision. None of these require advanced capabilities. They require only a system shipped without the basic governance questions answered.

This is good news, because it means most deployment risk is manageable with discipline rather than genius. Below is a practical pre-deployment review — the questions worth answering before anything touches a real user.

1. What decision does this system actually influence?

Start here, because everything else scales from it. A system that drafts marketing copy carries different stakes from one that influences who gets a loan, a job interview, or a benefit. Map the real-world consequence of the system being wrong. The higher the stakes, the more of the rest of this list you need.

Be honest about indirect influence too. A tool framed as 'just a recommendation' that humans rubber-stamp 99% of the time is effectively making the decision.

2. Who could this system harm, and how?

List the people affected — not just users, but third parties and bystanders. Then ask how the system could disadvantage them: through inaccuracy, through bias against a group, through exclusion of edge cases the training data underrepresented, or through over-reliance by an operator who trusts it too much. You can't mitigate harms you haven't named.

3. Where did the data come from, and what's missing from it?

Data is where most quiet failures originate. Two questions matter most: Is this data representative of the people the system will actually affect? and What's systematically absent? A model trained mostly on one population will perform worse on others, often invisibly. Underrepresentation in training data becomes underperformance in production.

Also confirm you had the right to use the data at all, and that using it doesn't leak anything sensitive.

4. Can a human understand and override it?

For any consequential system, two capabilities are non-negotiable. Explainability: when someone asks why the system did what it did, can you give a meaningful answer? Override: when the system is wrong, can a human step in, reverse it, and is it clear who that human is? A system with no override path and no explanation is a liability waiting for its first hard case.

5. How will you know if it's going wrong?

Deployment is the beginning, not the end. Decide in advance:

• What you'll monitor — accuracy, complaint rates, performance across different groups, drift over time.
• What thresholds trigger action — at what point do you intervene, retrain, or pull it?
• Who's watching — a named owner, not a diffuse 'the team.'
• How users report problems — a real, used channel, not a buried form.

Many failures are gradual. Without monitoring, you find out from a viral complaint rather than your own dashboard.

6. What's the fallback when it fails?

Every deployed system will, eventually, be unavailable or wrong. What happens then? Is there a manual process, a graceful degradation, a clear escalation? Systems that have no answer to 'what if it's down or wrong' tend to fail loudly at the worst moment.

7. Who is accountable?

Finally, the question that ties it together: when this system causes a problem, who owns it? Not in a blame sense — in an ownership sense. A named person or team responsible for the system's behaviour, with the authority to change or stop it. Diffuse accountability is the same as no accountability.

Make it routine

None of this requires a research lab. It requires treating AI deployment with the same seriousness as any other consequential business process. Run this review before launch, write down the answers, and revisit them when the system or its context changes. The organisations that get into trouble are almost never the ones that asked these questions and got them slightly wrong — they're the ones that never asked.