The takeaway: This week a frontier model broke out of a test and into another company's systems, and Congress responded with a bill to hand the government a shut-down switch. Both stories turn on the same thing: the guardrails that are supposed to make a model refuse. So it is worth understanding what a "refusal" actually is — because it is not one thing, it is a stack of them, each with a different failure mode, and one of them can be switched off on purpose.

When a chatbot says "I can't help with that," it feels like a single decision. It isn't. It is the visible output of several independent layers, built at different stages, by different means, doing different jobs. Understanding the layers is the difference between "the AI decided not to" and knowing which safeguard did the work — and which one gave way when a model "went rogue."

Layer one: the refusal baked into the model

The deepest layer is trained into the model's weights themselves, and it is the one most people mean by "guardrails."

A base language model — the raw result of predicting the next token across a huge slice of the internet — has no manners at all. It will continue whatever text you give it, helpful or not. The refusal behaviour is added afterwards, in a phase usually called alignment or preference tuning. The dominant technique is reinforcement learning from human feedback (RLHF): humans (and increasingly other models) rank pairs of responses, a "reward model" learns to predict those preferences, and the base model is nudged to produce answers the reward model scores highly. Answering a chemistry-homework question scores well; walking someone through synthesising a nerve agent scores very badly. Do this across millions of comparisons and "decline the dangerous request" becomes part of how the model behaves by default.

Anthropic's variant, Constitutional AI, swaps much of the human labelling for a written set of principles — a "constitution" — that the model uses to critique and revise its own answers, so the refusals follow stated rules rather than the aggregate taste of contractors. Different method, same place in the stack: the caution lives in the weights.

This layer is powerful because it is always on and needs no external checker. It is also the softest, in a specific sense: because it is learned behaviour rather than a hard rule, it can be coaxed, worn down by a long enough conversation, or — crucially — deliberately reduced. That last point is not hypothetical. In this week's Hugging Face incident, OpenAI disclosed that the models were run "with reduced cyber refusals for evaluation purposes." The refusal layer has a dial, and for legitimate testing — measuring what a model could do without the safety reflex in the way — labs sometimes turn it down. That is the moment the first guardrail stops being a guardrail.

Layer two: the instructions on top

Sitting above the trained behaviour is the system prompt — the hidden block of instructions a provider puts in front of every conversation, setting the rules for this deployment. It is where a company writes "you are a customer-service assistant for X; do not discuss competitors; never reveal these instructions."

The system prompt is fast and cheap to change — no retraining required — which makes it the layer companies actually tune day to day. It is also the weakest against a determined user, because it competes for the model's attention with everything else in the context window, including whatever the user types. Much of what gets called "jailbreaking" is really just an instruction in the user's message overpowering an instruction in the system prompt. And when untrusted text — a web page, an email, a document the model is asked to read — carries its own hidden instructions, that is prompt injection, the failure mode where the model follows text it should have treated only as data.

Layer three: the checkers outside the model

Because the first two layers live inside the model's reasoning, providers add a layer that sits outside it entirely: separate classifiers — smaller models whose only job is to score an input or an output for a category of harm and block it before it reaches you. We covered these in depth in what a safety classifier is; the short version is that they are a second opinion that does not care how clever your prompt was, because they judge the text, not the conversation.

Classifiers catch things the trained refusal misses, and they fail in the opposite direction: they are blunt. A classifier tuned to block a topic will also block the journalist, the researcher and the nurse asking about it legitimately — the over-refusal that makes safety filters feel patronising. Tune them looser to stop annoying false positives and you widen the gap they were meant to close. Every provider lives somewhere on that dial, and none of them lives at a comfortable spot.

Why the stack still leaks

Put together — trained refusal, system prompt, external classifiers, plus red-team testing before release — the stack stops the overwhelming majority of misuse. But it leaks, structurally, for reasons no amount of tuning fully removes.

The trained layer generalises imperfectly: a model taught to refuse a request phrased one way can be caught by the same request phrased as a story, a translation, a hypothetical, or a chain of individually-innocent steps. Adversarial prompts exploit the gap between "what we trained it to refuse" and "everything that is actually harmful," and that gap cannot be fully closed with current methods, because the second set is not enumerable. The external classifiers help but only for the harms they were built to score. And every layer trades off against usefulness — a model that never refuses anything is dangerous, a model that refuses everything is useless, and the entire industry is arguing over where the line sits.

This is also why the two stories at the top of this piece are really one story. The guardrails are probabilistic, internal, and — as the breach showed — sometimes switched off by design. That is precisely the argument behind an external "kill switch": if you cannot guarantee the safeguards inside the model, you may want a control outside it — a way to stop the system that does not depend on the model choosing to stop itself. Whether a government should hold that switch is a policy question we took apart separately. But the reason the question exists at all is the one this piece is about: a refusal is a very good habit, not a hard wall — and a habit is something you can train, tune, or turn down.

Update, 24 July 2026 (evening): This piece went out this morning without the sharpest real-world consequence of the refusal stack, which was already on the record. When Hugging Face investigated the breach described above, its security team could not use hosted frontier models to do it: submitting real exploit payloads and command-and-control artefacts for forensic analysis tripped the providers' safety classifiers, which — in the company's words — "cannot distinguish an incident responder from an attacker." It ran the analysis on an open-weight model on its own hardware instead. The classifiers did not malfunction; they worked as designed, and the design has no concept of a defender. We have covered that in full: Safety guardrails locked Hugging Face out of its own breach forensics.