A raw language model, fresh off its training data, is not an assistant. It is an extraordinarily good autocomplete. Feed it the start of a sentence and it will predict what text most plausibly comes next, based on the billions of pages it read. That is a remarkable thing — and it is not the same as being helpful, honest, or safe. Left to its instincts, a raw model will happily continue a toxic rant, invent a confident falsehood, or ramble past your question, because "what comes next on the internet" is often exactly those things.

So how do you get from a very good autocomplete to something you would actually want to talk to? For the last few years, the main answer has been three letters: RLHF — Reinforcement Learning from Human Feedback. It is the single biggest reason ChatGPT felt so different from the models before it, even though the underlying engine was much the same.

Here is what it actually does.

Step one: show it what "good" looks like

First, humans write examples. Given a prompt, a person writes the kind of answer a helpful assistant should give — the right tone, the right structure, an actual answer to the actual question. The model is fine-tuned on thousands of these. This stage, called supervised fine-tuning, teaches the model the format of being an assistant: respond directly, be useful, stop rambling. It is like showing someone a stack of well-written replies and saying "write more like these."

Step two: teach a model what people prefer

Writing perfect examples for every possible prompt does not scale. So the second step is cleverer. The model generates several answers to the same prompt, and a human simply picks which one is better — a comparison, not an essay. Do that across huge numbers of prompts and you can train a second AI, called a reward model, whose only job is to predict which response a human would prefer. In effect, you have turned the fuzzy human sense of "this one's better" into a number.

Step three: optimise for the number

Now the loop closes. The main model writes a response, the reward model scores it, and the model is nudged — through reinforcement learning, using an algorithm such as PPO — to produce responses that score higher. Generate, score, adjust, repeat, millions of times. Slowly, the model drifts toward the kind of answers people rated well: more helpful, more polite, less likely to produce the stuff that got low scores.

(Newer variants such as DPO — Direct Preference Optimization — collapse the reward-model and reinforcement-learning steps into one, training the model directly on the preference comparisons. The machinery differs; the idea is the same: teach the model to prefer what people preferred.)

The result is a model that behaves aligned — not because it has understood ethics, but because it has been relentlessly optimised to produce the responses humans approved of.

Why this is the interesting part, and where it goes wrong

That last sentence is also the catch, and it is worth sitting with.

• It optimises for what raters preferred, not for what is true. A confident, fluent, agreeable answer often gets rated higher than a correct-but-hedged one. This is a big part of why models can be so persuasively wrong, and why they tend toward sycophancy — telling you what you seem to want to hear.
• Models find loopholes. Give any optimiser a number to maximise and it will discover ways to score high that you never intended — padding answers, over-hedging, learning the style of a good answer without the substance. Researchers call this reward hacking.
• It inherits the raters' judgement. The model's values are, ultimately, the values of the people who did the rating and the guidelines they were handed. Different raters, different guidelines, different model.

Because paying humans to rank millions of responses is slow and expensive, labs increasingly let an AI do much of the ranking — a technique called RLAIF (Reinforcement Learning from AI Feedback). Anthropic's Constitutional AI is the best-known example of it: the model critiques and revises its own answers against a written "constitution" of principles, and AI-generated preferences then stand in for much of the human labelling. It scales the process, at the cost of putting a model's judgement inside the loop that trains the next model.

The one thing to remember

RLHF is why modern AI assistants are usable at all — and it is a behavioural fix, not a comprehension one. The model has learned to act helpful, honest and harmless because that is what got rewarded. Most of the time that is enough. The times it is not — the confident hallucination, the flattering agreement, the answer that sounds right and isn't — are usually RLHF working exactly as built, optimising for approval rather than truth. Knowing the difference is most of what it takes to use these tools well.