When people talk about what an AI model costs, they usually mean the training run — the one dramatic number, tens or hundreds of millions of dollars of compute, spent once to create the model. It makes the headlines. It is also, over the life of a popular model, usually the smaller bill.

The larger one is inference: the cost of actually running the model every time someone uses it. Training happens once. Inference happens on every single query, forever, for as long as anyone is using the thing. Understanding the difference is the key to understanding almost everything about the AI build-out — why companies are spending hundreds of billions on data centres, why prices for using models keep falling even as those bills climb, and why "how much did it cost to train" is the wrong question.

Two different cost regimes

Training is a capital cost. You assemble a huge cluster of GPUs, feed the model trillions of words, and run it for weeks or months. Enormous, but one-time — and, crucially, it is amortised. Spread that cost across every query the model will ever answer, and per-query it can round to nothing. A model used a billion times has effectively paid off its training in fractions of a penny each.

Inference is an operating cost. It does not amortise, because you pay it fresh every time. Ask a model a question and it has to load its billions of parameters, run the maths, and generate an answer — and it generates that answer one token at a time, each new word requiring another full pass through the network. A long answer is not one computation; it is hundreds of them in sequence. Multiply that by millions of users making billions of requests, and inference is where the real money goes.

This is the counter-intuitive part: the training number is the one you hear, but for any model with real usage, the lifetime spend on serving it dwarfs the cost of building it.

Why serving is the hard part

A few things make inference stubbornly expensive:

• It is sequential. Text is generated token by token, and each token depends on the last, so you cannot simply parallelise your way out of a long response the way you can with training.
• Memory, not just maths. As a conversation grows, the model keeps a running store of everything said so far — the KV cache — and that store grows with the length of the chat. Longer contexts mean more memory and more bandwidth per token, which is why long conversations get slower and pricier.
• It runs on scarce, expensive hardware. The same kind of chips that train models also serve them, and serving at scale needs a lot of them, sitting in data centres, drawing power, all day every day. When Amazon blamed a capex hike on surging memory-chip prices, that is the inference era talking: a fast-growing share of that build-out is about serving models to users, not just training the next one.

Why prices still fall

If inference is so expensive, why does the cost of using AI keep dropping? Because inference cost is where nearly all the engineering effort goes, and it responds. A stack of techniques exists purely to drive down the cost — and the latency — of each token. Some cut the raw work per token: quantization lowers the numerical precision so the model needs less memory and bandwidth; mixture-of-experts designs activate only a fraction of the network per token; distillation trains a small cheap model to mimic a big one. Others raise how much a given chip can serve: speculative decoding lets a small model draft several tokens that the big one verifies in a single pass, so answers come faster and the hardware clears more of them; and batching many users' requests together keeps the GPUs busy. Every one of these lowers the served cost per token, and together they are why last year's capability keeps getting cheaper to run even as frontier capability stays expensive.

The takeaway

Training builds the model; inference is the business. The training run is a bet — a large, one-time wager that the model will be used enough to justify it. Inference is the recurring cost of collecting on that bet, and it is the number that actually scales with success: the more people use a model, the more it costs to keep answering them. That is why the industry's spending has shifted from "who can afford to train the biggest model" toward "who can afford to serve one to hundreds of millions of people" — and why the cost of building AI and the cost of using it can climb and fall at the same time. They are two different bills, paid at two different times, for two different things.