What a 1M context actually costs a machine

The limit on long context is almost never the model's capability. It's the memory and bandwidth of the server running it.

A model that advertises a 1M-token context really does support it — GLM 5.2, which we serve, does so natively. But “supports” means a single request goes through. It says nothing about how many long sessions a machine can hold at once, and that is where everything is decided.

On every decoding step, the GPU must re-read the KV cache of every active sequence. Model weights are amortized across the whole batch; the KV cache never is — each session pays for its own. So the server's memory bandwidth divides by the number of active sessions multiplied by their context length.

On a dense attention architecture the effect is brutal: going from 64K to 1M of context multiplies the per-session memory cost by sixteen, and divides the number of sessions the machine can host by the same factor. It isn't that each session becomes slow — it's that the machine holds sixteen times fewer of them.

Sparse attention architectures change the arithmetic. GLM 5.2 uses a mechanism that reuses the same indexer across each group of four layers, sharply reducing per-token compute at long context. Cache storage stays roughly linear — you have to keep it all in order to select from it — but the per-step read no longer does.

The lesson for anyone sizing inference infrastructure: don't derive this number by hand. Modern servers compute it at startup and print it. For us, that measurement — not an extrapolation — is what sets what we guarantee to customers.