The Hidden Cost of a Bigger Context Window

Every few months a new model lands with a bigger context window. 128K. 200K. A million tokens. The pitch is intoxicating: just paste everything — the whole codebase, the entire knowledge base, every past message — and let the model sort it out. No more chunking, no more retrieval, no more clever engineering. Context is all you need.

I've fallen for this pitch more than once. And every time, the same lesson shows up a few weeks later, on the invoice and in the latency graphs.

Bigger context windows are real and useful. But they are not free — and the cost doesn't grow the way most people assume.

Attention is quadratic, and that math never sleeps

The core of a transformer is self-attention: every token looks at every other token. That "every-to-every" comparison is the whole point — it's what lets the model connect a pronoun on line 900 to a name on line 3. But it also means that if you double the number of tokens, you don't double the work. You quadruple the attention computation. This is the O(n²) scaling that's been baked into transformers since the original 2017 Attention Is All You Need paper.

The KV cache — the memory the model keeps so it doesn't recompute past tokens at every step — grows linearly, O(n). That sounds gentler, but at long context lengths the KV cache is what actually fills up GPU memory and caps how many requests you can serve at once.

So when you go from a 4K prompt to a 40K prompt, you're not paying 10× more. On the compute side you're closer to paying 100× more for attention, plus 10× more memory pressure. The demo where you paste one giant document feels magical. The production system serving thousands of those requests an hour feels like a bill you didn't sign up for.

"Just put it all in context" quietly hurts quality too

Here's the part that surprised me most: filling the window doesn't only cost money — it can make answers worse.

Researchers documented a "lost in the middle" effect: models are good at using information at the very start and the very end of a long context, and noticeably worse at using information buried in the middle (Liu et al., 2023, Lost in the Middle: How Language Models Use Long Contexts). Stuff a model with 80K tokens of background and the one sentence that actually matters might land right in its blind spot.

So the "paste everything" strategy can be the worst of both worlds: you pay the quadratic compute bill and you dilute the signal the model needs.

What I actually do now

A few rules of thumb I've landed on:

• Retrieve, then reason. Pulling the 5 most relevant chunks into a 4K prompt usually beats dumping 100K tokens of "maybe relevant" context — cheaper, faster, and often more accurate.

• Put the important stuff at the edges. If you do need a long prompt, lead with the key instruction and repeat the critical constraint at the end.

• Measure tokens like you measure latency. Track input tokens per request the same way you track p95 response time. It's a real cost driver, not an afterthought.

• Treat the big window as a ceiling, not a default. A million-token context is a fantastic escape hatch for the rare huge task. It's an expensive habit if it becomes your everyday prompt.

The takeaway

Bigger context windows expanded what's possible, and that's genuinely exciting. But possible and efficient are different words. The window is a tool, not a strategy — and the teams that win are usually the ones being deliberate about what they put in it.

At BrightLearn this matters more than it might sound. When you're generating personalized lessons and feedback for a lot of students at once, every wasted token gets multiplied across thousands of requests. Being disciplined about context isn't just an engineering nicety — it's part of what keeps good AI tutoring affordable enough to actually reach the students who need it.