The context-window trap
Big context windows are marketed as a capability. They are also a billing mechanism. A 10-turn conversation carrying 50k tokens of retrieved context costs 10× a single-turn call — on a $3/M input model that's about $1.50 per conversation. Fine for an enterprise tool; lethal for a consumer chatbot at $20 a month.
Three fixes, in order of return
Summarize the conversation every 4–6 turns and discard raw history (a 90% token reduction is normal). Cache the fixed portion — system prompt and stable retrieved chunks — for 75–90% off that slice. Retrieve less: top-3 well-ranked chunks usually beat top-10 mediocre ones, at a third of the tokens.
- • Summarize every 4–6 turns; keep a rolling gist, not a transcript.
- • Cache the system prompt and stable context.
- • Cut top-k retrieval and rerank instead of over-fetching.
- • Trim tool definitions — they're input tokens on every single call.
Long context also degrades quality
This isn't only a cost argument. Models reliably show 'lost in the middle' behaviour, where information buried in a long context is retrieved less accurately than the same information at the start or end. Shorter, better-curated context is usually both cheaper and more accurate.
Related guides
Long-form playbooks on the same topic, written by the RevenueLab editorial team.
FAQ
Why does my chat cost more than my per-turn estimate?
Because every turn resends the system prompt, retrieved context, and the full prior conversation. Turn 10 pays for turns 1–9 all over again. A 10-turn conversation typically costs 8–15× a single turn, not 10× a user message.
How much does a 200k-token context call cost?
At $3 per million input tokens, a single 200k-token call costs $0.60 in input alone — before output. Ten such turns is $6 per conversation. This is why long-context workflows need caching or aggressive retrieval limits.
How much does summarization actually save?
Summarizing every 4–6 turns and discarding raw history typically cuts history tokens by around 90%. On a long conversation that's commonly a 40–70% reduction in total input spend.
Is caching better than summarization?
They solve different halves. Caching discounts the fixed portion (system prompt, stable docs) by 75–90%. Summarization shrinks the growing portion (conversation history). Production systems should do both.
Should I just use a bigger context window instead of RAG?
Rarely. Stuffing everything into context costs more per call and retrieves less accurately for information buried mid-context. RAG keeps the prompt small and targeted — cheaper and usually more accurate.
Do tool definitions count as input tokens?
Yes, and they're easy to forget. A dozen verbose tool schemas can be several thousand tokens resent on every single call. Trim descriptions and only expose the tools relevant to the current step.
How this calculator is built
Independently maintained
Written by Sam Doshi and the RevenueLab editorial team. We don't sell the data feeds this tool is built on.
Sourced from primary data
Benchmarks come from public AdSense / Stripe / IRS disclosures and reader-submitted data — never third-party "$X per view" claims. Full methodology.
Last editorial review
Reviewed on a rolling quarterly cycle. Dated reviews are published on the methodology record for each calculator.
Editorial standards
See our editorial policy and disclaimer. Results are estimates, not advice.