Infrastructure
Making LLM costs predictable
Language model cost scales with the text processed: input and output tokens. What inflates a bill is rarely the number of requests but the context carried needlessly on every one, and answers longer than anyone reads. Applied together, these four changes typically cut spend by more than half.
- Published
- Built by
- Zero-Shot Team
- 3 min read
First, see where it goes
Optimising without measuring is guessing. Step one is logging every request with its input tokens, output tokens, duration and the feature it came from.
Once that log exists, the picture is nearly always the same: a small share of requests produces most of the cost. Usually one feature, and often one of the least used.
Tier your models
Sending every task to the same large model is like having a lawyer write all your emails. Classification, formatting, short summaries and data extraction come out of small models at nearly the same accuracy.
A workable rule: try the small model first, escalate to the large one when confidence is low. Track the escalation rate; if it is high, the threshold is set wrong.
Do not make this change without an evaluation set. Otherwise it takes months to notice that you cut cost and accuracy together.
Prompt caching
In most applications every request starts with the same text: the system instruction, the format description, examples, sometimes an entire document. That fixed portion is billed again on every call.
Provider prompt caching makes that repetition substantially cheaper. The condition is that the fixed portion sits at the very start of the prompt and genuinely does not change. Putting a username or a date inside the system instruction disables caching entirely — a common and expensive mistake.
Output length
Output tokens cost more than input. Yet most applications ask for free-form text and then use half of what comes back.
Two interventions suffice: bind the shape of the answer (structured output or a schema) and set an upper limit. Saying which fields you want, rather than asking for brevity, makes the result both cheaper and more usable.
Ceilings and alarms
Every feature needs a per-request token ceiling, a per-user daily limit and an alert threshold on total spend. These are not tuning knobs; they are accident prevention.
An agent stuck in a loop, a hostile input or a misconfigured batch job can burn a monthly budget in one night when nothing caps it.
In short
- Log every request by tokens, duration and feature; cost concentrates in a few places.
- Try the small model and escalate when needed — but build the evaluation set first.
- Put fixed instructions at the start of the prompt and keep variables out of them, or caching never engages.
- Bind output to a schema and cap its length; output tokens cost more.
- Set per-request ceilings, per-user daily limits and a spend alarm.
Common questions
- Is self-hosting an open model cheaper?
- At high, steady volume it can be. But compare total cost of ownership rather than token price: hardware, idle time, scaling, updates and the person who operates all of it. At low or spiky volume, a provider API usually stays cheaper.
- Context windows are huge now — is RAG obsolete?
- Fitting is not the same as being economical. Carrying a hundred thousand tokens on every request costs many times more than finding and carrying the relevant five thousand — and is usually less accurate, because the model gets lost among irrelevant text.
- How much accuracy does cost reduction cost?
- Only an evaluation set can answer that for your case. What we see in practice: tiering and caching barely move accuracy; the real risk is trimming output length too far.