Your AI feature worked in the demo. Then real users arrived, token bills spiked, and leadership asked why the "cheap chatbot" now costs more than a junior engineer. Cost blowups rarely come from the model alone. They come from unbounded context, retries, and no budget guardrails.
This guide is a practical playbook for shipping LLM features that stay useful and stay affordable.
Start With a Cost Budget, Not a Model Name
Before you pick GPT-5, Claude, or a local model, write three numbers:
Monthly ceiling — the most you will spend on this feature.
Cost per successful action — what one useful outcome (answered ticket, ranked listing, summarized doc) may cost.
Hard kill switch — what happens when you hit 80% and 100% of budget.
Without those numbers, every "just add more context" change feels free. It is not.
Convert the monthly ceiling into a daily and hourly burn so alerts fire early. A $300/month feature that burns $40 in one afternoon needs a circuit breaker, not a postmortem.
Measure Tokens the Way You Measure Latency
You cannot control what you do not measure. Log for every LLM call:
model and provider
prompt tokens and completion tokens
cache hits (if the provider supports prompt caching)
latency and retry count
feature name / route / tenant
outcome: success, empty, refusal, or tool-loop failure
Store these in the same observability stack you use for APIs. A simple table or OpenTelemetry attributes is enough to start. Weekly, rank features by spend and by cost-per-success. The expensive feature that users love is fine. The expensive feature nobody opens is the first cut.
Shrink Context Before You Upgrade the Model
The fastest cost win is fewer tokens in, fewer tokens out.
Retrieve less, not more. Prefer top-k of 3–8 focused chunks over dumping an entire knowledge base into the prompt. Rank by relevance and freshness. Drop boilerplate headers and footers from retrieved docs.
Summarize long histories. Chat threads grow forever. Keep a short rolling summary plus the last few turns instead of the full transcript.
Use structured outputs. Asking for a JSON object with fixed fields usually produces shorter, cheaper, more usable replies than free-form essays.
Cap max tokens. Set max_tokens (or equivalent) to what the UI actually needs. A status badge does not need 2,000 tokens of prose.
Many teams cut spend 30–60% with context discipline alone, before touching model choice.
Match Model Size to the Job
Not every call needs your strongest model.
Classification, routing, extraction: Small / cheap model
Short rewrite or tone fix: Mid-tier model
Hard reasoning, multi-step planning: Strong model
Bulk backfill / offline jobs: Batch APIs or smaller models
Route by task type. A support bot can classify intent with a cheap model, then call a stronger model only when the ticket is complex. That hybrid pattern often cuts cost more than shaving a few tokens from a single mega-prompt.
Cache Aggressively Where Providers Allow It
Static system prompts and shared tool schemas are excellent cache candidates. If your provider supports prompt caching or you control the gateway:
Keep the stable prefix identical across requests so cache keys hit.
Put volatile user content at the end.
Avoid regenerating the same system prompt string with tiny whitespace diffs.
For identical reads (FAQ answers, policy lookups), a short application-level cache keyed by normalized question can skip the LLM entirely. Even a five-minute TTL saves money on popular queries.
Bound Tools, Retries, and Agent Loops
Agent-style features are where costs explode.
Cap tool-call rounds (for example, 3–5).
Fail closed when the loop cannot finish; do not retry forever.
Deduplicate identical tool calls in one turn.
Prefer deterministic code for math, sorting, and database filters; reserve the LLM for language and judgment.
Retries after timeouts should use exponential backoff and a hard limit. Blind retries on a hung tool can multiply spend without improving quality.
Price Quality With Evaluation, Not Vibes
Cutting cost is easy if you accept worse answers. Protect quality with a small, fixed eval set:
20–50 real examples per feature
Expected outputs or scoring rubrics
Automatic checks for format, required fields, and refusal policy
Spot human review on a sample each week
When you switch to a cheaper model or shorter prompt, run the eval before shipping. Keep the change only if quality stays within your bar. That turns cost work into engineering, not gambling.
Add Product Guardrails Users Can Feel
Cost control is also product design:
Rate-limit per user and per tenant.
Offer "quick answer" vs "deep research" modes with different budgets.
Show when the system used cache or skipped the model.
Degrade gracefully: template reply, search results, or human handoff when budget is exhausted.
Users tolerate limits when the product stays honest. They do not tolerate silent failures or sudden feature death with no message.
A One-Week Cost Cut Plan
Day 1: Log tokens and cost per feature.
Day 2: Cap
max_tokensand trim system prompts.Day 3: Reduce retrieval chunk count; add summarization for long chats.
Day 4: Route simple tasks to a cheaper model.
Day 5: Enable prompt/app caching for stable prefixes and FAQs.
Day 6: Cap agent tool loops and retries.
Day 7: Run evals; keep only changes that hold quality.
Most teams find the biggest wins in days 2–5.
What "Good" Looks Like
A healthy LLM feature has:
a written monthly budget and alerting
per-call cost metrics in production
context that is retrieved and trimmed on purpose
model routing by task difficulty
caching for repeated work
bounded agent loops
evals that block quality regressions
You do not need a research lab to run this playbook. You need the same discipline you already apply to database queries and third-party APIs.
Closing
LLM spend becomes a problem when generation is treated as free compute. Treat each call like a paid API with a measurable outcome. Measure tokens, shrink context, route models by job, cache repeats, bound loops, and verify quality with evals. Do that, and you can keep shipping AI features without a surprise invoice—or a surprise drop in answer quality.