Cutting Your LLM Bill 10x - Caching, Model Routing, Quantization, and Local Inference with vLLM

Draft handbook for review
Audience: engineers, AI product builders, technical founders, and teams moving LLM features from prototype to production.
Core promise: reduce LLM spend without blindly destroying product quality.

Source Notes

This draft is based on current public material from provider documentation and engineering references, including OpenAI prompt caching documentation, vLLM documentation and the original vLLM PagedAttention article, BentoML's LLM inference and quantization material, and recent production-focused writing on semantic caching, model routing, prompt compression, and LLM cache invalidation.

Key source themes used in this handbook:

• OpenAI documents prompt caching as an automatic provider-side optimization that can reduce latency and input-token cost for repeated prompt prefixes.
• vLLM documents efficient LLM serving with PagedAttention, continuous batching, prefix caching, streaming, and OpenAI-compatible APIs.
• BentoML and related inference references describe quantization as a way to reduce model memory footprint and improve deployment economics, while warning about quality and hardware tradeoffs.
• Semantic caching references consistently identify cache invalidation, user context, retrieval freshness, and safety as the real production challenges.

This handbook avoids treating any single technique as magic. The practical thesis is that major LLM savings come from layering multiple controls: measurement, prompt trimming, caching, routing, output limits, agent budgets, evals, quantization, and only then local inference.

1. Why LLM Bills Explode After the Demo

Most LLM applications start cheap. A small team builds a prototype, sends a few prompts during development, maybe demos it to leadership, and the monthly bill looks harmless. The app feels magical. The cost seems manageable. The team ships.

Then real users arrive.

The prompts get longer. The product team asks for better answers, so engineers add more instructions. The support team wants personalization, so the system includes account history. The legal team wants disclaimers. The growth team wants tone control. The data team wants retrieval-augmented generation. The agent now calls tools, retries failed actions, summarizes long conversations, and stores memory. Every layer seems reasonable in isolation, but together they turn one user action into thousands or tens of thousands of tokens.

The surprise is not that LLMs cost money. The surprise is how quickly product complexity turns into token spend.

A simple prototype might look like this:

User question -> LLM answer

A production system often looks more like this:

User question
-> safety classification
-> query rewrite
-> embedding lookup
-> vector search
-> keyword search
-> reranking
-> context assembly
-> main LLM answer
-> structured output repair
-> moderation check
-> follow-up suggestion generation
-> conversation summary update

Each step may be defensible. Each step may improve quality. But each step also has cost, latency, and failure modes.

This is why teams need an LLM cost strategy. Not because cost is more important than quality, but because uncontrolled cost eventually limits product quality. If every request is expensive, teams become afraid to add features, evaluate outputs, run experiments, or serve lower-value users. Cost control is not just finance hygiene. It is product scalability.

The best cost optimization does not start with “replace the model.” It starts with understanding the workload.

You want to answer questions like:

• Which features create the most LLM spend?
• Which users or accounts create the most spend?
• Which prompts are longest?
• Which model is used most often?
• Which calls have low value or repeated answers?
• How many retries happen?
• How many tool calls happen per agent run?
• How often does retrieval include irrelevant context?
• How much output does the model generate unnecessarily?
• Which requests could safely use a cheaper model?

Without this information, optimization is guesswork. With it, cost reduction becomes an engineering process.

2. The Basic Economics: Where the Money Goes

LLM cost usually comes from several places.

2.1 Input Tokens

Input tokens are everything you send to the model:

• system prompt
• developer instructions
• user message
• chat history
• retrieved documents
• tool outputs
• examples
• schema instructions
• hidden framework messages

Input tokens are easy to underestimate because many of them are invisible to the user. A user may type one short sentence, but the application may send a huge prompt behind the scenes.

For example:

User: Can I return this item?

The actual model input might include:

• 1,000-token system prompt
• 500-token safety policy
• 2,000-token customer account context
• 4,000-token return policy context
• 1,000-token prior conversation summary
• 500-token output format schema
• 8-token user question

The user sees a small question. The model sees a large document.

2.2 Output Tokens

Output tokens are what the model generates. They often cost more per token than input tokens, depending on provider and model.

Many teams optimize input context and forget output length. But verbose answers, long JSON outputs, repeated disclaimers, and unnecessary reasoning text can become a major cost center.

Output control is one of the simplest levers:

• set max_tokens
• ask for concise answers by default
• avoid generating explanations when only a classification is needed
• use structured outputs with minimal fields
• separate internal reasoning from user-facing output when supported by the model/provider

If a task only needs a label, do not ask the model for a paragraph.

2.3 Model Choice

Model choice is the most obvious cost lever. Premium frontier models are powerful, but many production tasks do not need them.

Common tasks that often work with cheaper models:

• classification
• routing
• sentiment detection
• simple extraction
• title generation
• short summarization
• rewriting text in a fixed style
• tagging support tickets
• deciding whether to escalate

Common tasks that may need stronger models:

• complex multi-step reasoning
• ambiguous legal or financial analysis
• high-stakes customer-facing answers
• coding tasks with large context
• agent planning
• synthesis across many conflicting sources
• tasks requiring strong instruction following under messy context

The goal is not to always use the cheapest model. The goal is to use the cheapest model that reliably meets the quality requirement for that task.

2.4 Retrieval and Reranking

RAG systems add cost in places teams forget:

• embedding documents
• embedding queries
• vector database calls
• keyword search
• reranking
• summarizing retrieved chunks
• sending retrieved chunks into the final prompt

RAG can reduce hallucination and improve freshness, but badly tuned RAG can also explode input tokens. The system retrieves too many chunks, includes irrelevant context, and forces the model to read a pile of text to answer a simple question.

Cost-effective RAG is not “retrieve more.” It is “retrieve enough relevant context.”

2.5 Agent Loops

Agents are powerful because they can plan, call tools, observe results, and continue. They are expensive for the same reason.

A single user request can become:

LLM planning call
Tool call
LLM observation call
Tool call
LLM observation call
Tool call fails
Retry
LLM recovery call
Final answer call
Memory update call

If the agent has no budget, no stopping rule, and no loop detection, cost can become unpredictable. This is especially dangerous when a tool fails repeatedly or returns confusing output.

For production agents, cost control is part of reliability engineering.

2.6 Retries and Error Handling

Retries are hidden spend. A flaky provider, a schema validation failure, a timeout, or a tool error may cause the application to call the model again. If the retry uses the same huge context, cost doubles.

Structured output repair is another hidden cost. If a model returns invalid JSON, the system may send the invalid output back to the model and ask it to fix it. That may be necessary, but it should be measured.

2.7 Evaluation and Observability

Evaluation also costs money. LLM-as-judge systems, regression tests, trace analysis, and synthetic test generation all call models. These are valuable, but they should be budgeted.

A mature team tracks production inference cost and evaluation cost separately.

3. Step One: Measure Before You Optimize

You cannot cut an LLM bill intelligently if you do not know what created it.

At minimum, log these fields for every LLM call:

request_id
user_id or account_id
feature_name
task_type
model
provider
input_tokens
output_tokens
total_tokens
estimated_cost
latency_ms
cache_hit
retry_count
tool_call_count
success_or_failure

If you use RAG, also log:

retrieved_chunk_count
retrieved_token_count
reranker_used
embedding_model
vector_index
source_documents

If you use agents, log:

agent_turn_count
tool_names_called
failed_tool_calls
loop_stopped_by_budget
final_status

The first useful dashboard is simple:

MetricQuestion It AnswersCost by featureWhich product area is expensive?Cost by accountWhich customers drive spend?Cost by modelAre premium models overused?Avg input tokensAre prompts/context too large?Avg output tokensAre answers too verbose?Cache hit rateAre repeated requests being reused?Retry rateAre failures doubling cost?Agent turns per requestAre agents looping?Cost per successful taskAre failed requests wasting spend?

Measurement often reveals embarrassingly simple waste:

• the same system prompt repeated everywhere
• a full conversation sent when a summary would do
• 20 retrieved chunks when 4 would do
• a premium model used for boolean classification
• a model asked to produce long explanations nobody reads
• failed JSON outputs being repaired repeatedly
• tool results not cached
• agents retrying bad tool calls

The first 20–40% savings often come from removing obvious waste, not from sophisticated infrastructure.

4. Token Reduction: The Cheapest Token Is the One You Never Send

Token reduction is the most universal optimization. It works whether you use hosted APIs, open-source models, local inference, RAG, or agents.

4.1 Shorten System Prompts

System prompts tend to grow over time. Every incident adds another instruction. Every stakeholder adds another requirement. Eventually the prompt becomes a policy document.

Review your system prompt like production code.

Ask:

• Is this instruction still needed?
• Is this repeated elsewhere?
• Can this be shorter?
• Does this belong in code instead of prompt text?
• Is this only needed for one task, not every task?

Bad pattern:

Every request receives the same giant universal prompt.

Better pattern:

Common base prompt + task-specific compact instructions.

Do not send instructions for summarization, classification, code generation, and customer support if the current task is only classification.

4.2 Reduce Retrieved Context

RAG systems often over-retrieve because teams are afraid of missing context. But sending too much context can hurt both cost and quality. The model may pay attention to irrelevant chunks, mix sources, or produce hedged answers.

Improve retrieval by:

• using better chunking
• using hybrid search when keyword matching matters
• reranking top candidates
• limiting final chunks
• removing duplicate chunks
• compressing retrieved context
• storing source metadata
• using query-specific retrieval settings

A practical rule: measure answer quality as you reduce the number of chunks. Many systems can drop from 10–20 chunks to 3–6 strong chunks with little quality loss, especially when reranking is used.

4.3 Summarize Long Conversation History

Chat history can become one of the largest costs in assistant products. Sending the full transcript on every turn is simple, but expensive.

Alternatives:

• rolling summary
• memory slots
• recent messages only
• task-specific state
• user profile facts
• retrieved prior events only when relevant

Do not treat conversation memory as a raw transcript problem. Treat it as a state management problem.

The model usually does not need every word the user said yesterday. It needs the current task, relevant preferences, unresolved decisions, and recent context.

4.4 Limit Output Length

Many products let the model talk too much.

For high-volume tasks, set strong defaults:

• short answer by default
• expand only when user asks
• bullet points instead of prose when appropriate
• strict JSON fields for extraction
• no redundant preamble
• no repeated safety disclaimers unless needed

Output token limits can improve both cost and UX. Users often prefer concise answers.

4.5 Avoid LLM Calls for Deterministic Work

Not every language-shaped task needs an LLM.

Use code for:

• formatting dates
• validating JSON
• checking permissions
• exact lookup
• arithmetic
• regex extraction when reliable
• simple routing rules
• template filling
• deduplication

One common anti-pattern is asking an LLM to do work that a database query, rule engine, or string function can do perfectly.

LLMs are useful for ambiguity. They are wasteful for deterministic operations.

5. Caching: The Most Obvious Win, and Still Easy to Get Wrong

Caching is the technique most people know first. If the same work repeats, reuse the result instead of paying again.

But LLM caching has more nuance than normal web caching because the answer may depend on prompt wording, model version, retrieved documents, user permissions, tool outputs, and time.

5.1 Exact-Match Caching

Exact-match caching stores a response for an identical request.

Cache key example:

hash(model + system_prompt_version + user_prompt + parameters)

This works well for:

• repeated FAQ answers
• static documentation questions
• deterministic transformations
• repeated benchmark/eval prompts
• common onboarding questions
• internal tools with repeated queries

Exact caching is safe when inputs are truly identical and the answer does not depend on fresh state.

5.2 Provider Prompt Caching

Some providers offer prompt caching, where repeated prompt prefixes can be processed more cheaply and quickly. OpenAI documents prompt caching as automatic for supported models, especially when requests share long common prefixes. This is useful when many requests begin with the same system prompt, examples, or stable context.

This matters because many apps send the same prefix repeatedly:

System instructions
Few-shot examples
Tool schemas
Long policy text
User-specific context
Current user question

If the stable prefix is structured consistently, provider-side caching can reduce input-token cost and latency for that prefix.

Practical implications:

• put stable instructions before variable content
• avoid randomly changing the beginning of prompts
• version prompts intentionally
• keep tool schemas stable when possible
• use provider cache keys or routing hints when available

Prompt caching does not eliminate all cost. It reduces cost for repeated prefixes. You still pay for uncached parts and outputs.

5.3 Semantic Caching

Semantic caching tries to reuse answers for similar questions, not just identical prompts.

Example:

Question A: How do I reset my password?
Question B: I forgot my password, what should I do?

These are different strings but likely the same intent. A semantic cache embeds the new query, searches prior cached queries, and reuses a response if similarity is high enough.

This can create major savings for support bots, documentation assistants, and internal knowledge tools. But it is riskier than exact caching.

The hard questions are:

• How similar is similar enough?
• What if two questions look similar but require different user-specific answers?
• What if the policy changed yesterday?
• What if the cached answer was generated by an older model?
• What if the retrieved source documents changed?
• What if a user should not see another user's answer?

Semantic caching is powerful, but unsafe without boundaries.

5.4 Cache Invalidation

Cache invalidation is the core production challenge.

An LLM cache may need invalidation when:

• the system prompt changes
• the model changes
• decoding parameters change
• source documents change
• user permissions change
• pricing or policy changes
• tool behavior changes
• external data becomes stale
• the product changes answer format

A good cache key includes versions:

model_version
prompt_version
retrieval_corpus_version
tool_version
user_or_tenant_scope
response_schema_version

For RAG answers, consider storing source document IDs and versions with each cached answer. If a source document changes, invalidate cached answers derived from it.

5.5 What Not to Cache

Avoid or be very careful caching:

• medical, legal, financial, or security-sensitive answers
• answers involving user-private data
• authorization-dependent responses
• rapidly changing data
• answers derived from live inventory, pricing, or status
• agent actions that mutate state
• low-confidence responses

A stale cached answer can be worse than an expensive fresh answer.

5.6 Cache Tool Results Too

Many teams cache final LLM answers but forget tool calls. Tool results often repeat:

• user profile lookup
• account metadata
• product catalog search
• policy documents
• code search results
• database query results
• web search results

Caching tool results reduces latency and can reduce the amount of context sent back to the model.

6. Model Routing: Stop Using Your Best Model for Everything

Model routing is one of the highest-leverage cost strategies. The idea is simple: classify the task, then choose the cheapest model that can handle it.

A single-model architecture is easy:

All requests -> best model

A routed architecture is more efficient:

Simple classification -> small cheap model
Extraction -> cheap structured-output model
Normal chat -> mid-tier model
Complex reasoning -> premium model
High-risk answer -> premium model + review

6.1 Routing by Task Type

Different tasks have different difficulty.

TaskOften Suitable ModelBoolean classificationsmall/cheap model or rulesSentiment taggingsmall modelEntity extractionsmall or mid modelShort summarizationcheap/mid modelFAQ answer from known contextcheap/mid modelComplex synthesispremium modelCoding/debuggingstrong modelAgent planningstrong modelHigh-stakes advicestrong model + safeguards

Routing starts with naming your task types. If every request is just “chat,” routing is hard. If you separate classification, extraction, summarization, answering, planning, and verification, routing becomes straightforward.

6.2 Routing by Difficulty

Some systems use a cheap model first, then escalate when needed.

Pattern:

Cheap model attempts answer
-> confidence check
-> if low confidence, call stronger model

The confidence check can use:

• model self-rating
• rule-based validation
• retrieval score
• output schema validity
• contradiction detection
• user tier
• task risk level

Be careful with self-reported confidence. Models can be overconfident. Use external signals when possible.

6.3 Routing by User or Business Value

Not all requests have equal business value.

A free-tier user asking a casual question may not justify the same cost as an enterprise admin asking for production incident help. This does not mean free users get bad answers. It means product economics matter.

Possible routing dimensions:

• subscription tier
• feature tier
• request urgency
• risk level
• account value
• internal vs external use
• human escalation availability

6.4 Routing Failure Modes

Model routing can reduce cost, but it can also silently reduce quality.

Common mistakes:

• routing too aggressively to weak models
• no eval set before switching models
• no fallback path
• using one global threshold for all tasks
• ignoring multilingual or edge-case users
• optimizing average cost while hurting important cases

A router must be evaluated like any other model component.

Measure:

• cost reduction
• answer quality
• escalation rate
• latency
• user satisfaction
• failure cases
• false cheap-route rate
• false premium-route rate

The best routers are boring. They send easy work to cheap models, hard work to strong models, and uncertain work through a fallback path.

7. Agent Cost Control: Prevent Infinite Interns

Agents are like interns with a corporate credit card. They can be very useful, but you need spending limits.

An agent that can reason, call tools, retry, and continue has an open-ended cost surface. If the agent gets confused, it may loop. If a tool returns bad data, it may keep trying. If the prompt encourages thoroughness, it may over-plan.

7.1 Set Hard Budgets

Every agent run should have limits:

max_turns
max_tool_calls
max_runtime_seconds
max_input_tokens
max_output_tokens
max_cost_usd
max_retries_per_tool

When the budget is reached, the agent should fail gracefully:

I could not complete this automatically because the search tool failed twice. Here is what I found and what I recommend next.

A graceful partial answer is better than a runaway loop.

7.2 Cache Tool Outputs

Agents often call the same tool repeatedly with similar arguments. Cache tool outputs inside the agent run and across runs when safe.

Example:

Agent searches docs for “refund policy”
Agent reasons
Agent searches docs for “refund policy” again

The second call should usually be a cache hit.

7.3 Detect Repeated Failures

If a tool call fails with the same error twice, do not let the agent blindly try five more times.

Add loop detection:

• same tool + same arguments
• same error message
• same observation repeated
• no new information after N turns
• plan does not change

Stop and summarize.

7.4 Use Smaller Models Inside the Agent

An agent may not need a premium model for every step.

Possible split:

• small model for classification
• small model for query rewriting
• strong model for planning
• tool calls handled by code
• mid model for summarizing tool output
• strong model for final high-quality answer

This is model routing inside the agent.

7.5 Avoid “Thinking Out Loud” in Production Output

Some models and prompts produce long reasoning traces. Depending on provider and configuration, this can increase tokens, latency, and risk. For production, ask for concise final outputs unless intermediate reasoning is needed for audit or debugging.

Keep traces in internal logs when needed, not in user-facing verbose text.

8. Quantization and Smaller Models

Quantization reduces the numeric precision used to store and compute model weights. In simple terms, it compresses the model so it uses less memory and may run faster or fit on cheaper hardware.

A rough intuition:

• FP32: high precision, large memory footprint
• FP16/BF16: common deep learning inference/training precision
• INT8 / 8-bit: lower memory, often small quality loss
• 4-bit: much smaller, more quality risk, often useful with the right model/task

Quantization matters because GPU memory is one of the main constraints in LLM serving. If a model uses less memory, you may be able to:

• fit a larger model on the same GPU
• run more concurrent requests
• use cheaper hardware
• reduce energy cost
• increase throughput

8.1 Quantization Is Not Free

Quantization introduces approximation. The model may become slightly worse, especially on tasks requiring precision, reasoning, math, code, or nuanced instruction following.

The quality impact depends on:

• model architecture
• model size
• quantization method
• calibration data
• hardware
• task type
• decoding settings

A 4-bit model may be excellent for classification and poor for complex reasoning. An 8-bit model may be nearly indistinguishable for some tasks and noticeably worse for others.

8.2 Where Quantization Works Best

Good candidates:

• high-volume internal tasks
• classification
• extraction
• short rewriting
• summarization of simple text
• routing decisions
• autocomplete-like tasks
• batch processing where small errors are tolerable

Riskier candidates:

• legal/medical/financial advice
• complex code generation
• math-heavy reasoning
• tasks with strict factuality requirements
• user-facing answers where brand quality matters

8.3 Evaluate Before and After

Never switch to a quantized model based only on benchmark claims.

Build an eval set from real production examples:

• common cases
• edge cases
• hard cases
• multilingual cases if relevant
• adversarial prompts
• long-context examples
• examples where the current model failed

Compare:

• answer quality
• format compliance
• latency
• throughput
• cost
• escalation rate
• user complaints

Quantization is a cost lever, not a religion. Use it where it preserves enough quality.

9. Local Inference and vLLM: Why People Keep Talking About It

vLLM comes up in cost conversations because it is one of the most important open-source tools for serving LLMs efficiently.

vLLM is not a model. It is an inference engine/server.

Think of the stack like this:

Model: Llama, Mistral, Qwen, DeepSeek, etc.
Serving engine: vLLM
Hardware: GPUs
API layer: OpenAI-compatible endpoint or custom service
Application: your product

If OpenAI or Anthropic is like renting intelligence through an API, vLLM is part of the machinery you might use when you run an open model yourself.

9.1 What Problem Does vLLM Solve?

LLM serving is difficult because generation is memory-intensive and request lengths vary. Each request needs a KV cache, which stores attention-related information from previous tokens so the model does not recompute everything from scratch.

The KV cache grows as generation continues. In a busy server, many users have different prompt lengths and output lengths. Managing that memory efficiently is hard.

Naive serving wastes GPU memory. Wasted GPU memory means fewer concurrent requests. Fewer concurrent requests means lower throughput. Lower throughput means higher cost per token.

vLLM became famous for PagedAttention, a memory-management approach for KV cache inspired by operating-system paging. Instead of reserving large contiguous memory blocks inefficiently, vLLM manages cache memory in blocks/pages, reducing fragmentation and improving utilization.

The practical result: more requests can be served on the same GPU before running out of memory.

9.2 Continuous Batching

LLM requests do not arrive neatly in fixed batches. Users arrive at random times. Some prompts are short. Some are long. Some outputs finish quickly. Some keep generating.

Continuous batching lets the serving engine dynamically batch active requests together as they arrive and finish. This improves GPU utilization.

Better utilization matters because GPU cost is mostly fixed while the machine is running. If your GPU is idle, you are paying for unused capacity. If vLLM helps the GPU process more tokens per second, the cost per token goes down.

9.3 OpenAI-Compatible API

vLLM can expose OpenAI-compatible APIs. That matters because many applications are already written around OpenAI-style chat completions.

This can reduce migration friction:

Before:
App -> OpenAI API

After:
App -> vLLM OpenAI-compatible server -> self-hosted model

It is not always a drop-in replacement. Models differ, outputs differ, tool calling support differs, and quality must be evaluated. But API compatibility makes experimentation much easier.

9.4 Prefix Caching and Throughput Optimizations

vLLM also supports performance features such as prefix caching and optimized scheduling. Prefix caching is useful when requests share common prompt prefixes, similar in spirit to provider prompt caching: repeated prefixes can be reused instead of recomputed.

This matters for applications with stable system prompts, few-shot examples, long tool schemas, or shared context.

9.5 When vLLM Saves Money

vLLM can help cut costs when:

• you have high request volume
• traffic is steady enough to keep GPUs busy
• an open model meets your quality requirements
• your team can operate GPU infrastructure
• throughput matters more than paying per request
• you can batch many requests
• you have workloads suitable for smaller or quantized models

In that world, the economics can shift from:

Pay provider per token

to:

Pay for GPU capacity and maximize tokens per second

If utilization is high, self-hosting can be cheaper.

9.6 When vLLM Does Not Save Money

Self-hosting is not automatically cheaper.

It may be worse when:

• traffic is low
• traffic is spiky
• GPUs sit idle
• engineering time is expensive
• reliability requirements are high
• hosted model quality is much better
• your team lacks ML infrastructure experience
• you need fast access to the newest frontier models

A hosted API includes more than inference. It includes model hosting, scaling, reliability, security programs, monitoring, updates, and operational expertise. When you self-host, you inherit much of that burden.

So the honest advice is:

Do not start with vLLM. Graduate to vLLM when measurement shows inference volume is high enough and model quality is acceptable.

vLLM is special because it makes local/open-model serving much more practical. It is not special because it magically makes all LLM usage cheap.

10. Batching and Asynchronous Work

Batching is another underused lever. Some providers offer cheaper batch APIs for non-real-time jobs. Self-hosted systems also benefit from batching because GPUs are more efficient when processing multiple requests together.

Good batch candidates:

• nightly document summarization
• offline classification
• embedding large corpora
• eval runs
• report generation
• support ticket tagging
• data enrichment

Bad batch candidates:

• interactive chat needing low latency
• user-facing autocomplete
• urgent incident response
• real-time voice interactions

A common cost mistake is treating every job as interactive. If the user does not need the answer now, batch it.

You can design product UX around this:

• “Your report will be ready in a few minutes.”
• “We’ll email you when analysis completes.”
• “Bulk processing runs overnight.”

Latency requirements drive cost. Relax latency when product experience allows it.

11. RAG Cost Optimization

RAG can either reduce or increase cost. It reduces cost when it lets a smaller model answer with relevant context. It increases cost when it dumps huge irrelevant context into the prompt.

11.1 Better Chunking

Bad chunking creates bad retrieval. If chunks are too small, the model lacks context. If chunks are too large, you waste tokens. If chunks split concepts awkwardly, retrieval returns fragments.

Good chunks often follow document structure:

• headings
• sections
• paragraphs
• code blocks
• tables
• policy clauses
• semantic boundaries

The goal is not a magic token size. The goal is chunks that represent coherent units of meaning.

11.2 Hybrid Search

Vector search is good for semantic similarity. Keyword search is good for exact terms, IDs, error messages, product names, and legal/policy language. Hybrid search combines both.

Hybrid retrieval can reduce cost by improving relevance. Better retrieval means fewer chunks need to be sent to the model.

11.3 Reranking

A reranker scores candidate chunks and selects the best ones. Reranking adds cost, but it can reduce final prompt size and improve quality.

The tradeoff:

Pay small reranking cost -> send fewer/better chunks -> reduce main LLM cost and improve answer

This is often worth it when the final model is expensive.

11.4 Context Compression

Instead of sending raw retrieved chunks, compress them:

• remove irrelevant paragraphs
• extract only answer-bearing sentences
• summarize long sections
• deduplicate overlapping chunks
• keep citations/source IDs

Be careful: compression can remove nuance. Evaluate it.

11.5 Retrieval Gating

Not every question needs RAG. If the user asks a general question or asks about the current conversation, retrieval may be unnecessary.

Add a retrieval gate:

Does this request need external knowledge?
If yes, retrieve.
If no, skip retrieval.

Skipping unnecessary retrieval saves embedding, search, reranking, and input tokens.

12. A Practical 10x Cost Reduction Plan

A realistic 10x reduction usually comes from stacking multiple improvements.

Here is a practical sequence.

Phase 1: Instrumentation

Add logging for tokens, model, cost, latency, retries, cache hits, and feature names.

Deliverable:

• cost dashboard
• top expensive features
• top expensive prompts
• cost per successful task

Expected savings: none immediately, but it reveals the path.

Phase 2: Remove Obvious Waste

Actions:

• shorten system prompts
• remove duplicate instructions
• reduce retrieved chunks
• cap output length
• stop sending full history
• replace deterministic LLM calls with code

Expected savings: 20–50% in many messy systems.

Phase 3: Add Caching

Actions:

• exact cache repeated prompts
• cache tool results
• cache embeddings
• use provider prompt caching where available
• add semantic cache for safe repeated intents
• version cache keys

Expected savings: highly workload-dependent. Repetitive support/docs workloads can see very large savings; bespoke creative workloads may see less.

Phase 4: Add Model Routing

Actions:

• classify tasks
• route simple tasks to cheaper models
• keep premium model for hard/high-risk tasks
• add fallback/escalation
• evaluate quality

Expected savings: often one of the largest levers, especially if everything currently uses a premium model.

Phase 5: Control Agents

Actions:

• max turns
• max tool calls
• retry budgets
• loop detection
• tool output caching
• smaller models for substeps

Expected savings: large for agentic systems; smaller for simple chat.

Phase 6: Evaluate Smaller and Quantized Models

Actions:

• build eval set
• test open models
• test quantized variants
• compare quality/cost/latency
• use for narrow tasks first

Expected savings: significant for suitable high-volume tasks.

Phase 7: Consider vLLM/Self-Hosting

Actions:

• estimate GPU cost
• estimate utilization
• benchmark tokens/sec
• test latency under load
• compare reliability and ops burden
• deploy only if economics make sense

Expected savings: potentially large at scale, but not guaranteed.

Example Compound Savings

Suppose a system costs $10,000/month.

A possible path:

OptimizationRemaining CostStarting point$10,000Prompt/context reduction saves 30%$7,000Caching saves 30% of remaining$4,900Model routing saves 40% of remaining$2,940Agent/retry controls save 20% of remaining$2,352Smaller models/self-hosting for selected workloads saves 50% of remaining$1,176

That is not one magic trick. It is layered engineering.

13. Cost Optimization Decision Table

TechniqueSavings PotentialLatency ImpactQuality RiskComplexityBest ForPrompt trimmingMediumBetterLowLowEveryoneOutput limitsMediumBetterLow/MediumLowChat, generationExact cachingHigh when repeatedBetterLow/MediumMediumFAQs, evals, docsSemantic cachingHighBetterMedium/HighMedium/HighSupport, docsTool result cachingMedium/HighBetterMediumMediumAgents, RAGProvider prompt cachingMedium/HighBetterLowLow/MediumStable prompt prefixesModel routingHighMixedMediumMediumMulti-task appsAgent budgetsHigh for agentsBetterLow/MediumMediumTool-using agentsRAG rerankingMediumWorse or mixedLow/MediumMediumKnowledge assistantsQuantizationMedium/HighBetter or mixedMediumHighSelf-hosted modelsvLLM self-hostingHigh at scaleMixedMediumHighHigh-volume workloadsBatch processingMedium/HighSlower by designLowMediumOffline jobs

14. Common Mistakes

Mistake 1: Optimizing Without Measurement

Teams jump to smaller models or self-hosting before knowing where spend comes from. This can waste weeks while obvious prompt bloat remains untouched.

Mistake 2: Treating Caching as Always Safe

Caching can leak stale, wrong, or unauthorized information. Cache keys and invalidation rules matter.

Mistake 3: Over-Retrieving in RAG

More context is not always better. Irrelevant context costs money and can confuse the model.

Mistake 4: Using Premium Models for Simple Tasks

A frontier model doing sentiment tagging at scale is usually a routing failure.

Mistake 5: Ignoring Output Tokens

Verbose outputs can be expensive. Concise defaults help.

Mistake 6: Letting Agents Run Without Budgets

Agents need limits. Otherwise failures become invoices.

Mistake 7: Self-Hosting Too Early

vLLM is powerful, but GPU infrastructure is not free. Low utilization kills the economics.

Mistake 8: No Evaluation Before Switching Models

A cheaper model that reduces conversion, increases support tickets, or creates factual errors may be more expensive in business terms.

15. A Simple Architecture for Cost-Controlled LLM Apps

A production cost-aware architecture might look like this:

User Request
  |
  v
Request Logger / Cost Context
  |
  v
Task Classifier
  |
  +--> Deterministic handler when possible
  |
  +--> Cache lookup
  |       |
  |       +--> Cache hit -> return
  |       +--> Cache miss
  |
  v
Retrieval Gate
  |
  +--> No retrieval
  +--> RAG pipeline: search -> rerank -> compress
  |
  v
Model Router
  |
  +--> cheap model
  +--> mid model
  +--> premium model
  +--> self-hosted model via vLLM
  |
  v
Output Validator
  |
  +--> success -> cache if safe -> return
  +--> failure -> repair/escalate within budget

This architecture makes cost a first-class concern without making cost the only concern.

The important design principle is that every expensive step has a gate:

• Do we need an LLM at all?
• Do we already know the answer?
• Do we need retrieval?
• How much context is enough?
• Which model is sufficient?
• Is this safe to cache?
• Should the agent continue?

Cost control is mostly about asking these questions automatically.

16. What to Say When Someone Asks “Why vLLM?”

Here is the plain-language answer:

vLLM matters because serving LLMs efficiently is hard, and vLLM solves important serving problems. It manages the KV cache efficiently with PagedAttention, batches requests continuously, supports high-throughput inference, and can expose an OpenAI-compatible API. That makes it one of the practical tools teams use when they want to run open-source models themselves.

But vLLM is not the first step in reducing cost. It is a later-stage option.

Use this phrasing:

vLLM is special because it helps you get more tokens per second out of your GPUs. It reduces memory waste during generation and improves batching, which lowers cost per token when you have enough volume. But it only saves money if self-hosting makes sense. If traffic is low or model quality is not good enough, hosted APIs may still be cheaper overall.

That answer is balanced and credible.

17. Final Checklist

If you want to cut your LLM bill, work through this checklist in order:

1. Log tokens, model, feature, cost, latency, retries, cache hits.
2. Build a cost dashboard.
3. Identify top expensive features and prompts.
4. Shorten prompts.
5. Reduce retrieved context.
6. Stop sending full chat history.
7. Cap output length.
8. Replace deterministic LLM calls with code.
9. Cache exact repeated requests.
10. Cache embeddings and tool results.
11. Use provider prompt caching-friendly prompt structure.
12. Add semantic caching only where safe.
13. Version cache keys and invalidation rules.
14. Split tasks by type.
15. Route easy tasks to cheaper models.
16. Add fallback to stronger models.
17. Put budgets on agents.
18. Detect loops and repeated tool failures.
19. Build evals before changing models.
20. Test smaller and quantized models.
21. Consider batch APIs for offline work.
22. Benchmark self-hosting only after volume justifies it.
23. Use vLLM if open-model serving economics are favorable.
24. Keep measuring after every change.

18. Conclusion

Cutting an LLM bill by 10x is possible, but not usually because of one trick. Caching helps. Model routing helps. Prompt trimming helps. Agent budgets help. Quantization helps. vLLM can help at scale. The real win comes from combining them in the right order.

The most important mindset shift is this: LLM inference is production infrastructure. It needs observability, routing, caching, budgets, evals, and capacity planning. A prototype can send everything to the best model and hope. A production system cannot.

Start with measurement. Remove waste. Cache safe repeated work. Route by difficulty. Control agent loops. Evaluate cheaper models. Only then consider local inference with vLLM.

The goal is not to make the system cheap at any cost. The goal is to spend intelligence where it matters.