LLM Cost Optimization

Overview

LLM inference is expensive. A production application processing 1M requests/month can easily spend $10K-100K+ on API costs. This page covers the practical levers for reducing spend: token economics, caching, model routing, quantization, and batching. For deep dives on quantization and batching, see dedicated pages.

Token Economics

Pricing Structure

All major providers price by token, with output tokens costing 3-5× more than input tokens:

Prices as of mid-2025. Always verify at OpenAI, Anthropic, Google.

Cost Per Request Estimation

cost = (input_tokens × input_price + output_tokens × output_price) / 1_000_000

Key insight: For RAG-heavy applications, input tokens dominate cost (retrieved context). For generation-heavy applications (summarization, code generation), output tokens dominate. Optimize the dominant dimension first.

Caching Strategies

Prompt Caching (Provider-Side)

Providers cache the KV cache for repeated prompt prefixes, eliminating redundant compute:

Best practice: Structure prompts so the system prompt, few-shot examples, and tool definitions form a static prefix that is cached across requests. Only the user query and retrieved RAG context should vary per request. Reference: Anthropic Prompt Caching.

Semantic Caching (Application-Side)

Store LLM responses keyed by the semantic similarity of the input query. When a new query is semantically similar to a cached query, return the cached response instead of calling the LLM.

Query: "What is the refund policy?" → Cache miss → Call LLM → Cache result
Query: "How do I get a refund?" → Cosine sim > 0.95 → Cache hit → Return cached response

Trade-offs: Semantic caching saves cost and latency but risks stale responses when underlying data changes. Cache invalidation must be tied to data updates. Not suitable for highly personalized or time-sensitive queries.

Model Routing

Not every query needs the most expensive model. Route queries to the cheapest model that can handle them:

Router Implementation

# Simple router pattern
def route_query(query: str) -> str:
    intent = lightweight_classifier(query)  # GPT-4o-mini or local model
    if intent in ["simple_qa", "classification", "extraction"]:
        return "gpt-4o-mini"
    elif intent in ["complex_reasoning", "code_generation"]:
        return "gpt-4o"
    elif intent in ["creative_writing", "brainstorming"]:
        return "claude-3.5-sonnet"
    else:
        return "gpt-4o"  # default to capable model

Cascading routing is more sophisticated: try the cheapest model first, then fall back to a more expensive one if the output quality is insufficient (measured by a lightweight quality classifier). This maximizes savings while maintaining quality.

Quantization

Quantization reduces model precision (FP16 → INT8 → INT4) to shrink memory and compute requirements. Essential for self-hosted models; not applicable when using API providers.

See quantization.md for full details on methods, benchmarks, and implementation.

Batching

Batching processes multiple requests simultaneously, amortizing weight loading across the batch. This is the single most impactful optimization for self-hosted model throughput.

See batching.md for full details on continuous batching, PagedAttention, and implementation.

Cost Optimization Decision Framework

graph TD
    START[Need to reduce LLM costs?] --> FIRST{Biggest cost driver?}
    FIRST -->|API provider| CACHING[Prompt caching + semantic caching]
    FIRST -->|Self-hosted| QUANT[Quantization + batching]
    FIRST -->|Mixed workload| ROUTING[Model routing]

    CACHING --> TOKENS[Reduce input tokens: compress prompts, fewer RAG chunks]
    QUANT --> SERVE[Optimize serving: vLLM, PagedAttention, FP8]
    ROUTING --> EVAL[Evaluate: can smaller models handle some queries?]

    TOKENS --> MONITOR[Monitor: cost per request, budget alerts]
    SERVE --> MONITOR
    EVAL --> MONITOR

Interview Questions

Q1: How would you reduce LLM costs for a production RAG application spending $50K/month?

Answer: Systematic approach: (1) Profile: Identify where tokens are spent (input vs output, which queries are most expensive). (2) Prompt caching: Structure prompts with static prefix (system prompt + examples) for provider-side caching (50-90% savings on cached tokens). (3) Model routing: Classify query complexity; route simple queries to a cheaper model (GPT-4o-mini at 15× cheaper). (4) Reduce context: Rerank aggressively to use fewer retrieved chunks (5 instead of 15); truncate conversation history. (5) Semantic caching: Cache responses for common/similar queries. (6) Output length control: Set max_tokens and use concise system prompts. (7) Monitor: Track cost per request and set budget alerts. Expected savings: 50-80% without quality degradation.

Q2: When would you choose self-hosting over API-based LLMs?

Answer: Self-host when: (1) Data privacy requires keeping data on-premises (healthcare, finance, defense). (2) Volume is high enough that GPU costs are cheaper than API costs (typically >1B tokens/month for a 7B model). (3) Low latency requirements make API round-trips unacceptable (sub-100ms TTFT). (4) Customization requires fine-tuned models not available through APIs. (5) Vendor independence is critical (avoid lock-in, ensure availability). Trade-offs: self-hosting requires ML infrastructure expertise, GPU procurement, model serving engineering, and operational overhead. For most startups, APIs are cheaper until significant scale.

Q3: Explain prompt caching and semantic caching. When would you use each?

Answer: Prompt caching (provider-side): The LLM provider caches the KV cache for repeated prompt prefixes. If multiple requests share the same system prompt, the provider skips recomputing those attention layers. No code changes needed for OpenAI (automatic). Anthropic requires explicit cache markers. Best for: applications with large static system prompts and many concurrent requests. Semantic caching (application-side): Your application stores LLM responses keyed by query embeddings. When a new query is semantically similar (cosine similarity > threshold), return the cached response. Best for: FAQ-like traffic where many users ask similar questions. Key difference: prompt caching reduces compute cost per request at the provider level; semantic caching eliminates the LLM call entirely.

Q4: How does output token pricing affect system design decisions?

Answer: Output tokens cost 3-5× more than input tokens, so output length is the dominant cost factor for generation tasks. System design implications: (1) Set max_tokens to the minimum needed — don’t default to the model’s maximum. (2) Use concise system prompts (“respond in under 100 words”) to constrain output length. (3) For extraction/classification tasks, use structured outputs (JSON Schema) which produce minimal, predictable-length outputs. (4) For summarization, specify target length explicitly. (5) Monitor average output tokens per query type and set alerts on increases. A 20% increase in average output length can increase costs by 20-50% depending on the input/output ratio.

References

1. OpenAI Pricing — https://platform.openai.com/pricing
2. Anthropic Pricing — https://docs.anthropic.com/en/docs/about-claude/pricing
3. Anthropic, “Prompt Caching” — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
4. Google AI Pricing — https://ai.google.dev/pricing
5. LiteLLM — https://github.com/BerriAI/litellm
6. GPTCache — https://github.com/zilliztech/GPTCache

Cross-References

• Quantization → Full quantization methods and benchmarks
• Batching → Continuous batching and PagedAttention
• Inference → Production inference optimization
• vLLM → High-throughput model serving
• Prompt Caching → Caching in prompt design
• RAG Cost → RAG-specific cost optimization