LLM Costs: Cut Bills 73% with Semantic Cache

The Semantic Caching Revolution: How Understanding *Meaning* Can Slash Your LLM Costs

Imagine your LLM bill growing 30% month-over-month, even as traffic increases at a slower pace. Sound familiar? The culprit isn’t necessarily more users, but rather the frustrating reality that people ask the same questions in countless different ways. We discovered this firsthand: queries like “What’s your return policy?”, “How do I return something?”, and “Can I get a refund?” were each triggering separate, full-cost LLM calls for nearly identical responses. Traditional, exact-match caching only scratched the surface, capturing a mere 18% of these redundant requests. The solution? A shift from caching text to caching meaning – a technique we call semantic caching.

Why Exact-Match Caching Falls Short in the Age of LLMs

For years, caching has relied on hashing query text. If the exact phrase matched a previous request, the cached response was served. This works beautifully for static content or predictable queries. But Large Language Models (LLMs) are designed to understand natural language, and users use natural language – which is inherently variable. Our analysis of 100,000 production queries revealed a stark truth: only 18% were exact duplicates. A whopping 47% were semantically similar, representing a massive, untapped opportunity for cost savings. Each of these similar queries was needlessly hitting the LLM, generating responses that were essentially the same.

How Semantic Caching Works: Embedding Meaning into Your Cache

Semantic caching replaces text-based keys with embedding-based similarity lookup. Instead of comparing strings, we compare vector representations of the query’s meaning. Here’s a simplified look at the architecture:

class SemanticCache:
    def __init__(self, embedding_model, similarity_threshold=0.92):
        self.embedding_model = embedding_model
        self.threshold = similarity_threshold
        self.vector_store = VectorStore()  # FAISS, Pinecone, etc.
        self.response_store = ResponseStore()  # Redis, DynamoDB, etc.
def get(self, query: str) -> Optional[str]:
    """Return cached response if semantically similar query exists."""
    query_embedding = self.embedding_model.encode(query)
    # Find most similar cached query
    matches = self.vector_store.search(query_embedding, top_k=1)
    if matches and matches[0].similarity >= self.threshold:
        cache_id = matches[0].id
        return self.response_store.get(cache_id)
    return None

def set(self, query: str, response: str):
    """Cache query-response pair."""
    query_embedding = self.embedding_model.encode(query)
    cache_id = generate_id()
    self.vector_store.add(cache_id, query_embedding)
    self.response_store.set(cache_id, {
        'query': query,
        'response': response,
        'timestamp': datetime.utcnow()
    })





The core idea is simple: embed queries into a vector space and find cached queries that fall within a defined similarity threshold.  But that threshold is where things get tricky.
The Critical Role of the Similarity Threshold – And Why One Size Doesn’t Fit All
Setting the similarity threshold is a balancing act. Too high, and you miss valid cache hits. Too low, and you risk returning incorrect responses.  We initially set our threshold at 0.85, assuming 85% similarity meant “the same question.” We were wrong.  At that level, we saw cache hits for demonstrably different questions, leading to potentially misleading answers.  
We quickly learned that optimal thresholds vary significantly by query type.  For example:

• FAQ-style questions: 0.94 (High precision is crucial; incorrect answers erode trust)
• Product searches: 0.88 (More tolerance for near-matches)
• Support queries: 0.92 (Balance between coverage and accuracy)
• Transactional queries: 0.97 (Very low tolerance for errors)

Implementing query-type-specific thresholds, powered by a query classifier, was a game-changer.

Beyond Thresholds: Tuning for Precision and Recall

Blindly adjusting thresholds isn’t effective. We needed ground truth – a way to definitively determine if two queries had the “same intent.” Our methodology involved:

1. Sampling query pairs: We sampled 5,000 query pairs across various similarity levels (0.80-0.99).
2. Human labeling: Three annotators labeled each pair as “same intent” or “different intent,” with a majority vote determining the final label.
3. Precision/Recall curves: We calculated precision (of cache hits, what fraction had the same intent?) and recall (of same-intent pairs, what fraction did we cache-hit?) for each threshold.
4. Cost-based optimization: We optimized for precision for high-stakes queries (like FAQs) and recall for cost-sensitive queries (like product searches).

The Performance Impact: Cost Savings and Latency Considerations

The results were dramatic. After three months in production:

• Cache hit rate: Increased from 18% to 67% (+272%)
• LLM API costs: Decreased by 73% (from $47K/month to $12.7K/month)
• Average latency: Improved by 65% (from 850ms to 300ms)
• False-positive rate: Remained minimal at 0.8%

While semantic caching introduces a small amount of latency (around 20ms for embedding and vector search), the overall net latency improvement is significant, thanks to the reduced reliance on expensive LLM calls.

The Future of Semantic Caching: Staleness, Personalization, and Beyond

Semantic caching isn’t a “set it and forget it” solution. Maintaining cache quality requires ongoing attention to cache invalidation. We employ a three-pronged approach: time-based TTLs, event-based invalidation (triggered by content updates), and staleness detection (periodically re-running queries to verify response accuracy).

Looking ahead, we’re exploring several exciting avenues: integrating semantic caching with retrieval-augmented generation (RAG) pipelines to further enhance LLM accuracy, dynamically adjusting thresholds based on real-time user feedback, and incorporating personalization to avoid caching responses that should be tailored to individual users. The rise of multimodal LLMs will also necessitate expanding semantic caching to handle images, audio, and video, not just text.

The key takeaway is clear: in the era of LLMs, understanding the meaning behind user queries is no longer a luxury – it’s a necessity for controlling costs and delivering a high-quality user experience. What strategies are you implementing to optimize your LLM infrastructure? Share your thoughts in the comments below!