Semantic caching is a technique that stores LLM responses keyed by meaning rather than exact text, allowing your application to return cached answers for queries that are semantically similar to previous ones. If you’re running LLM calls in production, you’ve likely seen costs climb with scale and latency spike during peak traffic. Semantic caching addresses both problems by intercepting similar queries before they hit your LLM provider.
In this guide, you’ll learn how semantic caching works under the hood, where it fits in your LLM or RAG pipeline, and how to implement it without breaking your application. We’ll cover architecture patterns, threshold tuning, and the pitfalls that catch most teams off guard.
Table of Contents
- What Is Semantic Caching for LLMs?
- Why Semantic Caching Matters for LLM Applications
- Core Building Blocks of a Semantic Cache
- Where Semantic Caching Fits in an LLM/RAG Pipeline
- Designing an Effective Semantic Caching Strategy
- Implementation Patterns
- Evaluating and Tuning Your Semantic Cache
- Advanced Techniques for Semantic Caching
- Common Pitfalls and How to Avoid Them
- Practical Checklist for Implementing Semantic Caching
What Is Semantic Caching for LLMs?
Semantic caching stores LLM responses indexed by vector embeddings of the input query, then retrieves cached responses when a new query is semantically similar to a previously seen one. Unlike traditional key-value caching that requires an exact string match, semantic caching uses embedding similarity to determine cache hits.
Here’s the core difference. Traditional caching works like a dictionary lookup:
Query: "What is Python?"
Cache key: hash("What is Python?")
Result: Cache HIT only if query is exactly "What is Python?"Semantic caching works differently:
Query: "Tell me about Python"
Cache key: embedding("Tell me about Python")
Similarity check: cosine_similarity(query_embedding, cached_embeddings)
Result: Cache HIT if similarity > threshold (e.g., 0.92)The second approach catches variations like “What is Python programming?”, “Explain Python to me”, and “Can you describe Python?” that would all miss in a traditional cache but share the same semantic intent.
This matters because natural language is inherently variable. Users ask the same question in dozens of different ways. A traditional cache sees each variation as a completely new request, forcing an expensive LLM call every time. A semantic cache recognizes these as similar enough to reuse the same response.
Why Semantic Caching Matters for LLM Applications
Semantic caching reduces both latency and cost by avoiding redundant LLM calls for queries your system has effectively already answered. The impact is substantial and well-documented.
Cost savings: AWS benchmarks using Amazon ElastiCache as a semantic cache with Amazon Bedrock showed an 86% reduction in LLM inference costs. Research on the GPT Semantic Cache approach reported up to 80% reduction in LLM usage costs by reusing responses for similar queries.
Latency improvements: The same AWS study demonstrated an 88% improvement in average end-to-end latency. Retrieving a cached response takes milliseconds compared to the 500ms to 2 seconds typical of LLM API calls.
Throughput gains: The VectorQ framework, which uses adaptive similarity thresholds, achieved up to 12x increases in cache hit rates compared to static threshold approaches. Higher hit rates mean more requests served from cache, which directly translates to handling more concurrent users without scaling your LLM infrastructure.
These numbers compound over time. Consider a customer support chatbot handling 100,000 queries per day. If 40% of those queries are semantically similar to previous questions (common for FAQ-style traffic), semantic caching can cut your monthly LLM bill by thousands of dollars while making your application feel significantly faster.
The benefits scale with traffic patterns. Applications with repetitive query patterns (support bots, documentation search, educational platforms) see the highest returns. Applications with highly unique, context-dependent queries (creative writing, personalized analysis) see lower but still meaningful improvements.
Core Building Blocks of a Semantic Cache
A semantic cache consists of three core components that work together: an embedding model, a vector store, and a cache management layer. Understanding each component helps you make better architecture decisions.
Embedding Model
The embedding model converts text queries into dense vector representations that capture semantic meaning. When a user submits “How do I reset my password?”, the embedding model produces a vector like [0.023, -0.156, 0.891, ...] with hundreds or thousands of dimensions.
Popular choices include:
- OpenAI text-embedding-3-small: 1536 dimensions, good balance of quality and cost
- Cohere embed-v3: Strong multilingual support
- sentence-transformers/all-MiniLM-L6-v2: Open-source, runs locally, 384 dimensions
- Voyage AI voyage-large-2: High accuracy for technical content
The embedding model is the most critical choice. A weak embedding model produces vectors that don’t capture semantic similarity well, leading to cache misses on similar queries and false hits on unrelated ones.
Vector Store
The vector store holds your cached query embeddings and enables fast similarity search. When a new query comes in, you need to find the most similar cached queries in milliseconds.
Common options include:
- Redis with vector search: Low latency, good for real-time applications, managed options available
- Pinecone: Fully managed, scales automatically, built for production
- Qdrant: Open-source, strong filtering capabilities
- pgvector (PostgreSQL): Use your existing database, simpler ops
- Azure Cosmos DB: Integrated vector search with Microsoft ecosystem
For most applications, Redis or pgvector provides the best starting point. You probably already run one of these, and adding vector search is straightforward. Dedicated vector databases like Pinecone or Qdrant make sense at scale or when you need advanced features like hybrid search.
Cache Management Layer
The cache management layer handles the logic around what to store, when to evict, and how to handle metadata. This includes:
- Similarity threshold: The minimum cosine similarity score required for a cache hit (typically 0.85 to 0.95)
- TTL (Time-to-Live): How long cached entries remain valid before automatic expiration
- Eviction policy: LRU (Least Recently Used), LFU (Least Frequently Used), or custom logic
- Metadata storage: Timestamps, usage counts, user/tenant IDs, model version used
The cache management layer is where most customization happens. A support chatbot might use aggressive caching with a 0.88 threshold and 7-day TTL. A financial analysis tool might use conservative caching with a 0.95 threshold and 1-hour TTL.
Where Semantic Caching Fits in an LLM/RAG Pipeline
Semantic caching sits between the user’s request and your LLM or RAG system, intercepting queries that can be served from cache before they trigger expensive computations. The placement matters because it determines what you’re caching and how much work you save.
The Basic Request Flow
Here’s how a request flows through a system with semantic caching:
- User query arrives: “How do I connect to the database?”
- Query normalization: Strip whitespace, lowercase, remove filler words (optional)
- Embed the query: Generate vector embedding of the normalized query
- Semantic cache lookup: Search vector store for similar cached queries
- Cache hit path: If similarity > threshold, return cached response immediately
- Cache miss path: If no match, proceed to LLM/RAG system
- Generate response: LLM produces the answer
- Cache write: Store query embedding and response for future reuse
- Return response: Send answer to user
The cache intercept happens early, before any RAG retrieval or LLM inference. This means a cache hit skips not just the LLM call but also any document retrieval, re-ranking, or prompt construction.
[MISS] | |
Return Cached RAG/LLM Call
Response |
Generate Response
|
Cache Write
|
Return ResponseUse color coding: green for cache hit path (fast), orange for cache miss path (slow) Include latency estimates on each path —>
What to Cache: Variants and Trade-offs
You can place semantic caching at different points in your pipeline, each with different trade-offs:
Caching raw LLM completions: Store the direct output from your LLM. Simple to implement, but cached responses don’t reflect updated system prompts or model changes.
Caching RAG final answers: Store the complete answer after retrieval and generation. Saves the most compute, but cached answers may become stale if your knowledge base updates frequently.
Caching retrieved contexts: Store the documents retrieved for a query, then regenerate the answer. Balances freshness with cost savings, since retrieval is often cheaper than generation.
Caching tool/agent outputs: For agentic systems, cache the results of tool calls (API responses, database queries). Particularly effective when tools have rate limits or high latency.
Most teams start by caching final answers, which provides the biggest immediate impact. As your system matures, you might add layered caching at multiple points.
Designing an Effective Semantic Caching Strategy
An effective semantic caching strategy requires deciding what to cache, how aggressively to match queries, and how to scope the cache for different users or contexts. Getting these decisions wrong leads to either low hit rates (wasted effort) or incorrect cached responses (user frustration).
Choosing What to Cache
Not all queries benefit equally from caching. Focus on:
High-frequency, stable queries: FAQ-style questions that many users ask and whose answers don’t change often. “What are your business hours?” or “How do I reset my password?”
Expensive operations: Queries that trigger complex RAG retrieval, multi-step reasoning, or tool calls. The more expensive the miss, the more valuable the hit.
Domain-specific knowledge: Technical documentation, product specifications, policy information. These tend to be stable and frequently queried.
Avoid caching: Personalized queries (“What’s in my cart?”), time-sensitive information (“What’s the current stock price?”), or creative/generative tasks where variety is the point.
Similarity Threshold Tuning
The similarity threshold is the most important parameter in your semantic cache. It determines the trade-off between hit rate and precision.
- Threshold too low (0.70-0.80): High hit rate, but queries get matched to cached responses that don’t actually answer them. Users see irrelevant or wrong answers.
- Threshold too high (0.95-0.99): High precision, but most queries miss the cache. You’re paying for embedding computation without getting cache benefits.
- Sweet spot (0.85-0.92): Catches genuinely similar queries while avoiding false matches. The exact value depends on your embedding model and domain.
Start conservative (0.90+) and lower the threshold gradually while monitoring quality. It’s easier to increase hit rate than to recover user trust after serving wrong cached answers.
Cache Scope and Personalization
Decide whether your cache is global (shared across all users) or scoped (per-user, per-tenant, per-session):
Global cache: All users share the same cache. Maximizes hit rate but only safe for non-personalized, factual queries. Works well for documentation search or product FAQs.
Tenant-scoped cache: Each organization or workspace has its own cache. Appropriate for B2B SaaS where different customers have different knowledge bases or configurations.
User-scoped cache: Each user has their own cache. Necessary when responses depend on user-specific context. Lower hit rates since cache isn’t shared.
Session-scoped cache: Cache only within a single conversation. Useful for multi-turn conversations where earlier context affects later answers.
Most applications use a combination: a global cache for factual queries plus user-scoped caching for personalized interactions.
Implementation Patterns
Let’s look at concrete implementation patterns for semantic caching, from a basic setup to production-ready architectures.
Basic Semantic Cache for a Chat LLM
Here’s a minimal implementation using sentence-transformers for embeddings and a simple in-memory vector store:
from sentence_transformers import SentenceTransformer
import numpy as np
from openai import OpenAI
class SemanticCache:
def __init__(self, threshold: float = 0.90):
self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
self.threshold = threshold
self.cache = [] # List of (embedding, query, response)
def _embed(self, text: str) -> np.ndarray:
return self.encoder.encode(text, normalize_embeddings=True)
def _find_similar(self, query_embedding: np.ndarray):
best_score = 0
best_entry = None
for embedding, query, response in self.cache:
# Cosine similarity (embeddings are normalized)
score = np.dot(query_embedding, embedding)
if score > best_score:
best_score = score
best_entry = (query, response)
if best_score >= self.threshold:
return best_entry, best_score
return None, best_score
def get_or_generate(self, query: str, client: OpenAI) -> str:
query_embedding = self._embed(query)
# Check cache
cached, score = self._find_similar(query_embedding)
if cached:
print(f"Cache HIT (similarity: {score:.3f})")
return cached[1] # Return cached response
# Cache miss: call LLM
print(f"Cache MISS (best similarity: {score:.3f})")
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": query}]
)
answer = response.choices[0].message.content
# Store in cache
self.cache.append((query_embedding, query, answer))
return answer
# Usage
cache = SemanticCache(threshold=0.88)
client = OpenAI()
# First call: cache miss
answer1 = cache.get_or_generate("What is Python?", client)
# Second call: cache hit (similar query)
answer2 = cache.get_or_generate("Tell me about Python", client)This implementation works for prototyping but has limitations: linear search (slow at scale), no persistence, no TTL. Let’s fix those.
Semantic Caching with Redis
For production, use Redis with its vector search capabilities:
import redis
from redis.commands.search.field import VectorField, TextField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.commands.search.query import Query
from sentence_transformers import SentenceTransformer
import numpy as np
import json
import hashlib
class RedisSemanticCache:
def __init__(
self,
redis_url: str = "redis://localhost:6379",
threshold: float = 0.90,
ttl_seconds: int = 86400 # 24 hours
):
self.client = redis.from_url(redis_url)
self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
self.threshold = threshold
self.ttl = ttl_seconds
self.index_name = "semantic_cache"
self.vector_dim = 384 # MiniLM dimension
self._create_index()
def _create_index(self):
try:
self.client.ft(self.index_name).info()
except redis.ResponseError:
# Create index if it doesn't exist
schema = (
TextField("query"),
TextField("response"),
VectorField(
"embedding",
"FLAT",
{
"TYPE": "FLOAT32",
"DIM": self.vector_dim,
"DISTANCE_METRIC": "COSINE"
}
)
)
self.client.ft(self.index_name).create_index(
schema,
definition=IndexDefinition(
prefix=["cache:"],
index_type=IndexType.HASH
)
)
def _embed(self, text: str) -> bytes:
embedding = self.encoder.encode(text, normalize_embeddings=True)
return embedding.astype(np.float32).tobytes()
def lookup(self, query: str) -> tuple[str | None, float]:
query_embedding = self._embed(query)
# Vector similarity search
q = (
Query(f"*=>[KNN 1 @embedding $vec AS score]")
.return_fields("query", "response", "score")
.dialect(2)
)
results = self.client.ft(self.index_name).search(
q, query_params={"vec": query_embedding}
)
if results.docs:
doc = results.docs[0]
similarity = 1 - float(doc.score) # Convert distance to similarity
if similarity >= self.threshold:
return doc.response, similarity
return None, 0.0
def store(self, query: str, response: str):
key = f"cache:{hashlib.md5(query.encode()).hexdigest()}"
embedding = self._embed(query)
self.client.hset(
key,
mapping={
"query": query,
"response": response,
"embedding": embedding
}
)
self.client.expire(key, self.ttl)
# Usage with any LLM client
cache = RedisSemanticCache(threshold=0.88, ttl_seconds=3600)
def ask_with_cache(query: str, llm_client) -> str:
# Check cache first
cached_response, similarity = cache.lookup(query)
if cached_response:
return cached_response
# Generate and cache
response = llm_client.generate(query)
cache.store(query, response)
return responseSemantic Caching in RAG Systems
For RAG applications, integrate semantic caching before the retrieval step:
class RAGWithSemanticCache:
def __init__(self, cache: RedisSemanticCache, retriever, llm_client):
self.cache = cache
self.retriever = retriever
self.llm = llm_client
def query(self, question: str) -> dict:
# Step 1: Check semantic cache
cached, similarity = self.cache.lookup(question)
if cached:
return {
"answer": cached,
"source": "cache",
"similarity": similarity
}
# Step 2: Retrieve relevant documents
docs = self.retriever.get_relevant_documents(question)
# Step 3: Generate answer with context
context = "\n\n".join([doc.page_content for doc in docs])
prompt = f"""Answer based on the context below.
Context:
{context}
Question: {question}
Answer:"""
answer = self.llm.generate(prompt)
# Step 4: Cache the answer
self.cache.store(question, answer)
return {
"answer": answer,
"source": "generated",
"documents": docs
}Productionizing as an API Gateway Layer
For larger deployments, implement semantic caching as a middleware or API gateway plugin:
from fastapi import FastAPI, Request
from pydantic import BaseModel
app = FastAPI()
class CompletionRequest(BaseModel):
model: str
messages: list[dict]
class CacheMiddleware:
def __init__(self, cache: RedisSemanticCache):
self.cache = cache
def extract_cache_key(self, request: CompletionRequest) -> str:
# Use the last user message as cache key
user_messages = [
m["content"] for m in request.messages
if m["role"] == "user"
]
return user_messages[-1] if user_messages else ""
async def process(self, request: CompletionRequest, call_next):
cache_key = self.extract_cache_key(request)
if not cache_key:
return await call_next(request)
# Check cache
cached, similarity = self.cache.lookup(cache_key)
if cached:
return {"choices": [{"message": {"content": cached}}], "cached": True}
# Forward to LLM
response = await call_next(request)
# Cache the response
if "choices" in response:
answer = response["choices"][0]["message"]["content"]
self.cache.store(cache_key, answer)
return responseEvaluating and Tuning Your Semantic Cache
A semantic cache is only valuable if it’s actually helping. You need metrics to know whether it’s working and instrumentation to tune it over time.
Key Metrics to Track
Cache hit rate: The percentage of queries served from cache. Target 30-60% for most applications. Below 20% suggests your threshold is too high or your traffic patterns don’t benefit from caching.
hit_rate = cache_hits / total_queriesLatency distribution: Compare P50, P95, and P99 latency for cache hits vs. misses. Cache hits should be 10-50x faster.
Cache hit P50: 45ms
Cache miss P50: 850msToken/cost savings: Track tokens saved by cache hits. Multiply by your per-token cost to get dollar savings.
tokens_saved = sum(cached_response_tokens for each cache_hit)
cost_saved = tokens_saved * cost_per_tokenError/mismatch rate: The percentage of cache hits where the cached response was inappropriate. Requires human evaluation or automated quality checks. Should be below 2%.
Quality Checks
Numbers alone don’t tell you if cached responses are appropriate. Implement quality checks:
Human spot checks: Regularly review random samples of cache hits. Flag cases where the cached response doesn’t adequately answer the query.
Automated semantic similarity on responses: Compare the cached response to what a fresh LLM call would produce. Large divergence suggests a bad cache match.
User feedback signals: Track thumbs-down ratings, follow-up clarification questions, or repeat queries. These indicate the cached response didn’t satisfy the user.
Iteration Loop
Use your metrics to tune the cache:
- Start conservative: Threshold at 0.92, short TTL (1 hour)
- Monitor for 1-2 weeks: Gather baseline metrics
- Lower threshold gradually: Drop to 0.90, then 0.88, watching error rate
- Extend TTL: If content is stable, increase to 24 hours or longer
- Segment analysis: Check if certain query types have higher error rates
- Repeat: Continuous tuning as traffic patterns evolve
Advanced Techniques for Semantic Caching
Once your basic semantic cache is working, these advanced techniques can push hit rates higher and reduce false positives.
Domain-Specific Embedding Models
General-purpose embedding models like OpenAI’s or sentence-transformers work well out of the box, but domain-specific embeddings can significantly improve cache precision.
If your application focuses on medical content, legal documents, or code, consider:
- Fine-tuning an open-source embedding model on your domain data
- Using domain-specific models (e.g., CodeBERT for code, BioBERT for biomedical text)
- Training a small adapter layer on top of a general embedding model
Domain-specific embeddings produce tighter clusters for semantically similar queries in your domain, reducing false positive cache hits from unrelated queries that happen to use similar general vocabulary.
Synthetic Data for Threshold Calibration
Determining the optimal similarity threshold is challenging. Synthetic data helps:
- Generate paraphrase pairs from your actual queries using an LLM
- Label pairs as “should match” or “should not match”
- Compute similarity scores for all pairs
- Find the threshold that maximizes correct matches while minimizing false positives
# Generate paraphrases for calibration
def generate_calibration_data(queries: list[str], llm_client) -> list[dict]:
calibration_pairs = []
for query in queries:
# Generate a paraphrase (should match)
paraphrase = llm_client.generate(
f"Rephrase this question differently: {query}"
)
calibration_pairs.append({
"query1": query,
"query2": paraphrase,
"should_match": True
})
# Generate an unrelated query (should not match)
unrelated = llm_client.generate(
f"Generate a completely different question about a different topic than: {query}"
)
calibration_pairs.append({
"query1": query,
"query2": unrelated,
"should_match": False
})
return calibration_pairsHybrid and Hierarchical Caching
Combine multiple caching strategies for better coverage:
Exact + Semantic: Check for exact string match first (fastest), then fall back to semantic similarity. Catches repeated queries instantly while still benefiting from semantic matching.
Hierarchical: Use multiple thresholds. At 0.98+ similarity, return cached response directly. At 0.90-0.98, return cached response but flag for async quality check. Below 0.90, treat as cache miss.
Query-type routing: Route different query types to different caching strategies. Factual questions use aggressive semantic caching. Personalized questions bypass the cache entirely.
Common Pitfalls and How to Avoid Them
Semantic caching can backfire if you’re not careful. Here are the most common mistakes and how to avoid them.
Stale Responses from Domain Drift
The problem: Your knowledge base updates, but the cache still serves old answers. A user asks about pricing, and they get last month’s prices because the query matched a stale cached response.
The fix:
- Set appropriate TTLs based on content volatility. Pricing information might need a 1-hour TTL; general documentation might tolerate 7 days.
- Implement cache invalidation hooks. When you update your knowledge base, invalidate related cache entries.
- Version your cache. Include model version and knowledge base version in cache keys so updates automatically miss the old cache.
Over-Aggressive Thresholds
The problem: You set the threshold too low to maximize hit rate, and now users are getting wrong answers. “How do I cancel my subscription?” matches “How do I upgrade my subscription?” and users get instructions for upgrading when they want to cancel.
The fix:
- Start conservative (0.92+) and lower gradually while monitoring quality
- Implement query-specific thresholds. Some query types tolerate lower thresholds than others.
- Add a confidence backoff: if the cached response seems off-topic based on keyword overlap, treat it as a miss even if similarity is high.
Ignoring Privacy and Multi-Tenancy
The problem: User A’s personalized query gets cached, and User B receives User A’s personalized answer because the queries were semantically similar.
The fix:
- Scope caches appropriately. User-specific queries need user-specific cache keys.
- Never cache queries containing PII or sensitive data without proper scoping.
- Implement tenant isolation for B2B applications. Each organization should have its own cache namespace.
- Audit cached content regularly for accidentally stored sensitive information.
Caching Non-Deterministic or Creative Outputs
The problem: You cache creative writing prompts or brainstorming queries. Users expect variety but get the same cached response every time.
The fix:
- Identify query types that shouldn’t be cached (creative, generative, exploratory)
- Use query classification to route non-cacheable queries around the cache
- For some use cases, cache multiple responses per query and randomly select from them
Practical Checklist for Implementing Semantic Caching
Ready to implement semantic caching? Here’s your action plan:
Phase 1: Foundation
- Choose an embedding model: Start with
text-embedding-3-small(OpenAI) orall-MiniLM-L6-v2(local) - Select a vector store: Redis with vector search, pgvector, or Pinecone
- Define cache scope: Global, tenant-scoped, or user-scoped based on your application
- Set initial threshold: Start at 0.90-0.92 (conservative)
- Configure TTL: Start with 1-hour TTL, adjust based on content stability
Phase 2: Implementation
- Build the cache lookup flow: Embed query, search similar, check threshold
- Add cache write on miss: Store embedding, query, and response
- Instrument metrics: Hit rate, latency, token savings
- Add logging: Log cache hits/misses with similarity scores for debugging
Phase 3: Rollout
- Start with one high-traffic, stable endpoint: FAQ bot, documentation search
- Shadow mode first: Log what would be cached without serving from cache
- Gradual rollout: 10% of traffic, then 50%, then 100%
- Monitor quality: Review cached responses, check user feedback
Phase 4: Optimization
- Tune threshold: Lower gradually while monitoring error rate
- Extend TTL: If content is stable, increase cache duration
- Add hybrid caching: Exact match layer in front of semantic layer
- Consider domain-specific embeddings: If false positives persist
Conclusion
Semantic caching transforms how LLM applications handle scale. By recognizing that “What is Python?” and “Tell me about Python” are the same question, you can serve the second query in milliseconds instead of seconds, at zero token cost instead of hundreds.
The key takeaways:
- Semantic caching matches by meaning, using embedding similarity instead of exact string matching
- Real-world impact is substantial: 80-88% cost reduction and similar latency improvements are achievable
- Start conservative: High threshold (0.90+), short TTL, narrow scope. Expand once you’ve validated quality.
- Measure everything: Hit rate, latency distribution, and error rate tell you if the cache is helping or hurting
- Watch for pitfalls: Stale responses, over-aggressive matching, and privacy leaks are the common failure modes
If you’re spending significant budget on LLM API calls and seeing repetitive query patterns, semantic caching should be near the top of your optimization list. The implementation isn’t complex, and the payoff compounds with every similar query you serve from cache.
AI gateways like ScaleMind, Portkey, and Helicone include semantic caching as a built-in feature, letting you enable it without building the infrastructure yourself. Whether you build or buy, the economics of semantic caching are hard to ignore once you’re running LLMs in production.