How Long Does LLM API Caching Last? Choosing Between 5 Minutes and 1 Hour

Prompt caching in LLM APIs typically offers two retention durations: 5 minutes and 1 hour. The choice depends on the median interval between consecutive requests in the same session, not on context length. Starting August 2026, some providers have launched explicit Prompt Cache Retention, allowing you to specify 5 minutes or 1 hour; Anthropic defaults to 5 minutes with extension to 1 hour; Google Gemini's explicit cache defaults to 1 hour. However, TTL is the maximum retention limit, not a guarantee, as actual eviction is influenced by node scheduling.

First Distinguish: Hit Rate and Retention Duration Solve Different Problems

Many people confuse two concepts: cache hit rate is determined by the stability of the prompt prefix, while retention duration is determined by your request cadence. If the prefix structure is unchanged but two requests are 10 minutes apart, the 5-minute tier will expire; conversely, if requests are 3 seconds apart but the prefix changes each time, a longer TTL won't help. So before choosing a tier, measure: the distribution of intervals between consecutive requests in the same session.

Four Key Points in the Cache Lifecycle: Write, Hit, Refresh, and Expiry

Using Anthropic as an example, the cache lifecycle has four key points: the first request writes to cache (billed at approximately 1.25 times the base input price); subsequent requests with matching prefixes hit (cache-read at about 10% of the price); a hit refreshes the sliding window (automatically extending 5 minutes); and missing beyond the TTL means expiry. Note that refresh semantics vary by provider: Anthropic resets the sliding window on hit, while Google Gemini's explicit cache is evicted based on a fixed TTL, requiring explicit calls to update the cache. When migrating, verify each provider's behavior.

Selecting a Tier Based on Request Intervals: Testing Methods for Three Traffic Patterns

Export the time differences between consecutive requests in the same session from your gateway logs, and calculate P50, P75, and P90. Specifically: group by session_id, take the timestamp differences between adjacent requests, remove cross-session noise exceeding 30 minutes, and compute percentiles separately for conversational, agent, and batch traffic. Decision rule: if P90 is less than the short tier duration, no explicit extension is needed. Human-bot conversations are influenced by user thinking time, with discrete intervals, so use P90 for judgment; agent loops are at second to tens of seconds, typically 5 minutes is sufficient; batch tasks span hours, and both tiers may be inadequate. See the table below:

5 Minutes or 1 Hour: Weighing Hit Probability, Discounts, and Write Premium

Estimate net benefit with this formula: Net benefit ≈ Hit probability × Cached tokens × (Base input price − Cache-read price) − Write premium. Write premium = Cached tokens × (1.25 − 1) × Base input price. The numbers below are examples; replace with actual prices from your model's pricing page. Example: assume base input price $5/M, cache hit 100K tokens, hit rate 70%, saving ≈ $0.315, write premium ≈ $0.125, net saving ≈ $0.19. For specifics, refer to each provider's pricing page. Also see AI API Price.

Prompt Structure Determines Cacheability: Fixed Prefix First, Variables Last

When arranging prompts, keep system prompts and tool definitions at the top and byte-level stable; place retrieval snippets and user input later; avoid putting timestamps, random IDs, or session summaries into the prefix. Changes in tool definition order can break the entire prefix, rendering TTL settings meaningless. Low hit rates in multi-turn conversations are often due to unstable prefixes, not insufficient retention duration.

Long-Context Agent Models: How to Coordinate with Cache Strategies

Recent years have seen small-activation MoE models, such as NVIDIA Nemotron 3.5 Lightning (30B hybrid MoE, 3B activated, 1M context, available on DeepInfra since August 2026), designed for high-throughput multi-turn agents. In high-frequency loops, these models have a high prefill ratio; skipping prefill can significantly reduce TTFT, making them more suitable for short-tier high-frequency refresh rather than long-tier accumulation. Whether a specific model supports caching depends on the vendor's documentation. For related practices, see AI API Cost Optimization.

When Extending Retention Is Pure Waste

Low-frequency single-turn calls, prefix tokens below the minimum cacheable length, changing system prompts on each request, A/B testing with frequent prompt version switches, and cross-session with no shared prefix—these cases only increase write and storage overhead with longer retention. Don't treat long TTL as a panacea. For related practices, see AI API Latency.

Don't Treat TTL as a Persistence Guarantee: What Determines Cache Expiry in LLM APIs

Setting 1 hour doesn't guarantee a hit. Distributed scheduling, LRU eviction under node memory pressure, and cross-node routing can all cause cold starts. TTL is the maximum retention limit, not a guarantee. Official benchmarks for cross-node hit rates are not public; use your own traffic for A/B comparisons. If you need a stable environment, consider Self-hosted AI API.

Regression Checklist When Changing Models or Providers

Cache behavior changes when switching providers, so use the same metrics for comparison. You can use NexAIX's OpenAI-compatible base_url (https://api.nexaix.net/v1）跑同一套压测脚本，在不同模型上对比三项指标。具体各模型是否支持缓存及计费口径，以NexAIX模型页、定价页与文档为准。回归清单：

FAQ

How long does prompt cache last?

Mainstream LLM APIs offer two tiers: 5 minutes and 1 hour. Anthropic defaults to 5 minutes, extendable to 1 hour; Google Gemini's explicit cache defaults to 1 hour; some inference providers let you specify 5 minutes or 1 hour in request parameters. TTL is the maximum retention, subject to node scheduling.

When does LLM API cache expire?

A cache entry expires if not hit within the TTL. Anthropic resets the sliding window on hit, while Google Gemini's explicit cache uses a fixed TTL and requires explicit updates. For specifics, refer to each vendor's documentation.

How to set KV cache retention duration?

Specify the retention tier explicitly via the provider's cache control field in the request body (for Anthropic, this is the cache_control breakpoint; other providers' field names may vary—see their API docs). First, measure the P90 of intervals between consecutive requests in the same session; if it's less than 5 minutes, choose the short tier; otherwise, evaluate the long tier.

Why is the cache hit rate low in multi-turn conversations?

Often due to unstable prefixes, such as placing timestamps or random IDs in the prefix, or frequently changing system prompts. Optimize prompt structure first, keeping the fixed prefix at the top and variables at the end.

Will agent loop calls hit the cache?

Yes, provided the interval between two calls is less than the TTL and the prefix is consistent. Agent loops typically have second-level intervals, so 5 minutes is sufficient. If intervals often exceed 5 minutes, consider the 1-hour tier.

What is the price difference between cache read and cache write?

Using Anthropic as an example, the initial write is about 1.25 times the base input price, and cache read is about 10% (90% discount), a price difference of about 9 times. Actual benefit depends on hit probability and the amount of cached tokens.

How to use caching to optimize slow first token for long contexts?

Ensure a stable prefix and choose an appropriate TTL tier, so that frequently reused long prefixes hit the cache, skipping prefill and significantly reducing first-token latency. If intervals are too long, consider shortening the context or splitting the cache.