Prompt / KV Caching

Prompt caching can reduce latency and cost when you send multiple requests that reuse the same prompt context. This page explains how reuse works, how to get more of it, and how to keep your cached context separated from other customers with a cache salt.

What It Is

Large language model requests often repeat the same content: system instructions, tool definitions, examples, documents, repository context, or a long conversation history. With prompt caching, the API can reuse work from a recent request instead of processing the repeated prompt prefix from scratch.

This is sometimes called KV caching because the reused data is the model's derived key/value state for that prompt prefix. You do not need to manage this state yourself.

Availability

KV caching is available on some models. In the Model Catalog, models that show a price for cached tokens generally support prompt/KV caching.

Cache use is not guaranteed for any individual request. A request may miss the cache because the prompt has not been seen recently, the reused portion does not match closely enough, or the cache was evicted to make room for other workloads.

What It Means for You

When a cache hit happens, repeated input tokens may be billed at the cached-token price and the request may start faster. The model output is still generated normally for each request.

Caching is most useful for workloads with large, repeated context, such as:

• Coding assistants that repeatedly send repository, file, or conversation context
• Agents with stable system prompts and tool definitions
• Document analysis where the same long document is queried multiple times
• Multi-turn conversations where earlier context is reused across requests

Caching should be treated as an optimization, not as storage or application state. Your application should behave correctly whether a request hits the cache or not.

Improving Cache Reuse

You cannot force a cache hit, but you can make reuse more likely:

• Put stable, repeated content first in the request.
• Put user-specific or per-request content near the end.
• Keep system prompts, tool definitions, examples, and long context blocks identical across related requests.
• Send related requests within a short timeframe when possible.
• Avoid changing timestamps, random IDs, or other volatile text inside the reusable prompt prefix.

In general, prompt/KV cache reuse is more likely when multiple requests with the same long prefix arrive close together. There are no guarantees for how long a prompt prefix remains cached.

Cache Isolation Between Customers

By default, whether a request reuses cached blocks depends on the prompt content alone. If two customers send the same prompt prefix, both draw on the same cached blocks. Sharing never exposes prompt content: the cache holds derived model state, not text, and outputs are generated per request. What sharing does leave is a timing signal. A fast response to an identical prompt tells the sender that someone sent that prompt recently. If you want strict separation, send a cache salt.

Add cache_salt to a request and the salt becomes part of the cache key. Reuse then requires the same prompt prefix and the same salt. Requests with different salts never share blocks, and salted requests never share with unsalted ones.

curl https://api.inference.nebul.io/v1/chat/completions \
  -H "Authorization: Bearer $NEBUL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3",
    "messages": [
      {"role": "system", "content": "You are the assistant for Northwind Traders."},
      {"role": "user", "content": "Summarize the quarterly report below. ..."}
    ],
    "max_tokens": 50,
    "cache_salt": "northwind-prod"
  }'

Send the same payload twice and both requests return 200. The first response shows usage.prompt_tokens_details.cached_tokens: 0. The second shows a count close to the prompt length, as long as the first request is still cached. Change the salt and the count starts from 0 again.

That count is also how you verify isolation yourself. Repeat a request with the same salt and watch cached_tokens rise, then send the identical prompt with a new salt and watch it return to 0. A nonzero count across different salts would mean reuse crossed namespaces; that would be a bug we want reported.

The same field works in Python:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.inference.nebul.io/v1",
    api_key=os.environ["NEBUL_API_KEY"],
)

resp = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[
        {"role": "system", "content": "You are the assistant for Northwind Traders."},
        {"role": "user", "content": "Summarize the quarterly report below. ..."},
    ],
    max_tokens=50,
    extra_body={"cache_salt": "northwind-prod"},
)
print(resp.usage.prompt_tokens_details.cached_tokens)

What the salt prevents

Cache reuse across customers. The engine splits each request's KV cache into fixed-size blocks. Only complete blocks are cached. Each block is keyed by a SHA-256 hash over the tokens in the block, the hash of the previous block, and the request's extra identifiers: your salt, any LoRA adapter in use, and the hash of any multimodal inputs. A block can match only when the entire prefix before it matches token for token with the same identifiers. Your salt is folded into the first block's hash, and every later block chains from it. Change the salt and every hash in the request changes. There is nothing to match, so nothing is reused.

Timing inference. Without a salt, someone who sends your exact prompt gets a fast response while your copy is still cached. That response time tells them the prompt was sent recently, by someone. With your salt on your requests, their probe carries a different salt or none. It always computes from scratch, and its response time says nothing about your traffic.

Treat the salt as a shared secret for your trust group. Anyone who sends the same salt joins that cache namespace, so don't publish it. For the engine-level design, see the "Cache Isolation for Security" section of vLLM's prefix caching documentation.

What is and isn't guaranteed

With a non-empty cache_salt on your requests, you get these properties on every chat model:

• Reuse requires an exact hash match, and a different salt makes every hash in the request differ. Requests with another salt, or none, cannot resolve to your cached blocks.
• The block keys, not the storage tier, decide what matches. A key that doesn't match in GPU memory doesn't match in CPU offload or disk either, so isolation holds in every tier your prefix touches.
• Your own repeats still reuse your cache and still bill at the cached-token price.
• A probe with a different salt always runs the full prefill, so its response time carries no signal about your traffic.

These are outside what a salt gives you:

• Namespace secrecy. The salt is a namespace key, not authentication. Someone who learns your salt value joins that namespace.
• Capacity. All namespaces share the cache's memory and evict least-recently-used. Other customers' traffic can shorten how long your prefixes stay resident. That is pressure on hit rates, never access to your data.
• Caching itself. Salted prefixes evict and miss like any other cache. The availability rules earlier on this page still apply.

Using a salt well

• Pick one stable salt per user, session, or API key and send it on every request. Your own repeats still hit the cache and still bill at the cached-token price.
• Don't generate a new random salt per request. That splits your own traffic into separate namespaces and throws away the reuse and the cached-token price with it.
• Every chat model supports prompt caching and accepts cache_salt. Embedding and rerank endpoints accept the field but ignore it.
• /v1/chat/completions, /v1/completions, /v1/responses, and /v1/messages accept it.

Two other request forms have the same effect on chat models: the nvext.cache_salt body field and the x-tenant-id header, which is a cache selector, not authentication. When more than one is present, the header wins. Pick one form and use it consistently across your traffic.

If you want isolation without sending a salt, dedicated endpoints run a model on capacity reserved for you alone.

Data Storage

The cache is tiered, using the same level semantics as LMCache:

Which tiers a model uses depends on its serving setup. Every tier holds derived model state, never your raw prompts, and every tier evicts least-recently-used. A salted prefix is salted in every tier it touches: block keys are computed once, before a tier is chosen, so isolation never depends on where the bytes sit.

This behavior may evolve as we continue improving caching capabilities. We will update this page if storage behavior or cache guarantees change.