Concept

KV cache

also: prompt cache · prefix cache

The attention keys and values already computed for a prompt. Reusing a shared prefix skips recomputing it; llama.cpp reports the reused share as f_keep.

Related
6 articles
04 · SOVEREIGN INFERENCEA 26B model on 24 GB, without swapping

Gemma-4 26B swapped 27,848 times in a real agentic run. One flag reported zero, on an easy test. Under the load we actually run, it took two settings together: experts on the CPU and an 8-bit KV cache. No new runtime needed.

1 Nov 2026
04 · SOVEREIGN INFERENCE24.2 seconds to 1.0: one flag in three places

Our local agent took 24 seconds to say 'OK'. The loop was fine. A cache flag was set to zero, in three places, and the one we edited first was not the live one.

1 Nov 2026
04 · SOVEREIGN INFERENCEOrder the prompt by volatility

On a paid API, prompt size is the enemy. On a local model with a prompt cache, volatility is. A 21,900-token per-turn bill, where the tool schemas were the bigger half, and the one ordering rule that serves both.

1 Nov 2026
01 · RETRIEVAL THAT WORKSSearch is retrosynthesis

Retrieval finds the nearest item. Search composes an answer from parts, the way a chemist plans a molecule backwards from what is on the shelf. The map we took from the chemistry, and what it changes in an agent.

1 Nov 2026
04 · SOVEREIGN INFERENCESeventy seconds before the first token

Inside one run, our local 12B model reuses 97% of its prompt. Starting the next run, it reuses 6.5% and spends 70 seconds re-reading instructions it already knew. Read from the server's own log.

1 Nov 2026
04 · SOVEREIGN INFERENCE57 runs: the settings that did nothing

Context size, prompt-cache allocation, batch size: the standard advice for running a model locally. We tested it on one 12B model with tasks that had failed in production. None of it moved quality or speed. Bigger batches made it worse.

1 Nov 2026

Glossary →