KV cache
also: prompt cache · prefix cacheThe attention keys and values already computed for a prompt. Reusing a shared prefix skips recomputing it; llama.cpp reports the reused share as f_keep.
Gemma-4 26B swapped 27,848 times in a real agentic run. One flag reported zero, on an easy test. Under the load we actually run, it took two settings together: experts on the CPU and an 8-bit KV cache. No new runtime needed.
Our local agent took 24 seconds to say 'OK'. The loop was fine. A cache flag was set to zero, in three places, and the one we edited first was not the live one.
On a paid API, prompt size is the enemy. On a local model with a prompt cache, volatility is. A 21,900-token per-turn bill, where the tool schemas were the bigger half, and the one ordering rule that serves both.
Retrieval finds the nearest item. Search composes an answer from parts, the way a chemist plans a molecule backwards from what is on the shelf. The map we took from the chemistry, and what it changes in an agent.
Inside one run, our local 12B model reuses 97% of its prompt. Starting the next run, it reuses 6.5% and spends 70 seconds re-reading instructions it already knew. Read from the server's own log.
Context size, prompt-cache allocation, batch size: the standard advice for running a model locally. We tested it on one 12B model with tasks that had failed in production. None of it moved quality or speed. Bigger batches made it worse.