Cluster 04 · Axis I

Sovereign inference

llama.cpp, caches, a Rust runtime, and what costs what.

04 · SOVEREIGN INFERENCEA 26B model on 24 GB, without swapping

Gemma-4 26B swapped 27,848 times in a real agentic run. One flag reported zero, on an easy test. Under the load we actually run, it took two settings together: experts on the CPU and an 8-bit KV cache. No new runtime needed.

1 Nov 2026
04 · SOVEREIGN INFERENCE24.2 seconds to 1.0: one flag in three places

Our local agent took 24 seconds to say 'OK'. The loop was fine. A cache flag was set to zero, in three places, and the one we edited first was not the live one.

1 Nov 2026
04 · SOVEREIGN INFERENCEOrder the prompt by volatility

On a paid API, prompt size is the enemy. On a local model with a prompt cache, volatility is. A 21,900-token per-turn bill, where the tool schemas were the bigger half, and the one ordering rule that serves both.

1 Nov 2026
04 · SOVEREIGN INFERENCERSS lied by four gigabytes

A plan blamed our local model for 13 GB of wired memory. Measured: its real footprint was 4.3 GB, its RSS said 0.19. Generation ran at a flat 11.5 tokens per second, and swap did not move during decode. A hypothesis of ours fell on the way.

1 Nov 2026
04 · SOVEREIGN INFERENCESeventy seconds before the first token

Inside one run, our local 12B model reuses 97% of its prompt. Starting the next run, it reuses 6.5% and spends 70 seconds re-reading instructions it already knew. Read from the server's own log.

1 Nov 2026
04 · SOVEREIGN INFERENCE57 runs: the settings that did nothing

Context size, prompt-cache allocation, batch size: the standard advice for running a model locally. We tested it on one 12B model with tasks that had failed in production. None of it moved quality or speed. Bigger batches made it worse.

1 Nov 2026
04 · SOVEREIGN INFERENCEWhat we build

Sovereign AI systems: they run on hardware you own, and the data never leaves it. What that buys, what it costs, and the numbers.

1 Nov 2026