Four SIGKILLs tonight (19:01, 19:32, 20:42, 21:00), each at ~45 GB RSS on a 54 GB
box, every one of them the nvidia-gguf engine mid-load — and with it every model
resident on that engine, which is why unrelated requests 502'd.

The chain: gemma-4-31B at n_ctx 128000 doesn't fit 24 GB of VRAM, llama.cpp fails
to create the context, and the retry silently allocates the KV cache in HOST RAM
(`CPU KV buffer size = 46750.00 MiB`). Nothing checked whether host RAM could take
it. The existing auto-offload sizes layers by weight only; its KV headroom
heuristic (2 GB + n_ctx/12000) predicted ~12.7 GB against a real 45.7.

So size the KV first and pick a placement that fits: VRAM when there's room, host
RAM when there isn't, a quantized KV when neither can hold it at f16, and a clean
refusal when even that won't fit — a failed load beats an OOM-killed engine. The
context is never shrunk; that stays the caller's decision.

It acts only on a MEASURED size. Predicting KV from GGUF metadata is
architecture-specific — Gemma-4 carries a per-layer head_count_kv array, separate
SWA key/value lengths and a sliding-window pattern — and the naive formula came out
5x high while the SWA-aware one came out 4x low on this very model. So the load now
parses llama.cpp's own "KV buffer size" lines, reports the real total split by
where it landed, and prints the measured_kv_gb line to record; estimates are logged
for visibility but never block a load.

Known gap: the first load of a model at a given n_ctx has no measurement yet, so it
can still be the one that OOMs. Tonight's value is known (45.7 GB at 128000) and can
be recorded by hand to cover it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY