-
Stefy Lanza (nextime / spora ) authored
Two fixes for the performance split OOM loop at large context: 1) The performance "fits on the fast card?" check used expected_vram_gb (WEIGHTS only), ignoring the KV cache + compute buffers. A ~20 GB model at 178k ctx looked like it fit a 24 GB card → tensor_split=[1.0,0.0] → llama_context creation OOM'd (KV didn't fit) in a retry loop. Add a context-scaled KV/compute estimate (~n_ctx/16000 + 1.5 GB, calibrated to observed ~10 GB at 178k) to the footprint before deciding what fits — tight enough not to over-load the slow card. 2) On llama_context creation failure, progressively offload to CPU: move ~20% more layers to CPU and retry, up to 5 times, then fail. A slow CPU-spilled load beats a hard failure / crash loop. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
f1f11723