• Stefy Lanza (nextime / spora )'s avatar
    gpu: performance-split KV-aware fit + progressive CPU-offload retry on OOM · f1f11723
    Stefy Lanza (nextime / spora ) authored
    Two fixes for the performance split OOM loop at large context:
    
    1) The performance "fits on the fast card?" check used expected_vram_gb (WEIGHTS
       only), ignoring the KV cache + compute buffers. A ~20 GB model at 178k ctx looked
       like it fit a 24 GB card → tensor_split=[1.0,0.0] → llama_context creation OOM'd
       (KV didn't fit) in a retry loop. Add a context-scaled KV/compute estimate
       (~n_ctx/16000 + 1.5 GB, calibrated to observed ~10 GB at 178k) to the footprint
       before deciding what fits — tight enough not to over-load the slow card.
    
    2) On llama_context creation failure, progressively offload to CPU: move ~20% more
       layers to CPU and retry, up to 5 times, then fail. A slow CPU-spilled load beats
       a hard failure / crash loop.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    f1f11723
Name
Last commit
Last update
..
__init__.py Loading commit data...
base.py Loading commit data...
cuda.py Loading commit data...
ds4.py Loading commit data...
vulkan.py Loading commit data...