• Stefy Lanza (nextime / spora )'s avatar
    gpu: performance-split KV-aware fit + progressive CPU-offload retry on OOM · f1f11723
    Stefy Lanza (nextime / spora ) authored
    Two fixes for the performance split OOM loop at large context:
    
    1) The performance "fits on the fast card?" check used expected_vram_gb (WEIGHTS
       only), ignoring the KV cache + compute buffers. A ~20 GB model at 178k ctx looked
       like it fit a 24 GB card → tensor_split=[1.0,0.0] → llama_context creation OOM'd
       (KV didn't fit) in a retry loop. Add a context-scaled KV/compute estimate
       (~n_ctx/16000 + 1.5 GB, calibrated to observed ~10 GB at 178k) to the footprint
       before deciding what fits — tight enough not to over-load the slow card.
    
    2) On llama_context creation failure, progressively offload to CPU: move ~20% more
       layers to CPU and retry, up to 5 times, then fail. A slow CPU-spilled load beats
       a hard failure / crash loop.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    f1f11723
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...