• Stefy Lanza (nextime / spora )'s avatar
    gpu: performance-split KV-aware fit + progressive CPU-offload retry on OOM · f1f11723
    Stefy Lanza (nextime / spora ) authored
    Two fixes for the performance split OOM loop at large context:
    
    1) The performance "fits on the fast card?" check used expected_vram_gb (WEIGHTS
       only), ignoring the KV cache + compute buffers. A ~20 GB model at 178k ctx looked
       like it fit a 24 GB card → tensor_split=[1.0,0.0] → llama_context creation OOM'd
       (KV didn't fit) in a retry loop. Add a context-scaled KV/compute estimate
       (~n_ctx/16000 + 1.5 GB, calibrated to observed ~10 GB at 178k) to the footprint
       before deciding what fits — tight enough not to over-load the slow card.
    
    2) On llama_context creation failure, progressively offload to CPU: move ~20% more
       layers to CPU and retry, up to 5 times, then fail. A slow CPU-spilled load beats
       a hard failure / crash loop.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    f1f11723
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...