• Stefy Lanza (nextime / spora )'s avatar
    engines: cross-engine VRAM eviction for co-located GGUF/torch engines · 88063940
    Stefy Lanza (nextime / spora ) authored
    After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA
    card but each can only evict its OWN models. So loading an image model on the torch
    engine while the gguf engine still holds a resident GGUF text model failed: local
    eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM.
    
    Fix — co-located VRAM release:
    - front passes each engine its same-GPU siblings' internal URLs via
      CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors).
    - engine registers an external_vram_releaser that POSTs to each sibling's new
      /internal/evict-vram when local eviction can't free enough.
    - /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy)
      models and returns GB freed; busy/actively-serving models are left alone.
    
    Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle
    diffusers models. Driver-level free VRAM (cross-process) is re-checked after each
    releaser, so the loader proceeds once the sibling has freed enough.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    88063940
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...