• Stefy Lanza (nextime / spora )'s avatar
    lora-train: cross-engine GPU lock so nothing reloads mid-training (fix intermittent OOM) · 18c6a358
    Stefy Lanza (nextime / spora ) authored
    evict_cosited_siblings() frees the co-located sibling's VRAM ONCE at training
    start, but training runs for minutes as an in-engine background job that the front
    swap-gate can't cover (its POST returns a job_id immediately). So a concurrent LLM
    request reloaded the gguf text model mid-training and OOM'd the trainer (one
    fighter LoRA failed while others succeeded).
    
    Add codai/models/gpu_lock.py: a cross-engine GPU reservation. Training reserves the
    card for its whole duration — locally AND on co-located siblings via new
    /internal/gpu-reserve + /internal/gpu-release endpoints — and every ordinary
    model-load path (manager.request_model, video _load_video_pipeline, image
    _load_diffusers_pipeline) calls wait_until_free() first. A request that needs a
    load during training now BLOCKS until training releases, then loads and serves —
    the same queue-behind-the-owner behaviour the swap-gate gives request-level work,
    now extended to cover training. The training thread is exempt from its own
    reservation, so loading the base model never self-deadlocks; waits are bounded
    (900s) so a stuck reservation can't hang forever.
    
    Validated: sibling reservation blocks a loader thread until release; the reserving
    thread never waits on itself.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    18c6a358
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...