• Stefy Lanza (nextime / spora )'s avatar
    lora-train: cross-engine GPU lock so nothing reloads mid-training (fix intermittent OOM) · 18c6a358
    Stefy Lanza (nextime / spora ) authored
    evict_cosited_siblings() frees the co-located sibling's VRAM ONCE at training
    start, but training runs for minutes as an in-engine background job that the front
    swap-gate can't cover (its POST returns a job_id immediately). So a concurrent LLM
    request reloaded the gguf text model mid-training and OOM'd the trainer (one
    fighter LoRA failed while others succeeded).
    
    Add codai/models/gpu_lock.py: a cross-engine GPU reservation. Training reserves the
    card for its whole duration — locally AND on co-located siblings via new
    /internal/gpu-reserve + /internal/gpu-release endpoints — and every ordinary
    model-load path (manager.request_model, video _load_video_pipeline, image
    _load_diffusers_pipeline) calls wait_until_free() first. A request that needs a
    load during training now BLOCKS until training releases, then loads and serves —
    the same queue-behind-the-owner behaviour the swap-gate gives request-level work,
    now extended to cover training. The training thread is exempt from its own
    reservation, so loading the base model never self-deadlocks; waits are bounded
    (900s) so a stuck reservation can't hang forever.
    
    Validated: sibling reservation blocks a loader thread until release; the reserving
    thread never waits on itself.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    18c6a358
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
gpu_lock.py Loading commit data...
gpu_query.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
peft_compat.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...