-
Stefy Lanza (nextime / spora ) authored
evict_cosited_siblings() frees the co-located sibling's VRAM ONCE at training start, but training runs for minutes as an in-engine background job that the front swap-gate can't cover (its POST returns a job_id immediately). So a concurrent LLM request reloaded the gguf text model mid-training and OOM'd the trainer (one fighter LoRA failed while others succeeded). Add codai/models/gpu_lock.py: a cross-engine GPU reservation. Training reserves the card for its whole duration — locally AND on co-located siblings via new /internal/gpu-reserve + /internal/gpu-release endpoints — and every ordinary model-load path (manager.request_model, video _load_video_pipeline, image _load_diffusers_pipeline) calls wait_until_free() first. A request that needs a load during training now BLOCKS until training releases, then loads and serves — the same queue-behind-the-owner behaviour the swap-gate gives request-level work, now extended to cover training. The training thread is exempt from its own reservation, so loading the base model never self-deadlocks; waits are bounded (900s) so a stuck reservation can't hang forever. Validated: sibling reservation blocks a loader thread until release; the reserving thread never waits on itself. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
18c6a358