Skip to content
Commit 1f42d08c authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

let a cache hit keep the card it was just sized for



A production run put 93% of a video model on the CPU beside an idle GPU, nine
times. The ladder was not at fault -- it evicted an active model, then a
keep-resident one, reached 24.9 GB free and concluded:

    Full-GPU load looks viable (22.1 GB peak need, 24.9 GB free) — using full GPU

and two minutes later the same load reported:

    [pipeline-cache] injected cached components → forcing hook-based 'model' CPU offload
    Video load strategy: model CPU offload
    [after load] VRAM 0.4/25.3 GB (24.9 GB free)
    footprint 22.11 GB total, of which 20.47 GB was offloaded to host RAM

The sizing expressed "use the whole card" by leaving `offload` as None, and
None also means "nobody chose". The cached-components path could not tell the
two apart, so it fell through to its own default -- 'model' CPU offload, which
is there for a dual-expert Wan2.2 whose two ~10 GB experts genuinely do not fit
in 24 GB, and which was then applied to every cached pipeline whatever its
size.

It did not heal either: a pipeline loaded under accelerate offload is never
re-cached ("cache must be built from a pre-offload pipeline"), so once a model
had been cached, every later load was forced onto the CPU and no clean copy
could replace it.

The verdict now travels with the request. A cache hit that was sized to fit
loads resident; the dual-expert default still applies when it does not fit, and
video_resident_experts still forces it on.

_full_gpu_viable is assigned unconditionally. Written where the sizing happens
it would have lived inside `if _offload is None:` → `try:` → `if
cuda.is_available():`, so a per-model strategy, a CPU-only host or any
exception there would raise NameError and break EVERY video request -- a much
worse failure than the one being fixed. The test asserts the assignment
DOMINATES the read, not merely that it precedes it, and fails if it is nested
back inside a branch.

Co-Authored-By: default avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
parent df66aa56
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment