a busy model is waited for and then evicted, with no exception

Measured on zeiss 2026-11-01: eviction freed 12.2 GB of the 18.0 GB needed and
stopped 1.5 GB short -- "held by a busy model or another process" -- and
Z-Image-Turbo loaded with CPU/disk offload. The room it needed was not
unavailable. It was in use, by a request that was about to finish.

Waiting for a model to go idle and then taking its VRAM was already the house
rule for the active model and the keep-resident set. Ordinary LRU models were
the exception: pass 1 printed "is busy serving a request — not evicting it" and
moved on. Three deadlines elsewhere were the same exception in a different
dress, because a deadline here does not avert a hang -- it converts "this load
waits a few seconds" into "this load runs on the CPU for its whole life", with
nothing in the log to say why.

So every pass now waits:

    idle LRU      evict (first: it costs nothing)
    busy LRU      wait, then evict      was "not evicting it"
    active model  wait, then evict      was 120s, else leave loaded
    keep-resident wait, then evict      was 120s, else leave loaded
    _evict_one    wait, then evict      was 120s, else "Skipping eviction"

and the give-up branches are deleted rather than left as dead code describing
an exception that no longer exists. _wait_until_idle defaults to no deadline.

The busy ones are a SECOND sweep rather than inline waits, so a cheap idle
eviction further down the LRU order never queues behind a wait that may turn
out to be unnecessary.

Two things keep an unbounded wait from being a hang, and both already held: the
pass never waits on `protect` -- the key the room is being made FOR, the one
model that provably cannot go idle because it is blocked on the memory being
freed -- and a model unloaded under us ends the wait. The wait also repeats
itself every 15 s, so a genuinely hung generation is legible AS a hung
generation instead of as an eviction that stopped responding.

Two waits stay bounded, deliberately, and are now asserted as such so neither
drifts back by accident:

* unload_model, an operator action with a caller waiting on an answer, where
  "not unloaded" is a perfectly good answer. Eviction has no such option.
* the host-RAM evictor, untouched. Waiting there risks the kernel's OOM killer
  taking the whole engine -- four times in five hours the night that code was
  written. Different resource, different failure mode, separate decision.

test_the_source_never_waits_on_the_protected_model read src[i:i+6000] and
failed because this function grew past it, while the guarantee it protects was
intact; it is now bounded by the next method, like the other source tests here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

tests/test_vram_estimate_and_protect.py

@@ -132,12 +132,21 @@ def test_the_source_never_waits_on_the_protected_model():
         os.path.abspath(__file__))), "codai/models/manager.py"),
         encoding="utf-8").read()
     i = src.index("def _evict_models_for_vram(")
-    body = src[i:i + 6000]
+    # Bounded by the next method, NOT by a character count. It was src[i:i+6000],
+    # and the moment the busy-model sweep was added inside this function the
+    # window stopped reaching the assertion below -- the test failed while the
+    # guarantee it protects was perfectly intact.
+    body = src[i:src.index("\n    def ", i + 10)]
     assert "protect: str = \"\"" in body
     # Pass 1 skips it outright…
     assert "if protect and key == protect:" in body
     # …and the active-model fallback does not fire for it.
     assert "self.active_in_vram != protect" in body
+    # …and the busy sweep waits only on what pass 1 collected, which is why it
+    # cannot wait on `protect` either: the skip above happens first.
+    assert "_busy_candidates = []" in body
+    assert "for key in _busy_candidates:" in body
+    assert body.index("if protect and key == protect:") > body.index("_busy_candidates = []")
 
 
 def test_the_embedding_retry_protects_the_model_it_is_making_room_for():

tests/test_wait_then_evict_has_no_exception.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""A busy model is waited for and then evicted. There is no exception.
+
+Measured on zeiss 2026-11-01: eviction freed 12.2 GB of the 18.0 GB needed and
+stopped 1.5 GB short, "held by a busy model or another process", and
+Z-Image-Turbo loaded with CPU/disk offload. The room it needed was not
+unavailable -- it was in use, by a request that was about to finish.
+
+The rule was already the house rule for the active model and the keep-resident
+set; ordinary LRU models were the exception, and three deadlines elsewhere were
+exceptions in a different dress. A deadline here does not avert a hang: it
+converts "this load waits a few seconds" into "this load runs on the CPU for
+its whole life", silently.
+"""
+
+import inspect
+import os
+
+from codai.models import manager as M
+
+
+_SRC = open(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                         "codai/models/manager.py"), encoding="utf-8").read()
+
+
+def _func(name):
+    """The whole method, bounded by the next one -- never by a character count.
+    A fixed window stops covering the code it was written for the moment
+    anything is added inside it."""
+    i = _SRC.index(f"def {name}(")
+    return _SRC[i:_SRC.index("\n    def ", i + 10)]
+
+
+def _code(name):
+    """The method WITHOUT its docstring.
+
+    These tests assert that certain sentences no longer appear, and the
+    docstrings quote those very sentences to record what changed -- so a check
+    against the whole method matches its own documentation and fails for the
+    wrong reason.
+    """
+    body = _func(name)
+    first = body.find('"""')
+    if first == -1:
+        return body
+    end = body.find('"""', first + 3)
+    return body[:first] + body[end + 3:] if end != -1 else body[:first]
+
+
+# --------------------------------------------------------------------------- #
+#  the wait itself
+# --------------------------------------------------------------------------- #
+
+def test_waiting_for_idle_has_no_deadline_by_default():
+    sig = inspect.signature(M.MultiModelManager._wait_until_idle)
+    assert sig.parameters["timeout"].default is None, \
+        "a default deadline is the exception, reintroduced"
+
+
+def test_the_wait_ends_if_the_model_is_unloaded_under_us():
+    """One of the two things that keep an unbounded wait from being a hang."""
+    body = _code("_wait_until_idle")
+    assert "key not in self.models" in body
+    assert "return True" in body
+
+
+def test_a_long_wait_is_never_silent():
+    """The other: a hung generation must read AS a hung generation, not as an
+    eviction that stopped responding."""
+    body = _code("_wait_until_idle")
+    assert "_WAIT_IDLE_SAY_EVERY_S" in body
+    assert "still waiting for" in body
+    assert isinstance(M._WAIT_IDLE_SAY_EVERY_S, (int, float))
+    assert 0 < M._WAIT_IDLE_SAY_EVERY_S <= 60
+
+
+# --------------------------------------------------------------------------- #
+#  no pass declines to evict a busy model
+# --------------------------------------------------------------------------- #
+
+def test_a_busy_lru_model_is_waited_for_and_not_passed_over():
+    body = _code("_evict_models_for_vram")
+    assert "not evicting it" not in body, \
+        "this was the first pass abandoning a busy model"
+    assert "_busy_candidates" in body
+    assert "for key in _busy_candidates:" in body
+
+
+def test_no_pass_reports_leaving_a_busy_model_loaded():
+    """Every one of these strings was an exception to the rule."""
+    body = _code("_evict_models_for_vram")
+    for phrase in ("leaving it loaded",
+                   "still busy after wait",
+                   "wait budget spent",
+                   "wait budget exhausted"):
+        assert phrase not in body, f"still declines to evict: {phrase!r}"
+
+
+def test_the_busy_sweep_runs_after_the_free_ones():
+    """Idle models cost nothing, so they go first: a cheap eviction must never
+    queue behind a wait that may turn out to be unnecessary."""
+    body = _code("_evict_models_for_vram")
+    collect = body.index("_busy_candidates.append(key)")
+    sweep = body.index("for key in _busy_candidates:")
+    assert collect < sweep
+
+
+def test_evict_one_waits_rather_than_skipping():
+    body = _code("_evict_one")
+    assert "Skipping eviction" not in body
+    assert "_wait_until_idle(key)" in body
+
+
+# --------------------------------------------------------------------------- #
+#  the two bounded waits that remain are bounded on purpose
+# --------------------------------------------------------------------------- #
+
+def test_unload_model_stays_bounded_and_says_why():
+    """An operator action with a caller waiting on an answer, where "not
+    unloaded" IS an answer. Eviction has no such option."""
+    body = _func("unload_model")
+    assert "_wait_until_idle(key, timeout=120.0)" in body
+    assert "on purpose" in body.lower() or "ON PURPOSE" in body
+
+
+def test_the_host_ram_evictor_is_left_alone_deliberately():
+    """Waiting there risks the kernel's OOM killer taking the whole engine --
+    four times in five hours the night that code was written. Different
+    resource, different failure mode; changing it is a separate decision."""
+    assert "not evicting it (RAM)" in _SRC