vram: the F5-TTS releaser was never actually called

Found in a production log, during a FULL VRAM CLEANUP ahead of a LoRA training
job:

  Warning in external VRAM releaser: release_f5_engine() takes 0 positional
  arguments but 1 was given

The manager calls every external releaser as `fn(needed_gb)` and sums what they
return. release_f5_engine() declared no parameter, so every call raised TypeError
straight into the manager's warning path: the cached F5-TTS engine held its VRAM
through every eviction, including the exclusive ones that exist precisely because
training needs the whole card. It failed quietly — the warning is one line in a
cleanup that otherwise reports success.

It now takes needed_gb and returns the VRAM delta it observed, like the other
four releasers.

The test checks the shape statically, across every
register_external_vram_releaser call in the tree, so the next releaser cannot be
registered with a signature the manager cannot call. Written against the real
contract: one positional argument, and a value returned. (It first asserted a
default was required too, which flagged _cosite_vram_releaser — but the manager
always passes the argument, so that was the test inventing a rule.)

11 tests. Added with -f: .gitignore ignores test_*.py by default.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
parent 2b094b26
......@@ -242,14 +242,30 @@ def _get_f5_engine():
return _f5_engine
def release_f5_engine() -> None:
"""Drop the cached engine (VRAM eviction hook)."""
def _free_vram_gb() -> float:
try:
import torch
if torch.cuda.is_available():
return torch.cuda.mem_get_info()[0] / (1024 ** 3)
except Exception:
pass
return 0.0
def release_f5_engine(needed_gb: float = 0.0) -> float:
"""Drop the cached engine (VRAM eviction hook).
The manager calls every external releaser as `fn(needed_gb)` and adds up what
they report, so this takes the argument and returns the VRAM delta it saw.
It used to take none, which made every call raise TypeError into the
manager's warning path — the engine was never actually released."""
global _f5_engine, _f5_engine_device
with _f5_lock:
if _f5_engine is None:
return
return 0.0
_f5_engine = None
_f5_engine_device = None
before = _free_vram_gb()
try:
import gc
import torch
......@@ -258,7 +274,9 @@ def release_f5_engine() -> None:
torch.cuda.empty_cache()
except Exception:
pass
print(" [voice-clone] F5-TTS engine released", flush=True)
freed = max(0.0, _free_vram_gb() - before)
print(f" [voice-clone] F5-TTS engine released ({freed:.1f} GB)", flush=True)
return freed
# Caching the engine means it holds VRAM between requests, so let the model
......
"""Every external VRAM releaser must match the manager's calling convention.
The manager calls each one as `fn(needed_gb)` and sums what they return. A
releaser that declares no parameter does not fail loudly: the TypeError is caught
and printed as a warning, and the VRAM it holds is simply never freed. That is
what happened to the F5-TTS engine — found in a production log as
Warning in external VRAM releaser: release_f5_engine() takes 0 positional
arguments but 1 was given
during a FULL VRAM CLEANUP. This checks the shape statically, so a new releaser
cannot be registered with the wrong signature.
"""
import ast
import pathlib
import pytest
ROOT = pathlib.Path(__file__).resolve().parents[1]
REGISTER = "register_external_vram_releaser"
def _registrations():
"""(file, callable-name) for every register_external_vram_releaser(...) call."""
for path in (ROOT / "codai").rglob("*.py"):
try:
tree = ast.parse(path.read_text(encoding="utf-8"))
except SyntaxError: # pragma: no cover
continue
for node in ast.walk(tree):
if not isinstance(node, ast.Call) or not node.args:
continue
fn = node.func
name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", None)
if name != REGISTER:
continue
arg = node.args[0]
target = arg.attr if isinstance(arg, ast.Attribute) else getattr(arg, "id", None)
if target:
yield path, target, tree
def _find_def(tree, name):
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name:
return node
return None
def test_there_are_releasers_to_check():
"""A rename that silently emptied this list would make every case below vacuous."""
assert len(list(_registrations())) >= 4
@pytest.mark.parametrize("path,target,tree", list(_registrations()),
ids=lambda v: v if isinstance(v, str) else "")
def test_a_releaser_accepts_needed_gb(path, target, tree):
fn = _find_def(tree, target)
if fn is None:
pytest.skip(f"{target} is not defined in {path.name}")
params = [a.arg for a in fn.args.args if a.arg != "self"]
assert params, (
f"{path.relative_to(ROOT)}:{fn.lineno} {target}() takes no argument, but the "
f"manager calls every releaser as fn(needed_gb) — it would raise TypeError "
f"into the warning path and free nothing")
# Extra parameters are fine only if the manager's single argument still
# satisfies them — anything beyond the first must have a default.
required = len(params) - len(fn.args.defaults)
assert required <= 1, (
f"{target}() requires {required} arguments; the manager passes exactly one")
@pytest.mark.parametrize("path,target,tree", list(_registrations()),
ids=lambda v: v if isinstance(v, str) else "")
def test_a_releaser_returns_a_number(path, target, tree):
"""The manager adds up the return values, so a releaser that returns nothing
reports 0 GB freed and the eviction loop cannot tell it worked."""
fn = _find_def(tree, target)
if fn is None:
pytest.skip(f"{target} is not defined in {path.name}")
returns = [n for n in ast.walk(fn) if isinstance(n, ast.Return) and n.value is not None]
assert returns, f"{target}() never returns a value"
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment