use what the server already knew: precision, the VAE grid, espeak's names, the engine's own name

Four unrelated rows from an end-to-end pass over every capability, with one
thing in common: in each case the information needed to get it right was
already in the server and something declined to read it.

**An image card with no precision loaded at f32.** The default here was f32 and
only --no-ram mode coerced it down, so whether a model occupied 2 bytes per
parameter or 4 came down to an unrelated flag. Measured on one run:
CyberRealisticPony and wai-illustrious (no_ram=True) loaded at 7.49 GB, while
PonyRealism and pornmaster-pro -- same SDXL architecture, no_ram=False --
measured 14.98 GB, exactly double, filled a 24 GB card and failed with "CUDA out
of memory. Tried to allocate 12.00 MiB". The inflation then spread to the
estimate, where the f32 precision factor turned 14.98 GB into a 30.9 GB need and
triggered a round of eviction that could not possibly help ("this model needs
more than the whole card"). Nothing was gained: these checkpoints are published
at half width. The default is now bf16 -- bf16 rather than fp16 because it keeps
f32's range, the same reason the --no-ram branch gives. An explicit f32 on a
card is still honoured.

**LTX refused the width and height.** Every family in video_geometry declares a
side_multiple -- 64 for Wan's VAE cell, 32 for H3 -- and nothing anywhere read
it; the request path never consulted the table at all. So 848x480 reached
LTX-2.5 and came back "`height` and `width` have to be divisible by 32 but are
848 and 480", with the server making the caller do arithmetic it had the numbers
for. Sides are now snapped to the nearest legal cell, and LTX has a family of
its own: it had none, so it inherited Wan's 16, under which 848 was already
"legal" -- snapping to the wrong multiple is indistinguishable from not snapping.
The new entry differs from the Wan fallback in exactly the one field a refusal
proved.

**Kokoro could not speak English.** espeak-ng names English by region and
refuses the bare code, while every OpenAI-compatible client sends "en" -- so the
Italian voices worked and the English ones raised `RuntimeError: language "en"
is not supported by the espeak backend`, as a 500, which reads as a broken
model. Regionless codes espeak rejects are mapped ("en" -> "en-us"); codes it
accepts bare ("it", "fr", "de") pass through untouched; anything it still does
not know is now a 4xx naming the language and what to send.

**A pinned model landed on the wrong engine.** by_name() matched an engine by
NAME or by BACKEND in one pass and returned the first healthy hit. A
GGUF-isolation split runs two engines on one card and both declare the backend
"nvidia", so a transformers model pinned to "nvidia" could resolve to the GGUF
sibling and be refused for a capability the engine it actually named has:
"model 'longcat-avatar' is pinned to 'nvidia' but engine 'nvidia' (backend
'nvidia') can't serve a 'transformers' model (capabilities: ['gguf'])". The
sibling fallback could not recover it either -- it appends "-gguf" to the PIN,
so it tried the same engine twice and never the one named "nvidia". An exact
name is unambiguous, so it now wins, in its own pass.
Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
parent fdbc7701
......@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here.
__version__ = "0.5.28"
__version__ = "0.5.29"
__tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
# Configure the CUDA caching allocator BEFORE torch is imported anywhere.
......
......@@ -699,7 +699,33 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
# All loading parameters come from the per-model configuration.
no_ram = bool(_cfg('no_ram', False))
precision = _cfg('precision', 'f32') or 'f32'
# A card that does not state a precision gets a HALF one, not f32.
#
# f32 was the default here, and only --no-ram mode coerced it down, so
# whether a model loaded at 2 bytes or 4 per parameter came down to an
# unrelated flag. Measured on one production run: CyberRealisticPony and
# wai-illustrious (no_ram=True) loaded at 7.49 GB, while PonyRealism and
# pornmaster-pro -- the same SDXL architecture, no_ram=False -- measured
# 14.98 GB, exactly double, filled the card and failed with "CUDA out of
# memory. Tried to allocate 12.00 MiB". The inflation then spread to the
# estimate: the f32 precision factor turned that 14.98 GB into a 30.9 GB
# need, which triggered a round of eviction that could not possibly help
# ("this model needs more than the whole card").
#
# Nothing was gained for it. These checkpoints are published in fp16/bf16,
# so f32 doubles the weights by upcasting what was never computed at that
# width. bf16 rather than fp16 because it keeps f32's range: a flow-matching
# DiT run in fp16 goes past 65504, the latents go NaN, and the VAE decodes
# NaN to uniform black -- answered 200 OK with a plausible PNG (the same
# reasoning as the --no-ram branch below).
#
# An explicit 'f32' on the card is still honoured: this changes the default,
# not the operator's choice.
_precision_cfg = _cfg('precision')
precision = str(_precision_cfg or '').strip() or 'bf16'
if not _precision_cfg:
print(f"Using precision: {precision} (card states none; f32 would "
f"double the weights of a checkpoint published at half width)")
precision_map = {
'bf16': torch.bfloat16,
'f32': torch.float32,
......
......@@ -138,6 +138,28 @@ def _resolve_kokoro_files(model_path: str, config: dict) -> Tuple[str, str]:
return onnx_path, voices_path
#: espeak-ng names English by REGION -- ``en-us``, ``en-gb`` -- and rejects the
#: bare code outright: `language "en" is not supported by the espeak backend`.
#: Every OpenAI-compatible client in existence sends "en", so Kokoro's English
#: voices were unreachable while its Italian ones worked, which reads as the
#: model being broken rather than as a two-letter code needing a region.
#: Only codes espeak actually refuses belong here: "it", "fr" and "de" are
#: accepted bare and must pass through untouched.
_ESPEAK_REGIONLESS = {
"en": "en-us",
"pt": "pt-br",
"zh": "cmn",
}
def espeak_language(lang: str) -> str:
"""``lang`` as espeak-ng names it."""
want = str(lang or "").strip().lower().replace("_", "-")
if not want:
return "en-us"
return _ESPEAK_REGIONLESS.get(want, want)
class _KokoroBackend:
family = "kokoro"
default_voice = "af_sarah"
......@@ -150,10 +172,24 @@ class _KokoroBackend:
def synthesize(self, text: str, voice: str, speed: float, lang: str,
emotion: str = "", style: str = "") -> Tuple[np.ndarray, int]:
samples, sr = self._kokoro.create(
text, voice=voice or self.default_voice, speed=speed or 1.0,
lang=lang or "en-us",
)
_lang = espeak_language(lang)
try:
samples, sr = self._kokoro.create(
text, voice=voice or self.default_voice, speed=speed or 1.0,
lang=_lang,
)
except Exception as exc:
# A language espeak does not know is a request problem, not a
# server one, and the message has to say which language and what
# to send instead -- "RuntimeError: language ... is not supported
# by the espeak backend" reached the client as a 500.
if "espeak" in str(exc).lower():
raise ValueError(
f"this voice cannot speak {lang!r}: espeak-ng has no such "
f"language (tried {_lang!r}). espeak names some languages "
f"by region — send 'en-us' or 'en-gb' rather than 'en'."
) from exc
raise
return np.asarray(samples, dtype=np.float32), int(sr)
def voices(self):
......
......@@ -2196,7 +2196,8 @@ def _load_rife(model_name: str, device: str):
# Generation logic
# =============================================================================
def _build_call_kwargs(request: VideoGenerationRequest) -> dict:
def _build_call_kwargs(request: VideoGenerationRequest, model_name: str = "",
model_cfg: dict = None) -> dict:
# NOTE: this populates steps/guidance ONLY when the REQUEST set them. The
# caller's accel and model-card defaults are layered on afterwards with
# setdefault, and both of those depend on an unset key meaning "the caller
......@@ -2213,8 +2214,23 @@ def _build_call_kwargs(request: VideoGenerationRequest) -> dict:
if request.num_frames:
kw['num_frames'] = request.num_frames
if request.width and request.height:
kw['width'] = request.width
kw['height'] = request.height
# Snapped to the family's VAE grid, not passed through. Every family in
# video_geometry declares a side_multiple -- 64 for Wan, 32 for LTX, 16
# for H3 -- and nothing read it, so an ordinary 848x480 request reached
# LTX-2.5 and was refused with "`height` and `width` have to be
# divisible by 32 but are 848 and 480". 480 was already legal; 848 is
# 26.5 cells. The server knows the multiple, so it does the rounding
# instead of making the caller discover it from an error.
from codai.models import video_geometry
_geom = video_geometry.geometry_for(model_name or "", model_cfg or {})
_w, _h, _moved = video_geometry.snap_frame_size(
request.width, request.height, _geom)
if _moved:
print(f" [video] {request.width}x{request.height} is not on this "
f"model's {_geom.get('side_multiple')}-pixel grid — "
f"generating {_w}x{_h}", flush=True)
kw['width'] = _w
kw['height'] = _h
if request.seed is not None:
import torch
kw['generator'] = torch.Generator().manual_seed(request.seed)
......@@ -3000,11 +3016,12 @@ def _pipeline_takes(pipe, name: str) -> bool:
return name in params
def _generate_video(pipe, request: VideoGenerationRequest):
def _generate_video(pipe, request: VideoGenerationRequest, model_name: str = "",
model_cfg: dict = None):
mode = request.mode or ('i2v' if (request.image or request.init_image)
else 'v2v' if request.video else 't2v')
fps = request.fps or 8
kw = _build_call_kwargs(request)
kw = _build_call_kwargs(request, model_name, model_cfg)
# Acceleration/distillation defaults (Lightning / Lightx2v): when the model has
# a fused distill LoRA, default to its low step-count / guidance instead of the
# standard 25 steps / 7.5 CFG. The request always wins if it set these — note
......@@ -5513,7 +5530,7 @@ async def video_generations(request: VideoGenerationRequest,
_gen_audio = {}
else:
frames, fps, _gen_audio = await _loop.run_in_executor(
None, _generate_video, pipe, request)
None, _generate_video, pipe, request, model_name, _model_cfg)
_last_err = None
break # success → return to client
except Exception as e:
......@@ -5557,7 +5574,8 @@ async def video_generations(request: VideoGenerationRequest,
_gen_audio = {}
else:
frames, fps, _gen_audio = await _loop.run_in_executor(
None, _generate_video, pipe, request)
None, _generate_video, pipe, request, model_name,
_model_cfg)
_last_err = None
break # success → return to client
except Exception as _e3:
......
......@@ -263,21 +263,43 @@ class EngineRegistry:
return next((e for e in self._engines.values() if e.healthy), prim)
def by_name(self, name: Optional[str]) -> Optional[Engine]:
"""Resolve an engine by its declared name (or, failing that, its backend).
"""Resolve an engine by its declared NAME, or failing that by its backend.
Used for the configured default engine and per-model pins. Prefers a healthy
match but returns an unhealthy one too, so callers can decide."""
match but returns an unhealthy one too, so callers can decide.
The two matches are tried in two passes, and that ordering is the whole
point. A GGUF-isolation split runs two engines on one card -- ``nvidia``
(transformers) and ``nvidia-gguf`` -- and BOTH declare the backend
``nvidia``. A single pass accepting either kind of match returned
whichever healthy engine came first in the dict, so a model pinned to
``nvidia`` could land on the GGUF sibling and be refused for a
capability the engine it actually named has:
WARNING: model 'longcat-avatar' is pinned to 'nvidia' but engine
'nvidia' (backend 'nvidia') can't serve a 'transformers' model
(capabilities: ['gguf']); failing the request
-- and the warning says "engine 'nvidia'" for an engine named
``nvidia-gguf``, because it prints the pin rather than what was
resolved. The sibling fallback could not save it either: it appends
``-gguf`` to the PIN, so it tried the same engine twice and never the
one named ``nvidia``. An exact name is unambiguous, so it wins."""
if not name:
return None
name = name.strip().lower()
with self._lock:
engines = list(self._engines.values())
match = None
for e in engines:
if (e.name or "").lower() == name or (e.backend or "").lower() == name:
for key in ("name", "backend"):
for e in engines:
if (getattr(e, key, None) or "").lower() != name:
continue
if e.healthy:
return e
match = match or e
if match is not None:
return match
return match
def remove(self, engine_id: int) -> None:
......
......@@ -88,6 +88,31 @@ FAMILIES = {
"side_multiple": 16,
"continuation": "chained",
},
"ltx": {
"family": "ltx",
"label": "LTX-2.x",
# Everything here except the side multiple is the Wan fallback LTX was
# already being described with, kept deliberately: the only property this
# entry exists to correct is the one a refusal proved.
#
# "`height` and `width` have to be divisible by 32 but are 848 and 480"
# -- LTX's VAE has a 32-pixel cell, where Wan's is 16, and with no family
# of its own LTX inherited 16. So an 848-wide request was already "on the
# grid" as far as the server was concerned and was then refused by the
# pipeline. Snapping to the wrong multiple is indistinguishable from not
# snapping at all.
"native_fps": 16,
"segment_frames": None,
"cond_frames": 0,
"frame_base": 1,
"frame_step": 4,
"min_frames": 5,
"max_frames": None,
"max_frames_per_render": 81,
"default_frames": None,
"side_multiple": 32,
"continuation": "chained",
},
}
# The family used when nothing matches. Wan is the historical shape of every
......@@ -132,6 +157,15 @@ def family_for(model_name: str = "", config: dict = None) -> str:
except Exception:
if "minimax-h3" in str(model_name or "").lower().replace("_", "-"):
return "h3"
# LTX by name, over the id AND the configured path: a card registered under
# an alias ("ltx-2.3-pinkcherry") and one registered as the repo
# ("Lightricks/LTX-2.5-Diffusers") must both find the 32-pixel cell.
_hay = " ".join(str(x or "").lower() for x in (
model_name, cfg.get("path"), cfg.get("model"), cfg.get("alias"),
(cfg.get("_raw_cfg") or {}).get("path") if isinstance(
cfg.get("_raw_cfg"), dict) else ""))
if "ltx" in _hay:
return "ltx"
return DEFAULT_FAMILY
......@@ -181,6 +215,48 @@ def snap_frames(num_frames, geom: dict, mode: str = "nearest") -> int:
return max(base, out)
def snap_side(value, geom: dict, *, minimum: int = 0) -> int:
"""One side of the frame moved onto the family's legal grid.
Every family in the table above declares a ``side_multiple`` -- 64 for Wan's
VAE cell, 32 for LTX, 16 for H3 -- and until this existed nothing read it.
So a perfectly ordinary 848x480 request reached LTX-2.5 and came back
```height` and `width` have to be divisible by 32 but are 848 and 480`:
480 is legal, 848 is 26.5 cells, and the caller is told to do arithmetic the
server already knows how to do. (The frame COUNT had been snapped for ages
by :func:`snap_frames`; the sides were simply forgotten.)
Rounds to the NEAREST legal side rather than down, so an odd width moves by
at most half a cell and the aspect ratio barely shifts, and never returns
less than one cell.
"""
mult = max(1, int(geom.get("side_multiple") or 1))
try:
v = int(value)
except (TypeError, ValueError):
return int(max(minimum, mult))
if v <= 0:
return int(max(minimum, mult))
out = int(round(v / float(mult))) * mult
if out < mult:
out = mult
if minimum:
while out < minimum:
out += mult
return out
def snap_frame_size(width, height, geom: dict) -> tuple:
"""``(width, height)`` both on the family's grid, and whether either moved."""
w = snap_side(width, geom)
h = snap_side(height, geom)
try:
moved = (int(width) != w) or (int(height) != h)
except (TypeError, ValueError):
moved = True
return w, h, moved
def segments_for(total_frames, geom: dict) -> int:
"""How many model calls a total of ``total_frames`` takes on this family.
......
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
"""Fixes from one end-to-end pass over every capability on a live install.
Each test below is a row that came back failing, reduced to the decision that
actually produced it. They are unrelated bugs with one thing in common: in every
case the server knew enough to get it right and declined to use what it knew.
"""
import io
import os
import pytest
_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def _src(rel):
return io.open(os.path.join(_ROOT, rel), encoding="utf-8").read()
# ---------------------------------------------------------------------------
# "PonyRealism and pornmaster don't load" -- they loaded at fp32
# ---------------------------------------------------------------------------
def test_an_image_card_without_a_precision_does_not_get_f32():
"""The two that failed measured 14.98 GB; their SDXL siblings measured 7.49.
Exactly double, because f32 was the default and only --no-ram coerced it
down -- so a model's width in bytes came down to an unrelated flag."""
src = _src("codai/api/images.py")
assert "_cfg('precision', 'f32')" not in src
assert "precision = str(_precision_cfg or '').strip() or 'bf16'" in src
def test_an_explicit_f32_is_still_honoured():
"""Changing a default must not overrule an operator who chose."""
src = _src("codai/api/images.py")
head = src.index("_precision_cfg = _cfg('precision')")
assert "_cfg('precision')" in src[head:head + 200]
# The default is applied only when the card states nothing.
assert "if not _precision_cfg:" in src
# ---------------------------------------------------------------------------
# "LTX x3 reject the width and height"
# ---------------------------------------------------------------------------
def test_every_family_declares_a_side_multiple():
from codai.models.video_geometry import FAMILIES
for name, geom in FAMILIES.items():
assert int(geom.get("side_multiple") or 0) > 0, name
def test_the_ltx_request_that_was_refused_is_now_snapped():
"""`height` and `width` have to be divisible by 32 but are 848 and 480:
480 was already legal, 848 is 26.5 cells of 32."""
from codai.models import video_geometry as G
geom = G.geometry_for("Lightricks/LTX-2.5-Diffusers", {})
assert geom["side_multiple"] == 32
w, h, moved = G.snap_frame_size(848, 480, geom)
assert w % 32 == 0 and h % 32 == 0
assert h == 480 # already legal, so untouched
assert moved # the width moved, and the caller is told
def test_snapping_moves_a_side_by_less_than_one_cell():
"""Rounding to NEAREST, not down: the aspect ratio barely shifts."""
from codai.models import video_geometry as G
geom = {"side_multiple": 64}
for value in (100, 500, 721, 1081, 1920):
out = G.snap_side(value, geom)
assert out % 64 == 0
assert abs(out - value) <= 32
def test_a_legal_size_is_left_exactly_alone():
from codai.models import video_geometry as G
geom = {"side_multiple": 32}
w, h, moved = G.snap_frame_size(832, 480, geom)
assert (w, h, moved) == (832, 480, False)
def test_a_side_never_snaps_to_zero():
"""Rounding 10 pixels down to a multiple of 64 is zero, and a zero-width
frame is a different error further in."""
from codai.models import video_geometry as G
geom = {"side_multiple": 64}
assert G.snap_side(10, geom) == 64
assert G.snap_side(0, geom) == 64
assert G.snap_side(None, geom) == 64
assert G.snap_side("nonsense", geom) == 64
def test_the_request_path_actually_snaps():
"""The multiple was declared for every family and read by nothing — the
table existed and the request path never consulted it."""
src = _src("codai/api/video.py")
assert "snap_frame_size(" in src
assert src.count("None, _generate_video, pipe, request, model_name") == 2
# ---------------------------------------------------------------------------
# "Kokoro English: espeak doesn't support 'en'"
# ---------------------------------------------------------------------------
def test_a_bare_en_becomes_an_espeak_language():
"""espeak-ng names English by region and refuses the bare code, while every
OpenAI-compatible client sends 'en'. Kokoro's Italian worked and its
English did not, which reads as a broken model."""
from codai.api.tts_backends import espeak_language
assert espeak_language("en") == "en-us"
assert espeak_language("EN") == "en-us"
assert espeak_language("en_US") == "en-us"
def test_languages_espeak_accepts_bare_are_untouched():
"""Rewriting 'it' would break the voices that were working."""
from codai.api.tts_backends import espeak_language
for code in ("it", "fr", "de", "es", "en-gb", "en-us"):
assert espeak_language(code) == code
def test_an_empty_language_still_has_a_default():
from codai.api.tts_backends import espeak_language
assert espeak_language("") == "en-us"
assert espeak_language(None) == "en-us"
def test_an_unknown_language_is_a_request_error_not_a_500():
"""It reached the client as RuntimeError: language "en" is not supported by
the espeak backend — a server fault for a request problem."""
src = _src("codai/api/tts_backends.py")
head = src.index("class _KokoroBackend")
body = src[head:src.index("class _TransformersBackend")]
assert "raise ValueError(" in body
assert "espeak" in body
# ---------------------------------------------------------------------------
# "longcat gives a 500 error" -- the pin resolved to the wrong engine
# ---------------------------------------------------------------------------
class _FakeEngine:
def __init__(self, name, backend, healthy=True):
self.name = name
self.backend = backend
self.healthy = healthy
@pytest.fixture
def registry_with_a_split_engine():
"""The GGUF-isolation split: two engines, one card, the same backend."""
from codai.frontproxy import registry as R
reg = R.EngineRegistry() if hasattr(R, "EngineRegistry") else None
if reg is None: # pragma: no cover
pytest.skip("registry class not found")
return reg
def test_an_exact_engine_name_beats_a_backend_match(registry_with_a_split_engine):
"""Both engines on a split declare backend 'nvidia'; only one is NAMED it.
A single pass accepting either returned whichever came first, so a
transformers model pinned to 'nvidia' could land on the GGUF sibling and be
refused for a capability the engine it named has."""
reg = registry_with_a_split_engine
gguf = _FakeEngine("nvidia-gguf", "nvidia")
torch_engine = _FakeEngine("nvidia", "nvidia")
# GGUF first in iteration order: the order that produced the bug.
reg._engines = {1: gguf, 2: torch_engine}
assert reg.by_name("nvidia") is torch_engine
assert reg.by_name("nvidia-gguf") is gguf
def test_a_backend_match_still_works_when_nothing_bears_the_name(
registry_with_a_split_engine):
reg = registry_with_a_split_engine
only = _FakeEngine("engine-0", "radeon")
reg._engines = {1: only}
assert reg.by_name("radeon") is only
def test_an_exact_name_wins_even_unhealthy(registry_with_a_split_engine):
"""The pin names a specific engine, and the caller checks is_alive() for
itself — the pin path has its own sibling fallback for the down case, and it
can only compute the sibling correctly once the name resolves correctly."""
reg = registry_with_a_split_engine
down = _FakeEngine("nvidia", "nvidia", healthy=False)
up = _FakeEngine("nvidia-gguf", "nvidia", healthy=True)
reg._engines = {1: up, 2: down}
assert reg.by_name("nvidia") is down
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment