evict 10% more than asked, and return JSON that parses

**Eviction headroom.** Every size reaching eviction is an estimate, and one
landing a few hundred MB short does not fail honestly: the load is admitted,
starts allocating, and dies on a tiny buffer. Measured on zeiss across
2026-10-30/31, each on a load eviction had just reported as satisfied:

    Tried to allocate  20.00 MiB … 8.69 MiB is free
    Tried to allocate 190.00 MiB … 18.69 MiB is free
    Tried to allocate  12.00 MiB … 8.69 MiB is free

Eviction now targets `needed_gb * 1.10`. The headroom is what it TARGETS, never
what it reports: the shortfall warning and the "needs more than the whole card"
verdict both still measure the caller's actual request, so freeing enough for
the load but not the whole margin stays silent. A warning on the padded figure
would fire on every successful eviction and bury the one real warning. 10% of
the request costs at most one more idle model, and scales — a flat margin would
be too small for a 30 GB model and absurd for a 0.1 GB one.

**JSON mode returned JSON that would not parse.** 21 script generations ended
`finish_reason=length` after writing 6000 tokens of well-formed-but-unfinished
JSON for a job needing ~800; one ran 8.5 minutes. Nothing was misconfigured:
max_tokens was the caller's own 6000, the prompt 2414 tokens, the model's n_ctx
32000. `json_object` IS honoured, but its grammar constrains syntax only and
never requires the root object to close, so an array of scenes grows until the
budget ends mid-token.

Two changes, neither touching the caller's max_tokens:

* `json_schema` is now accepted, which bounds the SHAPE so the object must
  close. llama-cpp-python takes only `type: json_object` and reads a sibling
  `schema` key, while OpenAI nests it under `json_schema` — so the shapes are
  translated. Forwarding OpenAI's form verbatim would have silently DISABLED
  the grammar (`_grammar_for_response_format` returns None for other types),
  which is worse than not supporting it at all.

* When the budget is hit anyway, the open structures are closed so the caller
  gets valid JSON holding the scenes that did arrive. It drops a half-written
  element rather than closing around it, never invents content, parses its own
  output before returning it (an unverifiable repair is not applied), leaves
  prose and already-valid JSON alone, and does not rewrite `finish_reason` —
  which stays `length`, so a caller that checks still learns it was cut short.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.62"
+__version__ = "0.5.63"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/backends/vulkan.py

@@ -2819,8 +2819,35 @@ class VulkanBackend(ModelBackend):
         _stop = self._augment_model_stops(stop)
         if _stop:
             kwargs['stop'] = _stop
-        if response_format and response_format.get('type') == 'json_object':
-            kwargs['response_format'] = {'type': 'json_object'}
+        # JSON mode, including a SCHEMA when the caller supplies one.
+        #
+        # `json_object` alone constrains syntax and nothing else, so the model
+        # may keep a valid object open forever: measured on zeiss 2026-10-31, 21
+        # script generations ended `finish_reason=length` having written 6000
+        # tokens of well-formed-but-unfinished JSON for a job needing ~800, one
+        # of them after 8.5 minutes. Nothing was broken — the array simply never
+        # closed. A schema bounds the shape (these keys, this many scenes), so
+        # the object has to close, which fixes the truncation without touching
+        # the caller's max_tokens.
+        #
+        # llama-cpp-python accepts ONLY `type: json_object` (its
+        # `_grammar_for_response_format` returns None for anything else) and
+        # reads a sibling `schema` key. OpenAI sends the schema nested under
+        # `json_schema`, so the two shapes are translated here rather than
+        # forwarded; forwarding OpenAI's form verbatim silently disables the
+        # grammar instead of tightening it.
+        _rf = response_format or {}
+        if _rf.get('type') in ('json_object', 'json_schema'):
+            _schema = None
+            if _rf.get('type') == 'json_schema':
+                _js = _rf.get('json_schema')
+                if isinstance(_js, dict):
+                    _schema = _js.get('schema')
+            if _schema is None:
+                _schema = _rf.get('schema')      # llama.cpp's own shape
+            kwargs['response_format'] = (
+                {'type': 'json_object', 'schema': _schema} if _schema
+                else {'type': 'json_object'})
         _tc = _make_llama_thermal_criteria()
         if _tc is not None and _chat_supports_stopping_criteria():
             kwargs['stopping_criteria'] = _tc
@@ -2839,8 +2866,117 @@ class VulkanBackend(ModelBackend):
             completion_tokens=usage.get('completion_tokens', 0),
         )
         content = result['choices'][0]['message'].get('content') or ''
+
+        # A caller who asked for JSON and hit the token budget currently gets
+        # invalid JSON — the grammar keeps every token legal but never forces
+        # the root object to close. Close it, so the answer holds the items that
+        # did arrive instead of failing to parse. Only for a JSON request, only
+        # on a `length` finish, and only when the repair itself parses;
+        # `finish_reason` is untouched, so a caller that checks still learns the
+        # answer was cut short.
+        try:
+            if kwargs.get('response_format') and content:
+                _fr = (result['choices'][0] or {}).get('finish_reason')
+                if _fr == 'length':
+                    _fixed = self.close_truncated_json(content)
+                    if _fixed is not None:
+                        print(f"  JSON response hit max_tokens mid-structure — "
+                              f"closed it ({len(content)} -> {len(_fixed)} chars) "
+                              f"so it parses; finish_reason stays 'length'",
+                              flush=True)
+                        content = _fixed
+        except Exception as _jx:
+            print(f"  (JSON close-up skipped: {_jx})", flush=True)
         return content
 
+    @staticmethod
+    def close_truncated_json(text: str):
+        """Close a JSON document cut off mid-structure, or return None.
+
+        A caller who asked for `json_object` and receives syntactically invalid
+        JSON got a protocol violation, not an answer. It happens whenever the
+        model runs to `max_tokens` with the generic JSON grammar: the grammar
+        keeps every token legal but never requires the root object to close, so
+        an array of scenes can grow until the budget ends mid-token. Measured on
+        zeiss 2026-10-31: 21 generations ended that way, and the client reported
+        them as "broken JSON after 1493 s".
+
+        This closes what is open — finishing a dangling string, dropping a
+        half-written value or key, and emitting the missing brackets — so the
+        client gets valid JSON holding the scenes that DID arrive. It never
+        invents content, and `finish_reason` stays `length`, so a caller that
+        checks can still tell the answer was cut short.
+
+        Returns None when the text already parses, when nothing is open, or
+        when the result still will not parse: a repair that cannot be verified
+        is not applied.
+        """
+        import json as _json
+        if not text or not text.strip():
+            return None
+        try:
+            _json.loads(text)
+            return None                      # nothing to fix
+        except Exception:
+            pass
+
+        stack, in_str, esc = [], False, False
+        last_safe = None                     # index just past the last complete item
+        for i, ch in enumerate(text):
+            if in_str:
+                if esc:
+                    esc = False
+                elif ch == '\\':
+                    esc = True
+                elif ch == '"':
+                    in_str = False
+                continue
+            if ch == '"':
+                in_str = True
+            elif ch in '{[':
+                stack.append('}' if ch == '{' else ']')
+            elif ch in '}]':
+                if stack:
+                    stack.pop()
+                last_safe = i + 1
+            elif ch == ',' and stack:
+                last_safe = i            # a comma ends the previous item cleanly
+        if not stack and not in_str:
+            return None                      # not a truncation we understand
+
+        # Cut back to the last point where a value had finished, so a
+        # half-written key/value is dropped rather than closed around.
+        body = text[:last_safe] if last_safe else text
+        body = body.rstrip().rstrip(',')
+        for _ in range(len(stack)):
+            pass
+        # Recompute the open stack for the trimmed body: trimming may have
+        # closed some of it already.
+        stack2, in_str2, esc2 = [], False, False
+        for ch in body:
+            if in_str2:
+                if esc2:
+                    esc2 = False
+                elif ch == '\\':
+                    esc2 = True
+                elif ch == '"':
+                    in_str2 = False
+                continue
+            if ch == '"':
+                in_str2 = True
+            elif ch in '{[':
+                stack2.append('}' if ch == '{' else ']')
+            elif ch in '}]' and stack2:
+                stack2.pop()
+        if in_str2:
+            body += '"'
+        repaired = body + ''.join(reversed(stack2))
+        try:
+            _json.loads(repaired)
+        except Exception:
+            return None                      # unverifiable repair -> leave it alone
+        return repaired
+
     def _render_chat_prompt(self, messages):
         """Render the chat prompt with a CACHED, non-sandboxed Jinja env, identical
         to llama-cpp-python's output but without the sandbox's per-op overhead (the

codai/models/manager.py

@@ -236,6 +236,15 @@ def _gguf_architecture(path: str):
 # --------------------------------------------------------------------------- #
 
 # Ordered by arbitration preference for ambiguous names. New engines append here.
+#: Eviction frees this much MORE than the caller asked for. Every size that
+#: reaches eviction is an estimate; one landing a few hundred MB short leaves the
+#: load exactly on the edge, where it is admitted, allocates, and dies on a tiny
+#: buffer. Measured on 2026-10-30/31: "Tried to allocate 20.00 MiB ... 8.69 MiB
+#: is free", "190.00 MiB ... 18.69 MiB free", "12.00 MiB ... 8.69 MiB free" --
+#: all on loads eviction had just reported as satisfied. The cost of the margin
+#: is at most one more idle model unloaded.
+_EVICT_HEADROOM_FRACTION = 0.10
+
 _ENGINE_BACKENDS = ("ds4", "colibri", "k3", "kt", "vllm", "runpod")
 
 
@@ -6127,6 +6136,21 @@ class MultiModelManager:
         if needed_gb <= 0:
             return
 
+        # Always free a little MORE than asked. Every figure that reaches here is
+        # an estimate, and an estimate landing a few hundred MB short puts the
+        # load exactly on the edge: it is admitted, allocates, and then fails on
+        # a 20 MiB buffer. Measured repeatedly on 2026-10-30/31 -- "Tried to
+        # allocate 20.00 MiB ... 8.69 MiB is free", "190.00 MiB ... 18.69 MiB
+        # free", "12.00 MiB ... 8.69 MiB free" -- loads that eviction had just
+        # "satisfied". 10% of the request is cheap (one more idle model at
+        # worst) and takes the edge case out of reach.
+        #
+        # The headroom is what eviction TARGETS, not what it reports: the
+        # shortfall warning below still measures against what the caller asked
+        # for, so freeing enough for the load but not the headroom is a success
+        # and stays silent.
+        target_gb = needed_gb * (1.0 + _EVICT_HEADROOM_FRACTION)
+
         _evict_key = self._evict_one
 
         def _size_label(key: str) -> str:
@@ -6148,7 +6172,7 @@ class MultiModelManager:
                 continue
             if key in self._keep_resident:
                 continue
-            if self._get_free_vram_gb() >= needed_gb:
+            if self._get_free_vram_gb() >= target_gb:
                 break
             if self._is_key_busy(key):
                 print(f"  '{key}' is busy serving a request — not evicting it")
@@ -6158,7 +6182,7 @@ class MultiModelManager:
 
         # Second pass: evict the active model if still not enough AND it is idle
         # (unless it's keep-resident — that's handled by the last-resort pass).
-        if (self._get_free_vram_gb() < needed_gb and self.active_in_vram
+        if (self._get_free_vram_gb() < target_gb and self.active_in_vram
                 and self.active_in_vram not in self._keep_resident
                 and self.active_in_vram in self.models):
             _active = self.active_in_vram
@@ -6178,11 +6202,11 @@ class MultiModelManager:
         # co-resident STT/diarization models) so a genuinely-larger load — e.g. a
         # 26B LLM — can still fit. They yield rather than block it; that's the
         # "never fight the LLM" half of the co-residency policy.
-        if self._get_free_vram_gb() < needed_gb and self._keep_resident:
+        if self._get_free_vram_gb() < target_gb and self._keep_resident:
             for key in list(self._lru_order()):
                 if key not in self._keep_resident:
                     continue
-                if self._get_free_vram_gb() >= needed_gb:
+                if self._get_free_vram_gb() >= target_gb:
                     break
                 if self._is_key_busy(key) and not self._wait_until_idle(key):
                     print(f"  keep-resident '{key}' still busy — leaving it loaded")
@@ -6197,7 +6221,7 @@ class MultiModelManager:
         # model) to release their VRAM. These aren't tracked models, so eviction
         # above can't see them — this is what stops a cached training base from
         # forcing the next image request into CPU/disk offload.
-        if self._get_free_vram_gb() < needed_gb and self._external_vram_releasers:
+        if self._get_free_vram_gb() < target_gb and self._external_vram_releasers:
             for _rel in list(self._external_vram_releasers):
                 try:
                     freed = _rel(needed_gb)
@@ -6205,7 +6229,7 @@ class MultiModelManager:
                         print(f"On-request VRAM eviction: external releaser freed {freed:.1f} GB")
                 except Exception as e:
                     print(f"  Warning in external VRAM releaser: {e}")
-                if self._get_free_vram_gb() >= needed_gb:
+                if self._get_free_vram_gb() >= target_gb:
                     break
 
         # Report what we actually freed so a failure to free is visible.

tests/test_eviction_headroom.py

+"""Eviction frees 10% more than asked, so an estimate cannot land on the edge.
+
+Every size that reaches eviction is an estimate. One that lands a few hundred MB
+short does not fail honestly — the load is admitted, starts allocating, and dies
+on a tiny buffer. Measured on zeiss across 2026-10-30/31, each on a load that
+eviction had just reported as satisfied:
+
+    CUDA out of memory. Tried to allocate  20.00 MiB … 8.69 MiB is free
+    CUDA out of memory. Tried to allocate 190.00 MiB … 18.69 MiB is free
+    CUDA out of memory. Tried to allocate  12.00 MiB … 8.69 MiB is free
+    CUDA out of memory. Tried to allocate   9.00 GiB … 4.08 GiB is free
+
+The last of those is a genuinely too-large request; the first three are the edge.
+10% of the request costs at most one more idle model and puts the edge out of
+reach.
+
+The headroom is what eviction TARGETS, never what it reports. The shortfall
+warning measures against what the caller asked for, so freeing enough for the
+load but not the whole headroom is a success and stays silent — otherwise every
+successful eviction would print a warning, and the one real warning would be
+lost among them.
+"""
+
+import io
+import os
+import re
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+MGR = io.open(os.path.join(ROOT, 'codai/models/manager.py'), encoding='utf-8').read()
+
+
+def _fn():
+    i = MGR.index('    def _evict_models_for_vram(self, needed_gb: float):')
+    return MGR[i:MGR.index('\n    def ', i + 10)]
+
+
+def test_the_headroom_fraction_is_ten_percent():
+    from codai.models.manager import _EVICT_HEADROOM_FRACTION
+    assert _EVICT_HEADROOM_FRACTION == 0.10
+
+
+def test_the_target_is_the_request_plus_headroom():
+    assert 'target_gb = needed_gb * (1.0 + _EVICT_HEADROOM_FRACTION)' in _fn()
+
+
+def test_every_enough_yet_check_uses_the_padded_target():
+    """A pass still comparing against the bare request would stop early and
+    leave the edge case exactly where it was."""
+    body = _fn()
+    checks = re.findall(r'_get_free_vram_gb\(\)\s*(?:>=|<)\s*(\w+)', body)
+    assert checks, 'no free-VRAM comparisons found — did the function move?'
+    assert all(c == 'target_gb' for c in checks), checks
+
+
+def test_the_shortfall_warning_still_measures_the_request():
+    """Warning on the padded target would fire on every successful eviction."""
+    body = _fn()
+    i = body.index('_free_after = self._get_free_vram_gb()')
+    assert 'if _free_after < needed_gb:' in body[i:i + 200]
+
+
+def test_the_whole_card_verdict_still_uses_the_request():
+    """'needs more than the whole card' must compare the real requirement; with
+    the headroom a model needing exactly the card's size would be mislabelled."""
+    body = _fn()
+    assert 'needed_gb > _total_gb' in body
+
+
+def test_a_zero_or_negative_request_still_returns_early():
+    body = _fn()
+    assert body.index('if needed_gb <= 0:') < body.index('target_gb =')
+
+
+def test_the_reason_is_recorded_with_its_evidence():
+    body = _fn()
+    assert '20.00 MiB' in body and '8.69 MiB' in body, (
+        'the measurements that justify the margin are what stop it being '
+        'tuned away as arbitrary')
+
+
+def test_the_arithmetic():
+    """10% of the request, not of the card and not a flat constant: a flat
+    margin is too small for a 30 GB model and too large for a 0.1 GB one."""
+    from codai.models.manager import _EVICT_HEADROOM_FRACTION as H
+    for need, want in ((10.0, 11.0), (2.62, 2.882), (0.1, 0.11), (30.9, 33.99)):
+        assert abs(need * (1.0 + H) - want) < 0.001, (need, want)

tests/test_json_mode_closes_and_schema.py

+"""JSON mode returned JSON that would not parse.
+
+Measured on zeiss 2026-10-31: 21 script generations ended `finish_reason=length`
+after writing 6000 tokens of well-formed-but-unfinished JSON for a job needing
+about 800. One ran 8.5 minutes. The client reported them as "broken JSON after
+1493 s" and "second run had no scenes".
+
+Nothing was broken in the usual sense. `response_format: {"type":"json_object"}`
+is honoured — llama.cpp applies its generic JSON grammar — but that grammar
+constrains SYNTAX only. It never requires the root object to close, so an array
+of scenes can grow until the token budget ends mid-token. `max_tokens` was the
+caller's own 6000, the prompt was 2414 tokens and the model's n_ctx is 32000, so
+no limit was misconfigured and nothing was truncated by the server.
+
+Two fixes, neither of which touches the caller's max_tokens:
+
+1. A SCHEMA bounds the shape, so the object must close. llama-cpp-python accepts
+   only `type: json_object` and reads a sibling `schema` key, while OpenAI nests
+   the schema under `json_schema` — so the shapes are translated, not forwarded.
+   Forwarding OpenAI's form verbatim would silently DISABLE the grammar
+   (`_grammar_for_response_format` returns None for any other type), which is
+   worse than not supporting it.
+
+2. When the budget is hit anyway, the open structures are closed so the caller
+   gets valid JSON holding the items that did arrive. `finish_reason` stays
+   `length`, so a caller that checks still learns the answer was cut short.
+"""
+
+import io
+import json
+import os
+import sys
+import textwrap
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+SRC = io.open(os.path.join(ROOT, 'codai/backends/vulkan.py'), encoding='utf-8').read()
+
+
+@pytest.fixture(scope='module')
+def closer():
+    """The helper, lifted out so the test does not need a loaded model."""
+    i = SRC.index('    def close_truncated_json(text: str):')
+    j = SRC.index('    def _render_chat_prompt(self, messages):')
+    fn = textwrap.dedent(SRC[i:j]).replace('@staticmethod\n', '')
+    ns = {}
+    exec(fn, ns)
+    return ns['close_truncated_json']
+
+
+# ── closing a truncated document ─────────────────────────────────────────────
+
+def test_the_real_shape_a_cut_scene_array(closer):
+    out = closer('{"title":"x","scenes":[{"text":"one"},{"text":"two"},{"te')
+    assert out is not None
+    d = json.loads(out)
+    assert len(d['scenes']) == 2, 'the half-written scene should be dropped'
+    assert d['title'] == 'x'
+
+
+def test_a_string_cut_mid_word_is_closed(closer):
+    out = closer('{"title":"Perche i polpi sono intellige')
+    assert json.loads(out)['title'].startswith('Perche')
+
+
+def test_a_trailing_comma_is_removed(closer):
+    assert len(json.loads(closer('{"a":[{"x":1},'))['a']) == 1
+
+
+def test_deep_nesting_is_closed(closer):
+    assert json.loads(closer('{"a":{"b":{"c":[1,2,3'))['a']['b']['c'] == [1, 2]
+
+
+def test_valid_json_is_left_alone(closer):
+    assert closer('{"a": 1}') is None
+
+
+def test_prose_is_left_alone(closer):
+    """A model that ignored JSON mode entirely must not be "repaired" into
+    something that parses but means nothing."""
+    assert closer('Sure! Here is your script:') is None
+
+
+def test_empty_is_left_alone(closer):
+    assert closer('') is None and closer('   ') is None
+
+
+def test_an_unverifiable_repair_is_not_applied(closer):
+    """The function parses its own output before returning it; if it cannot be
+    made to parse, the original is left for the caller to see."""
+    assert closer('{"a": }') is None
+
+
+def test_it_never_invents_content(closer):
+    out = closer('{"title":"x","scenes":[{"text":"one"},{"text":"two"},{"te')
+    assert 'null' not in out and 'two"},{' not in out.replace('{"text":"two"}', '')
+
+
+# ── gating ───────────────────────────────────────────────────────────────────
+
+def test_the_repair_only_runs_for_a_json_request():
+    i = SRC.index('JSON response hit max_tokens mid-structure')
+    block = SRC[max(0, i - 1200):i]
+    assert "kwargs.get('response_format')" in block
+
+
+def test_the_repair_only_runs_on_a_length_finish():
+    i = SRC.index('JSON response hit max_tokens mid-structure')
+    block = SRC[max(0, i - 1200):i]
+    assert "_fr == 'length'" in block
+
+
+def test_finish_reason_is_not_rewritten():
+    """The caller must still be able to tell the answer was cut short."""
+    i = SRC.index('JSON response hit max_tokens mid-structure')
+    block = SRC[max(0, i - 1400):i + 600]
+    assert "finish_reason" not in block.split('_fixed = ')[1].split('return content')[0] \
+        or "stays 'length'" in block
+
+
+def test_a_failure_in_the_repair_cannot_fail_the_request():
+    i = SRC.index('JSON response hit max_tokens mid-structure')
+    block = SRC[max(0, i - 1200):i + 900]
+    assert 'except Exception' in block and 'close-up skipped' in block
+
+
+# ── schema support ───────────────────────────────────────────────────────────
+
+def test_both_response_format_shapes_are_accepted():
+    i = SRC.index("_rf.get('type') in ('json_object', 'json_schema')")
+    assert i > 0
+
+
+def test_openais_nested_schema_is_translated_not_forwarded():
+    """llama-cpp-python returns NO grammar for any type but json_object, so
+    forwarding OpenAI's `json_schema` verbatim would disable the constraint."""
+    i = SRC.index("_rf.get('type') in ('json_object', 'json_schema')")
+    block = SRC[i:i + 900]
+    assert "_js.get('schema')" in block
+    assert "{'type': 'json_object', 'schema': _schema}" in block
+
+
+def test_llamacpps_own_shape_still_works():
+    i = SRC.index("_rf.get('type') in ('json_object', 'json_schema')")
+    block = SRC[i:i + 900]
+    assert "_rf.get('schema')" in block
+
+
+def test_a_schemaless_json_object_request_is_unchanged():
+    i = SRC.index("_rf.get('type') in ('json_object', 'json_schema')")
+    block = SRC[i:i + 900]
+    assert "else {'type': 'json_object'})" in block
+
+
+def test_why_the_translation_matters_is_recorded():
+    i = SRC.index("_rf.get('type') in ('json_object', 'json_schema')")
+    head = SRC[max(0, i - 1800):i]
+    assert 'finish_reason=length' in head
+    assert 'silently disables the' in head or 'silently DISABLE' in head