compact an over-long conversation, against the window that is really there (v0.4.72)

Requests to a pod-served model were refused for exceeding the context window.
Auto-compaction has shipped for a while, but three things stood between it and
doing its job -- and one of them would have made enabling it worse than leaving
it off.

**The window was wrong.** A pod-served model's `ctx` lives in its runpod block
and becomes the pod's `--max-model-len`, but the backend applied the generic
`n_ctx` kwarg last -- and that kwarg carries the node-wide `vulkan.n_ctx`
fallback, default 2048, whenever the entry sets no `n_ctx` of its own. So a
model served with 32768 published 32768 on /v1/models (which reads the block)
and reported 2048 in every answer's `usage.context_size`. Compaction compacts to
a percentage of that number: switching it on would have crushed every
conversation to ~1300 tokens and looked like a model with no memory. The
block now wins, for pods and for vLLM, and a per-model `n_ctx` reaches the
launch so the reported window and the served one cannot disagree.

**The summary described the wrong messages.** It was built from "everything but
the last two turns", which includes the turns kept verbatim, so the kept head
and most of the kept tail were summarized AND kept -- paying twice in a window
that was already too small. The split is now decided once, and exactly the
dropped middle is summarized. The kept ends are configurable with it
(head_messages/head_tokens/tail_tokens/target_pct, globally and per model), the
head is reserved BEFORE the tail is filled (filling the tail first and adding
the head after overshoots the target by the whole head), and a head that would
end on an unanswered tool call is trimmed -- the mirror of the orphaned-result
rule the tail already had.

**The refusal was unreadable and final.** The numbers in it are the only exact
ones in the pipeline -- coderai estimates prompts at chars/4, the engine has
tokenized them -- and they were buried behind an apology, inside the pod's
wrapper, inside the orchestrator's, each prefixing its own status line and
120-character proxy URL. A client truncating the result got three layers of
prose and neither token count. Every layer now puts what it was told FIRST, the
message leads with `context_length_exceeded:` and both integers, and a refusal
is used rather than only reported: the real window is remembered for later
requests, the estimator is corrected by the ratio the engine just measured, and
the request is compacted against those numbers and sent once more
(compaction.retry_on_overflow, on by default).

Also fixes the layered fit, which mixed its units: the target was computed in
real tokens and handed to a splitter that measures in estimate tokens, so a
prompt the engine counts as too long could be judged already small enough and
left exactly as it was.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.71"
+__version__ = "0.4.72"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -2935,7 +2935,11 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                 "turboquant", "engine", "engine_fallback",
                 "quant_backend", "kv_cache_budget_mb", "kv_cache_slots", "mmproj",
                 "auto_compact", "auto_compact_pct", "auto_compact_strategy",
-                "auto_compact_model", "suppress_reasoning",
+                "auto_compact_model", "auto_compact_target_pct",
+                "auto_compact_head_messages", "auto_compact_head_tokens",
+                "auto_compact_tail_tokens", "auto_compact_tolerance_pct",
+                "auto_compact_min_output", "auto_compact_retry_on_overflow",
+                "suppress_reasoning",
                 # Speech-to-text (wav2vec2 / vosk / NeMo-Canary) options.
                 "languages", "supports_translation", "keep_resident",
                 # "local" pins the model here even when its whole capability is
@@ -4284,6 +4288,11 @@ def build_settings_dict(c, gpu_cards):
             "pct": c.compaction.pct,
             "strategy": c.compaction.strategy,
             "model": c.compaction.model,
+            "head_messages": c.compaction.head_messages,
+            "head_tokens": c.compaction.head_tokens,
+            "tail_tokens": c.compaction.tail_tokens,
+            "target_pct": c.compaction.target_pct,
+            "retry_on_overflow": c.compaction.retry_on_overflow,
         },
         "broker": {
             "enabled": c.broker.enabled,
@@ -5195,11 +5204,25 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
             except (TypeError, ValueError):
                 pass
         if "strategy" in cp:
-            _st = (cp.get("strategy") or "drop_oldest").strip()
-            if _st in ("drop_oldest", "keep_head_tail", "summarize"):
-                c.compaction.strategy = _st
+            # Accepted spellings live with the implementation, so a name the
+            # engine honours is never rejected here (and vice versa).
+            from codai.api.text import _COMPACT_STRATEGIES
+            _st = (cp.get("strategy") or "drop_oldest").strip().lower().replace("-", "_")
+            if _st in _COMPACT_STRATEGIES:
+                c.compaction.strategy = _COMPACT_STRATEGIES[_st]
         if "model" in cp:
             c.compaction.model = (cp.get("model") or "").strip()
+        # The kept ends. 0 is meaningful for all three (= "whatever fits"), so
+        # these are clamped at the bottom, not forced to a minimum.
+        for _k, _lo, _hi in (("head_messages", 0, 50), ("head_tokens", 0, 1_000_000),
+                             ("tail_tokens", 0, 1_000_000), ("target_pct", 10, 95)):
+            if _k in cp:
+                try:
+                    setattr(c.compaction, _k, max(_lo, min(_hi, int(cp[_k]))))
+                except (TypeError, ValueError):
+                    pass
+        if "retry_on_overflow" in cp:
+            c.compaction.retry_on_overflow = bool(cp["retry_on_overflow"])
 
     if "broker" in data:
         bro = data["broker"]

codai/admin/templates/models.html

@@ -1383,16 +1383,36 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
         <div class="form-row" style="margin:0">
           <label class="form-label">Trigger at <span class="muted">(% of context)</span></label>
           <input type="number" id="cfg-autocompact-pct" class="form-input" min="50" max="99" step="1" value="85">
-          <span class="form-hint" style="font-size:11px">Compacts to ~65% of the context window when the prompt reaches this %.</span>
+          <span class="form-hint" style="font-size:11px">Compacts down to the size below when the prompt reaches this %.</span>
         </div>
         <div class="form-row" style="margin:0">
           <label class="form-label">Strategy</label>
           <select id="cfg-autocompact-strategy" class="form-input" onchange="_toggleAutoCompact()">
             <option value="drop_oldest">Drop oldest — keep system + most recent turns</option>
-            <option value="keep_head_tail">Keep head + tail — also keep the first turn, drop the middle</option>
-            <option value="summarize">Summarize — replace the dropped middle with an LLM summary</option>
+            <option value="keep_head_tail">Keep head + tail — keep the opening turns and the recent ones, drop the middle</option>
+            <option value="summarize">Summarize the middle — keep both ends, replace the centre with an LLM summary</option>
           </select>
         </div>
+        <div class="form-row" style="margin:0">
+          <label class="form-label">Compact down to <span class="muted">(% of context)</span></label>
+          <input type="number" id="cfg-autocompact-target" class="form-input" min="10" max="95" step="1" value="65">
+          <span class="form-hint" style="font-size:11px">Lower leaves more room to grow before the next compaction.</span>
+        </div>
+        <div class="form-row" style="margin:0">
+          <label class="form-label">Keep at the start <span class="muted">(turns)</span></label>
+          <input type="number" id="cfg-autocompact-head-msgs" class="form-input" min="0" max="50" step="1" value="1">
+          <span class="form-hint" style="font-size:11px">From the first user turn. 0 = keep nothing at the start.</span>
+        </div>
+        <div class="form-row" style="margin:0">
+          <label class="form-label">Start cap <span class="muted">(tokens, 0 = auto)</span></label>
+          <input type="number" id="cfg-autocompact-head-tok" class="form-input" min="0" step="256" value="0">
+          <span class="form-hint" style="font-size:11px">0 = at most half the target, so the start never starves the recent end.</span>
+        </div>
+        <div class="form-row" style="margin:0">
+          <label class="form-label">Keep at the end <span class="muted">(tokens, 0 = the rest)</span></label>
+          <input type="number" id="cfg-autocompact-tail-tok" class="form-input" min="0" step="256" value="0">
+          <span class="form-hint" style="font-size:11px">Recent conversation kept verbatim.</span>
+        </div>
         <div class="form-row" id="cfg-autocompact-model-row" style="margin:0;grid-column:1/-1;display:none">
           <label class="form-label">Summarizer model <span class="muted">(summarize only)</span></label>
           <select id="cfg-autocompact-model" class="form-input">
@@ -4195,6 +4215,10 @@ function openCfgModal(idx, cfgIdx){
   document.getElementById('cfg-autocompact').checked = !!s.auto_compact;
   document.getElementById('cfg-autocompact-pct').value = s.auto_compact_pct != null ? s.auto_compact_pct : 85;
   document.getElementById('cfg-autocompact-strategy').value = s.auto_compact_strategy || 'drop_oldest';
+  document.getElementById('cfg-autocompact-target').value = s.auto_compact_target_pct != null ? s.auto_compact_target_pct : 65;
+  document.getElementById('cfg-autocompact-head-msgs').value = s.auto_compact_head_messages != null ? s.auto_compact_head_messages : 1;
+  document.getElementById('cfg-autocompact-head-tok').value = s.auto_compact_head_tokens || 0;
+  document.getElementById('cfg-autocompact-tail-tok').value = s.auto_compact_tail_tokens || 0;
   _populateCompactModelSelect(s.auto_compact_model || '');
   document.getElementById('cfg-suppress-reasoning').checked = !!s.suppress_reasoning;
   _toggleAutoCompact();
@@ -5233,6 +5257,10 @@ async function saveModelConfig(){
     auto_compact:      document.getElementById('cfg-autocompact').checked,
     auto_compact_pct:  parseInt(document.getElementById('cfg-autocompact-pct').value) || 85,
     auto_compact_strategy: document.getElementById('cfg-autocompact-strategy').value || 'drop_oldest',
+    auto_compact_target_pct: parseInt(document.getElementById('cfg-autocompact-target').value) || 65,
+    auto_compact_head_messages: parseInt(document.getElementById('cfg-autocompact-head-msgs').value) || 0,
+    auto_compact_head_tokens: parseInt(document.getElementById('cfg-autocompact-head-tok').value) || 0,
+    auto_compact_tail_tokens: parseInt(document.getElementById('cfg-autocompact-tail-tok').value) || 0,
     auto_compact_model: document.getElementById('cfg-autocompact-model').value || '',
     suppress_reasoning: document.getElementById('cfg-suppress-reasoning').checked,
     max_gpu_percent:   isNaN(maxGpu) ? null : maxGpu,

codai/admin/templates/settings.html

@@ -1536,10 +1536,39 @@
         <label class="form-label">Strategy</label>
         <select id="s-compact-strategy" class="form-input">
           <option value="drop_oldest">Drop oldest — keep system + most recent turns</option>
-          <option value="keep_head_tail">Keep head + tail — also keep the first turn, drop the middle</option>
-          <option value="summarize">Summarize — replace the dropped middle with an LLM summary</option>
+          <option value="keep_head_tail">Keep head + tail — keep the opening turns and the recent ones, drop the middle</option>
+          <option value="summarize">Summarize the middle — keep both ends, replace the centre with an LLM summary</option>
         </select>
       </div>
+      <div class="form-row">
+        <label class="form-label">Compact down to <span class="muted">(% of context)</span></label>
+        <input type="number" id="s-compact-target" class="form-input" min="10" max="95" step="1" placeholder="65">
+        <span class="form-hint">The size the compacted prompt aims for. Lower leaves more room for the conversation to grow again before the next compaction.</span>
+      </div>
+    </div>
+    <div style="display:grid;grid-template-columns:1fr 1fr 1fr;gap:.75rem">
+      <div class="form-row">
+        <label class="form-label">Keep at the start <span class="muted">(turns)</span></label>
+        <input type="number" id="s-compact-head-msgs" class="form-input" min="0" max="50" step="1" placeholder="1">
+        <span class="form-hint">Opening turns kept verbatim, counted from the first user turn — the instructions a long conversation drifts away from. 0 = keep nothing at the start.</span>
+      </div>
+      <div class="form-row">
+        <label class="form-label">Start cap <span class="muted">(tokens, 0 = auto)</span></label>
+        <input type="number" id="s-compact-head-tok" class="form-input" min="0" step="256" placeholder="0">
+        <span class="form-hint">Upper bound on the kept start. 0 = at most half the target, so the start can never starve the recent end.</span>
+      </div>
+      <div class="form-row">
+        <label class="form-label">Keep at the end <span class="muted">(tokens, 0 = the rest)</span></label>
+        <input type="number" id="s-compact-tail-tok" class="form-input" min="0" step="256" placeholder="0">
+        <span class="form-hint">Recent conversation kept verbatim. 0 = everything the target still allows after the start.</span>
+      </div>
+    </div>
+    <div class="form-row">
+      <label style="display:flex;align-items:center;gap:.5rem;cursor:pointer">
+        <input type="checkbox" id="s-compact-retry">
+        <span style="font-size:13px;font-weight:500">Retry once when an engine refuses a request for length</span>
+      </label>
+      <span class="form-hint">A refusal states the real window and the real prompt size, which an estimate can both get wrong — a proxied model is launched elsewhere, and characters÷4 undercounts dense text. With this on, the request is compacted against those numbers and sent again instead of failing, and the corrected window is remembered for later requests.</span>
     </div>
     <div class="form-row">
       <label class="form-label">Summarizer model <span class="muted">(summarize strategy)</span></label>
@@ -2166,6 +2195,11 @@ async function loadSettings(){
     document.getElementById('s-compact-pct').value = compact.pct ?? 85;
     document.getElementById('s-compact-strategy').value = compact.strategy || 'drop_oldest';
     document.getElementById('s-compact-model').value = compact.model ?? '';
+    document.getElementById('s-compact-target').value = compact.target_pct ?? 65;
+    document.getElementById('s-compact-head-msgs').value = compact.head_messages ?? 1;
+    document.getElementById('s-compact-head-tok').value = compact.head_tokens ?? 0;
+    document.getElementById('s-compact-tail-tok').value = compact.tail_tokens ?? 0;
+    document.getElementById('s-compact-retry').checked = compact.retry_on_overflow !== false;
     toggleCompactFields();
     // Thermal protection
     const therm = d.thermal || {};
@@ -2749,6 +2783,11 @@ async function saveSettings(silent){
       pct: parseInt(document.getElementById('s-compact-pct').value) || 85,
       strategy: document.getElementById('s-compact-strategy').value || 'drop_oldest',
       model: document.getElementById('s-compact-model').value.trim(),
+      target_pct: parseInt(document.getElementById('s-compact-target').value) || 65,
+      head_messages: parseInt(document.getElementById('s-compact-head-msgs').value) || 0,
+      head_tokens: parseInt(document.getElementById('s-compact-head-tok').value) || 0,
+      tail_tokens: parseInt(document.getElementById('s-compact-tail-tok').value) || 0,
+      retry_on_overflow: document.getElementById('s-compact-retry').checked,
     },
     broker:{
       enabled: document.getElementById('s-broker-enabled').checked,

codai/api/context_overflow.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""Reading a "does not fit the context window" refusal, and answering one.
+
+Two jobs, both about the same two numbers -- the window, and what was asked of
+it.
+
+**Parsing.** Every engine words the refusal differently, and the numbers in it
+are the only exact ones in the whole pipeline: coderai's own prompt size is a
+chars/4 estimate, while the engine has tokenized the prompt. So when a request
+is refused, the refusal is worth more than the estimate that let it through --
+it says what the window really is (which can disagree with the configured
+``ctx``) and exactly how far over the prompt went. :func:`parse` extracts both,
+and :func:`remember_limit` keeps the window so the next request is compacted
+against the served truth instead of the configured guess.
+
+**Answering.** The numbers go FIRST in the message. This is not cosmetic. A
+refusal crosses as many as three processes -- vLLM inside the pod, the pod's own
+coderai, then this one -- and each wraps the one below it with its own prefix,
+status line and URL. A client that reads a truncated prefix of the result (300,
+400, 1000 characters: every client picks a cap, and the pod proxy URL alone can
+be 120 characters) was getting the apologies and none of the figures, so it
+could not tell an over-long conversation from any other 500 and had nothing to
+retry with. Front-loading them means the first clause carries everything a
+caller needs to shrink the request and try again:
+
+    context_length_exceeded: 35000 tokens requested, limit 32768 (34000 prompt
+    + 1000 completion) for model 'qwen38' -- ...
+
+A caller matching on ``context_length_exceeded`` or reading the two integers
+sees them whatever the cap, and the OpenAI-style ``code`` is the same string, so
+matching on either works.
+"""
+
+from __future__ import annotations
+
+import re
+import threading
+from typing import Dict, Optional
+
+#: The OpenAI error code for this condition, and the token clients match on.
+CODE = "context_length_exceeded"
+
+#: Phrasings seen in the wild, each yielding what it knows. Order matters only
+#: in that the first match wins, so the most specific patterns come first.
+_PATTERNS = (
+    # vLLM / OpenAI: the richest phrasing, and the one a pod answers with.
+    re.compile(
+        r"maximum context length is (?P<limit>\d+) tokens.{0,40}?"
+        r"you requested (?P<requested>\d+) tokens?"
+        r"(?:\D{0,40}?(?P<prompt>\d+) in the messages"
+        r"\D{0,40}?(?P<completion>\d+) in the completion)?",
+        re.IGNORECASE | re.DOTALL),
+    # vLLM V1 engine, prompt-only phrasing.
+    re.compile(
+        r"(?:decoder )?prompt \(length (?P<prompt>\d+)\) is longer than the "
+        r"maximum model length of (?P<limit>\d+)", re.IGNORECASE),
+    # llama-cpp-python.
+    re.compile(
+        r"requested tokens \((?P<requested>\d+)\) exceed context window of "
+        r"(?P<limit>\d+)", re.IGNORECASE),
+    # coderai's own pre-flight refusal (see codai.api.text), so a relayed
+    # message from a downstream coderai parses as well as an engine's.
+    re.compile(
+        r"(?P<requested>\d+) tokens requested, limit (?P<limit>\d+)",
+        re.IGNORECASE),
+    re.compile(
+        r"~?(?P<prompt>\d+) prompt tokens vs n_ctx=(?P<limit>\d+)",
+        re.IGNORECASE),
+)
+
+#: Text that means "it did not fit" even when no pattern yields numbers. Kept
+#: deliberately broad: treating an overflow as an ordinary 500 costs a retry
+#: that would have worked, while the reverse costs one pointless compaction.
+_MARKERS = (
+    "maximum context length", "context length", "context window",
+    "context size", "available context", "exceed context", "exceeds n_ctx",
+    "n_ctx", "maximum model length",
+    "kv cache is full", "context shift is disabled", "too many tokens",
+    "reduce the length of the messages", CODE,
+)
+
+#: ``{model: limit}`` learned from refusals. Process-local on purpose: it is a
+#: correction to a configured number, not a configuration, and it must not
+#: outlive a pod whose window really did change.
+_LEARNED: Dict[str, int] = {}
+_LOCK = threading.Lock()
+
+
+def parse(text) -> Optional[dict]:
+    """``{limit, requested, prompt, completion}`` from a refusal, or None.
+
+    Only the keys the message actually stated are present, except that
+    ``requested`` is derived from prompt+completion (or prompt alone) when the
+    engine reported the parts but not the total -- a caller comparing "asked"
+    against "allowed" should not have to know which phrasing it got.
+    """
+    s = str(text or "")
+    if not s:
+        return None
+    for pat in _PATTERNS:
+        m = pat.search(s)
+        if not m:
+            continue
+        got = {}
+        for key, raw in (m.groupdict() or {}).items():
+            if raw is None:
+                continue
+            try:
+                got[key] = int(raw)
+            except (TypeError, ValueError):
+                continue
+        if "requested" not in got:
+            if "prompt" in got:
+                got["requested"] = got["prompt"] + got.get("completion", 0)
+        if got.get("limit") or got.get("requested"):
+            return got
+    return None
+
+
+def is_overflow(text) -> bool:
+    """Whether this text is an engine refusing a request for being too long."""
+    low = str(text or "").lower()
+    if not low:
+        return False
+    if any(m in low for m in _MARKERS):
+        return True
+    # "exceeds the model's 32768 token limit" and similar: a token count and an
+    # exceedance, which together are not said about anything else.
+    return "token" in low and ("exceed" in low or "too long" in low)
+
+
+def remember_limit(model: str, limit) -> Optional[int]:
+    """Record the window an engine says it has, and return it.
+
+    The served window and the configured one disagree in practice, and only the
+    served one can refuse a request. Learning it from the refusal costs nothing
+    -- no probe, no extra request -- and it is what makes a retry land inside
+    the window rather than one estimate closer to it.
+    """
+    try:
+        n = int(limit or 0)
+    except (TypeError, ValueError):
+        return None
+    name = str(model or "").strip()
+    if not name or n <= 0:
+        return None
+    with _LOCK:
+        _LEARNED[name] = n
+    return n
+
+
+def learned_limit(model: str) -> Optional[int]:
+    with _LOCK:
+        return _LEARNED.get(str(model or "").strip())
+
+
+def learn_from(model: str, text) -> Optional[dict]:
+    """Parse a refusal and remember the limit it states. Returns the parse."""
+    info = parse(text)
+    if info and info.get("limit"):
+        remember_limit(model, info["limit"])
+    return info
+
+
+def forget(model: Optional[str] = None) -> None:
+    with _LOCK:
+        if model is None:
+            _LEARNED.clear()
+        else:
+            _LEARNED.pop(str(model or "").strip(), None)
+
+
+def message(info: Optional[dict] = None, model: str = "",
+            compacted: bool = False, raw: str = "") -> str:
+    """The client-facing refusal, numbers first. See the module docstring.
+
+    ``raw`` (the engine's own words) is appended rather than replaced: it can
+    name a cause the numbers do not, and by then it is past the figures that a
+    truncating client needs.
+    """
+    info = info or {}
+    limit = info.get("limit")
+    requested = info.get("requested")
+    bits = []
+    if requested and limit:
+        bits.append(f"{requested} tokens requested, limit {limit}")
+    elif limit:
+        bits.append(f"over the {limit}-token limit")
+    elif requested:
+        bits.append(f"{requested} tokens requested")
+    else:
+        bits.append("the request is longer than the context window")
+    parts = []
+    if info.get("prompt"):
+        parts.append(f"{info['prompt']} prompt")
+    if info.get("completion"):
+        parts.append(f"{info['completion']} completion")
+    if parts:
+        bits.append("(" + " + ".join(parts) + ")")
+    if model:
+        bits.append(f"for model '{model}'")
+    head = f"{CODE}: " + " ".join(bits)
+    tail = ("even after auto-compaction. Shorten the conversation, lower "
+            "max_tokens, or raise the model's context size."
+            if compacted else
+            "The conversation is too long for this model's context window. "
+            "Shorten it, lower max_tokens, enable auto-compaction for this "
+            "model, or raise its context size.")
+    out = f"{head} — {tail}"
+    raw = str(raw or "").strip()
+    if raw and raw not in out:
+        out += f" (engine said: {raw})"
+    return out

codai/backends/runpod.py

@@ -32,6 +32,14 @@ class RemoteModelUnknown(RuntimeError):
     """The pod answered that it does not know the model we asked for."""
 
 
+#: How much of a pod's explanation is kept. A pod's answer is itself a stack of
+#: wrapped messages -- vLLM's words inside the pod's coderai inside its HTTP
+#: error -- so 400 characters could end mid-prefix: a request refused for being
+#: too long arrived with the apology and WITHOUT either of the two token counts
+#: a caller needs to shrink it and retry, which read as an ordinary 500.
+_DETAIL_CHARS = 1200
+
+
 def _pod_detail(r) -> str:
     """What the pod SAID, not what requests says about the status line.
 
@@ -43,15 +51,15 @@ def _pod_detail(r) -> str:
     try:
         body = r.json()
     except Exception:
-        return (getattr(r, "text", "") or "").strip()[:400]
+        return (getattr(r, "text", "") or "").strip()[:_DETAIL_CHARS]
     if isinstance(body, dict):
         err = body.get("error")
         if isinstance(err, dict) and err.get("message"):
-            return str(err["message"])[:400]
+            return str(err["message"])[:_DETAIL_CHARS]
         for key in ("detail", "message", "error"):
             if body.get(key):
-                return str(body[key])[:400]
-    return str(body)[:400]
+                return str(body[key])[:_DETAIL_CHARS]
+    return str(body)[:_DETAIL_CHARS]
 
 
 class RunpodBackend(ModelBackend):
@@ -121,8 +129,17 @@ class RunpodBackend(ModelBackend):
         self._mcfg = runpod_worker.parse_model_runpod(block)
         # served name: explicit served_model wins, else the model's own name.
         self._served_name = self._mcfg.served_model or model_name or "runpod"
-        if self._mcfg.ctx:
-            self._ctx = self._mcfg.ctx
+        # The pod's own `ctx` wins over the generic kwarg, and that order is not
+        # a preference: `ctx` in the model's runpod block is what the pod is
+        # LAUNCHED with (`--max-model-len`), while the kwarg can be a global
+        # fallback no one chose for this model -- the node-wide `vulkan.n_ctx`,
+        # whose default is 2048. Taking the kwarg first made every pod-served
+        # model on an install without a per-model `n_ctx` report a 2048-token
+        # window while its pod served 32768: measured on a production install
+        # 2026-10-22, where `usage.context_size` said 2048, `/v1/models` said
+        # 32768 (it reads the runpod block), and auto-compaction, had it been
+        # enabled, would have crushed every conversation to ~1300 tokens to fit
+        # a window that was sixteen times larger.
         _ctx = kwargs.get("n_ctx", kwargs.get("ctx"))
         if isinstance(_ctx, (list, tuple)):
             _ctx = _ctx[0] if _ctx else None
@@ -130,7 +147,9 @@ class RunpodBackend(ModelBackend):
             _ctx = int(_ctx) if _ctx else 0
         except (TypeError, ValueError):
             _ctx = 0
-        if _ctx > 0:
+        if self._mcfg.ctx:
+            self._ctx = self._mcfg.ctx
+        elif _ctx > 0:
             self._ctx = _ctx
 
         self._mode = (self._mcfg.mode or "pods").lower()
@@ -342,8 +361,13 @@ class RunpodBackend(ModelBackend):
         detail = _pod_detail(r)
         if _is_unknown_model(r.status_code, (detail or "").encode("utf-8", "replace")):
             raise RemoteModelUnknown(detail)
+        # The pod's words first, its status code after: this message is wrapped
+        # once more before a client sees it, and the figures in it (which window,
+        # how many tokens) are what the client needs from a prefix.
         raise requests.HTTPError(
-            f"the pod answered {r.status_code}: {detail}", response=r)
+            f"{detail} (the pod answered {r.status_code})" if detail
+            else f"the pod answered {r.status_code} with no explanation",
+            response=r)
 
     def _teach(self, base: str, bearer: str = "") -> bool:
         """Tell this pod about our model after it said it has never heard of it.

codai/backends/vllm.py

@@ -75,13 +75,36 @@ class VllmBackend(ModelBackend):
             _ctx = int(_ctx) if _ctx else 0
         except (TypeError, ValueError):
             _ctx = 0
-        if _ctx > 0:
-            self._ctx = _ctx
         model_path = self._resolve_model_dir(model_name)
         # This model's own `vllm` block over the global template: parallelism,
         # the machines it spans, memory fraction, flags.
         from codai.backends.overrides import VLLM_FIELDS, apply, model_block
-        self._cfg = apply(self._cfg, model_block(model_name, "vllm"), VLLM_FIELDS, "vllm")
+        _blk = model_block(model_name, "vllm") or {}
+        self._cfg = apply(self._cfg, _blk, VLLM_FIELDS, "vllm")
+        # The window this model is SERVED with, which is the only one worth
+        # reporting: the model's own `vllm.ctx` when it sets one, else a
+        # per-model `n_ctx`, else the global template. The generic kwarg must not
+        # win over an explicit block, because it can be a node-wide fallback
+        # nobody chose for this model -- `vulkan.n_ctx`, whose default is 2048 --
+        # and a 2048 reported against a 32768 server makes auto-compaction crush
+        # every conversation to a sixteenth of the window it has (measured on a
+        # production install 2026-10-22).
+        _blk_ctx = 0
+        try:
+            _blk_ctx = int((_blk or {}).get("ctx") or 0)
+        except (TypeError, ValueError):
+            _blk_ctx = 0
+        if not _blk_ctx and _ctx > 0:
+            # An explicit per-model n_ctx has to reach the LAUNCH too, or the
+            # reported window and the served one disagree by design.
+            try:
+                self._cfg = apply(self._cfg, {"ctx": _ctx}, VLLM_FIELDS, "vllm")
+            except Exception:
+                pass
+        try:
+            self._ctx = int(getattr(self._cfg, "ctx", 0) or 0) or self._ctx
+        except (TypeError, ValueError):
+            pass
         # Per-model: serve each model-list entry under ITS OWN name (like nvidia/radeon),
         # not a single engine-wide model_id. Fall back to the config model_id only for the
         # single-model convenience (a model with no list entry).
@@ -136,6 +159,16 @@ class VllmBackend(ModelBackend):
         return self._model_id
 
     def get_context_size(self) -> int:
+        """The window this model is served with.
+
+        The SERVED number when the server has already been asked for it (see
+        context_length), because that is the only one that can refuse a request;
+        the configured one otherwise. They disagree in practice, and a caller
+        sizing a conversation against the larger of the two gets a 400.
+        """
+        served = getattr(self, "_ctx_len", None)
+        if isinstance(served, int) and served > 0:
+            return served
         return self._ctx
 
     def get_last_usage(self) -> dict:
@@ -284,11 +317,18 @@ class VllmBackend(ModelBackend):
         if len(detail) > 2000:
             detail = detail[:2000] + " ...[truncated]"
         import requests as _rq
+        # What the server SAID comes first, the status line and URL after it.
+        # This error is wrapped again by every layer above -- a pod's coderai,
+        # then the orchestrator -- and each prepends its own prefix, so putting
+        # a 120-character proxy URL in front of the explanation pushes it past
+        # wherever the client truncates. The numbers in "maximum context length
+        # is 32768 ... you requested 35000" are the whole message, and a caller
+        # that reads only a prefix was getting none of them.
+        where = (f"{code} {getattr(r, 'reason', '')} from "
+                 f"{getattr(r, 'url', '?')} on {what}")
         raise _rq.HTTPError(
-            f"{code} {getattr(r, 'reason', '')} from "
-            f"{getattr(r, 'url', '?')} on {what}"
-            + (f": {detail}" if detail else " (the server sent no explanation)"),
-            response=r)
+            f"{detail} ({where})" if detail
+            else f"{where} (the server sent no explanation)", response=r)
 
     #: Never overridable by a caller: these are the request's structure, not its
     #: sampling. Letting `passthrough` rewrite them would let a client retarget

codai/config.py

@@ -421,8 +421,19 @@ class CompactionConfig:
 
     Per-model settings in a models.json entry (``auto_compact``,
     ``auto_compact_pct``, ``auto_compact_strategy``, ``auto_compact_model``,
-    ``auto_compact_tolerance_pct``) OVERRIDE the values here; when a model leaves
-    one unset, the global default below applies. ``model`` selects which model
+    ``auto_compact_tolerance_pct``, ``auto_compact_head_messages``,
+    ``auto_compact_head_tokens``, ``auto_compact_tail_tokens``,
+    ``auto_compact_target_pct``, ``auto_compact_retry_on_overflow``) OVERRIDE the
+    values here; when a model leaves one unset, the global default below applies.
+
+    The strategies are ``drop_oldest`` (keep the system messages and the recent
+    tail), ``keep_head_tail`` (also keep the opening turns, drop the centre) and
+    ``summarize`` (keep both ends and replace the centre with an LLM summary of
+    exactly the messages being dropped -- "keep the start and the end, compact
+    the middle"; ``summarize_middle`` is accepted as a name for it). An unknown
+    name is read as ``keep_head_tail`` rather than silently dropping the head.
+
+    ``model`` selects which model
     performs the summarization for the ``summarize`` strategy — empty means use
     the same model that serves the request. Pointing it at a smaller/faster model
     lets that model summarize the old turns while the big model answers; the
@@ -442,6 +453,22 @@ class CompactionConfig:
     tolerance_pct: int = 20              # accept prompt+max_tokens up to this % over n_ctx (trim instead of compact)
     min_output: int = 512                # always try to leave at least this many tokens for the reply
     estimate_safety: float = 1.15        # inflate the chars/4 prompt estimate by this factor for fit/trim (it undercounts code/JSON)
+    # What the kept ends are, for keep_head_tail and summarize. The head is the
+    # opening instructions a long conversation keeps drifting away from, so it is
+    # sized in MESSAGES (one turn is the anchor; a handful is a briefing), while
+    # the tail is sized in TOKENS because what matters there is how much recent
+    # conversation survives verbatim, not how many turns it took. 0 means "as
+    # much as the target allows": the whole remaining budget for the tail, half
+    # of it for the head.
+    head_messages: int = 1               # leading turns kept verbatim (from the first user turn)
+    head_tokens: int = 0                 # cap on the head, in tokens (0 = half the target)
+    tail_tokens: int = 0                 # recent turns kept verbatim, in tokens (0 = the rest)
+    target_pct: int = 65                 # compact DOWN to this % of n_ctx (leaves room to grow)
+    # Retry a request an engine refused for length, compacting against the window
+    # and prompt size the refusal reports. On by default: without it a caller is
+    # told to shorten a conversation coderai was configured to shorten for them,
+    # and the correction is learned too late to help that request.
+    retry_on_overflow: bool = True
 
 
 @dataclass
@@ -2355,6 +2382,11 @@ class ConfigManager:
                 "tolerance_pct": self.config.compaction.tolerance_pct,
                 "min_output": self.config.compaction.min_output,
                 "estimate_safety": self.config.compaction.estimate_safety,
+                "head_messages": self.config.compaction.head_messages,
+                "head_tokens": self.config.compaction.head_tokens,
+                "tail_tokens": self.config.compaction.tail_tokens,
+                "target_pct": self.config.compaction.target_pct,
+                "retry_on_overflow": self.config.compaction.retry_on_overflow,
             },
             "broker": {
                 "enabled": self.config.broker.enabled,

tests/test_proxy_context_window.py

+"""The window a proxied model reports must be the one it is served with.
+
+2026-10-22, production: `qwen38` is served from a rented pod launched with
+`--max-model-len 32768`, `/v1/models` published 32768 -- and every answer's
+`usage.context_size` said **2048**. Two numbers for one model, from one process.
+
+The cause is a precedence, not a typo. A pod-served model's window lives in its
+`runpod` block (`ctx`, which becomes the pod's `--max-model-len`), while the
+backend also accepts a generic `n_ctx` kwarg -- and that kwarg carries the
+NODE-WIDE fallback `vulkan.n_ctx` when the model's entry sets no `n_ctx` of its
+own. Its default is 2048. The kwarg was applied last, so a global default for
+local llama.cpp models silently became the published window of every pod-served
+model on the install.
+
+Reporting 2048 was the visible half. The dangerous half is auto-compaction,
+which compacts to a percentage of exactly this number: enabling it would have
+crushed every conversation to ~1300 tokens to fit a window sixteen times
+larger, and the summaries would have looked like a model that could not
+remember anything.
+"""
+import types
+
+import pytest
+
+from codai.backends.runpod import RunpodBackend
+from codai.backends.vllm import VllmBackend
+
+
+# --------------------------------------------------------------- pods (proxy) #
+
+def _runpod(monkeypatch, block_ctx, **load_kwargs):
+    """A RunpodBackend loaded with a runpod block of ``block_ctx`` tokens."""
+    b = RunpodBackend.__new__(RunpodBackend)
+    b._ctx = 32768
+    b._model_id = b._served_name = "qwen38"
+    b._acct = None
+    b._headers = {}
+    b._pool = None
+    mcfg = types.SimpleNamespace(
+        ctx=block_ctx, served_model="", mode="serverless", endpoint_id="e1",
+        min_pods=0, max_pods=1)
+    monkeypatch.setattr("codai.api.runpod_worker.model_runpod_block",
+                        lambda name: {}, raising=False)
+    monkeypatch.setattr("codai.api.runpod_worker.parse_model_runpod",
+                        lambda blk: mcfg, raising=False)
+    monkeypatch.setattr("codai.api.runpod_worker.serverless_base_url",
+                        lambda acct, cfg: "https://pod:1/v1", raising=False)
+    monkeypatch.setattr("codai.api.runpod_worker.auth_headers",
+                        lambda acct: {}, raising=False)
+    b.load_model("qwen38", **load_kwargs)
+    return b
+
+
+def test_the_pods_own_ctx_beats_the_node_wide_default(monkeypatch):
+    """THE BUG: `vulkan.n_ctx` (default 2048) arrived as n_ctx and overwrote the
+    32768 the pod is actually launched with."""
+    b = _runpod(monkeypatch, 32768, n_ctx=2048)
+    assert b.get_context_size() == 32768
+
+
+def test_a_model_with_no_pod_ctx_still_takes_the_kwarg(monkeypatch):
+    """The fallback is a fallback, not dead code: with nothing configured for the
+    pod, the only number available is the one passed in."""
+    b = _runpod(monkeypatch, 0, n_ctx=8192)
+    assert b.get_context_size() == 8192
+
+
+def test_with_neither_the_backend_keeps_its_own_default(monkeypatch):
+    b = _runpod(monkeypatch, 0)
+    assert b.get_context_size() == 32768
+
+
+def test_a_junk_kwarg_cannot_shrink_the_window(monkeypatch):
+    """A list or a string reaching n_ctx must not silently become the window --
+    and must not raise inside the request path either."""
+    for junk in ([], "", "nonsense", None, [4096]):
+        b = _runpod(monkeypatch, 65536, n_ctx=junk)
+        assert b.get_context_size() == 65536
+
+
+# ------------------------------------------------------------- vLLM (in-process)
+
+def _vllm(monkeypatch, block, **load_kwargs):
+    from codai.config import VllmConfig
+    b = VllmBackend.__new__(VllmBackend)
+    b._cfg = VllmConfig()
+    b._model_id = "m"
+    b._svc_key = None
+    b._url = None
+    b._ctx = int(b._cfg.ctx)
+    b._served_name = "m"
+    b._last_usage = {}
+    b._last_logprobs = None
+    b._last_tool_calls = None
+    b._last_reasoning = None
+    monkeypatch.setattr(b, "_resolve_model_dir", lambda name: "/models/m")
+    monkeypatch.setattr("codai.backends.overrides.model_block",
+                        lambda name, key: dict(block))
+    monkeypatch.setattr("codai.api.vllm_worker.resolve_service_key",
+                        lambda cfg, path: (path, "svc"), raising=False)
+    monkeypatch.setattr("codai.api.vllm_worker.ensure_service",
+                        lambda cfg, **kw: "http://127.0.0.1:41721", raising=False)
+    b.load_model("m", **load_kwargs)
+    return b
+
+
+def test_the_models_vllm_block_beats_the_node_wide_default(monkeypatch):
+    b = _vllm(monkeypatch, {"ctx": 131072}, n_ctx=2048)
+    assert b.get_context_size() == 131072
+
+
+def test_a_per_model_n_ctx_reaches_the_launch_as_well(monkeypatch):
+    """Reporting a window the server was not started with is the same bug in
+    reverse, so an explicit n_ctx has to change what is launched too."""
+    b = _vllm(monkeypatch, {}, n_ctx=65536)
+    assert b.get_context_size() == 65536
+    assert int(b._cfg.ctx) == 65536        # ...and that is what vLLM is given
+
+
+def test_the_served_window_wins_once_the_server_has_been_asked(monkeypatch):
+    """The configured number is a request; the served one can refuse a request.
+    vLLM reports its own on /v1/models, and once that answer is in hand it is
+    the truth -- measured on an install where `ctx: 18432` was configured and
+    `--max-model-len 32768` was served."""
+    b = _vllm(monkeypatch, {"ctx": 18432})
+    assert b.get_context_size() == 18432
+    b._ctx_len = 32768                      # as context_length() caches it
+    assert b.get_context_size() == 32768
+
+
+def test_an_unknown_served_window_is_not_mistaken_for_zero(monkeypatch):
+    b = _vllm(monkeypatch, {"ctx": 32768})
+    b._ctx_len = None                       # context_length() tried and failed
+    assert b.get_context_size() == 32768