runpod: a proxy that edits the request decides for the server

Three silent drops on the way to a rented pod, all found while wiring Surya to
the model layer, all of the same shape: coderai accepted a field, forwarded a
reduced version of it, and the pod answered something plausible.

* Images. The front flattens multipart content to the literal text
  "[image_url content]" unless the backend claims vision, and only the llama.cpp
  backend ever did. So every page sent to a VLM served on RunPod — olmOCR in
  "model" mode, or Surya through a coderai-vllm pod — arrived as that
  placeholder, and the model answered fluently about nothing. A proxy is the
  wrong place to judge: forwarded, a text-only server returns a plain 400 and a
  VLM server reads the page.

* Structured output. Only {"type":"json_object"} survived, so a json_schema —
  the strict form current clients send, Surya's guided decoding among them — was
  dropped and the reply came back as free text the caller could not parse. The
  streamed path ignored the field entirely, json_object included.

* extra_args. Whitelisted per engine on a model's runpod block since pods
  shipped, parsed, stored, and never put on a command line. Setting it looked
  like tuning and did nothing, which is worse than refusing it: the pod boots,
  serves, and ignores the decision. A page-reading model wants prefix caching
  and the Qwen-VL pixel bounds, and until now the only way in was docker_args,
  which replaces the whole generated line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/api/runpod_worker.py

@@ -1350,7 +1350,25 @@ def _llamacpp_docker_args(mcfg: "RunpodModelConfig", served: str,
                 "direct download. A local .gguf path does not exist on a rented pod.")
     if mcfg.ctx and mcfg.ctx > 0:
         args += ["-c", str(mcfg.ctx)]
-    return " ".join(args)
+    return _with_extra_args(args, mcfg)
+
+
+def _with_extra_args(args: list, mcfg: "RunpodModelConfig") -> str:
+    """The generated flags, then whatever the operator added on the model entry.
+
+    ``extra_args`` has been accepted on a runpod block (and whitelisted per engine)
+    since pods shipped, and until now went nowhere: it was parsed, stored and never
+    put on a command line. Setting it looked like tuning and did nothing — which is
+    worse than refusing it, because the pod boots, serves, and quietly ignores the
+    decision. The two OCR VLMs are the case that noticed: a page-reading model wants
+    prefix caching (the prompt repeats on every page) and the Qwen-VL pixel bounds,
+    and neither could be expressed.
+
+    Appended verbatim and LAST, so an explicit flag overrides a generated one the way
+    it does on any command line. ``docker_args`` still replaces the whole line for
+    anyone who wants nothing of ours."""
+    extra = (getattr(mcfg, "extra_args", "") or "").strip()
+    return " ".join(args) + (" " + extra if extra else "")
 
 
 def _as_bool(v, default=False):
@@ -1885,7 +1903,7 @@ def _vllm_docker_args(mcfg: "RunpodModelConfig", served: str,
     if mcfg.quantization:
         args += ["--quantization", mcfg.quantization]
     args += _pod_lora_args(entry or {})
-    return " ".join(args)
+    return _with_extra_args(args, mcfg)
 
 
 #: Where a staged download lands inside the pod. Container disk by default —

codai/backends/runpod.py

@@ -54,6 +54,17 @@ class RunpodBackend(ModelBackend):
     # RunPod models never occupy local VRAM.
     uses_local_vram = False
 
+    #: Multipart (image) content is forwarded to the remote server VERBATIM.
+    #
+    # This is not a claim that every model behind this backend reads images — it is a
+    # statement about who gets to decide. The front flattens multipart content to the
+    # literal text "[image_url content]" unless the backend says otherwise, which for a
+    # proxy is the one unacceptable answer: the page is dropped and the model answers
+    # confidently about nothing. Forwarded, a text-only model's server returns a plain
+    # 400 and a VLM's server reads the image. (Found by olmOCR in "model" mode against
+    # a RunPod pod: every page arrived as that placeholder string.)
+    supports_vision = True
+
     def __init__(self, cfg=None):
         if cfg is None:
             from codai.config import RunpodConfig
@@ -231,8 +242,14 @@ class RunpodBackend(ModelBackend):
                 base, release = self._acquire_endpoint()
                 try:
                     payload = self._chat_payload(messages, max_tokens, temperature, top_p, stop, False)
-                    if response_format and response_format.get("type") == "json_object":
-                        payload["response_format"] = {"type": "json_object"}
+                    # Forward the structured-output request AS ASKED. This used to
+                    # keep only {"type":"json_object"}, so a json_schema — the strict
+                    # form every current client sends, Surya's guided decoding among
+                    # them — was dropped on the way out and the server answered free
+                    # text that the caller then failed to parse. The remote decides
+                    # what it supports; a proxy that edits the request decides for it.
+                    if response_format:
+                        payload["response_format"] = response_format
                     if tools:
                         payload["tools"] = tools
                     r = pod_http.post(base + "/chat/completions", json=payload,
@@ -265,6 +282,11 @@ class RunpodBackend(ModelBackend):
                 # Pod acquire may provision (blocking) — do it off the event loop.
                 base, release = await asyncio.to_thread(self._acquire_endpoint)
                 payload = self._chat_payload(messages, max_tokens, temperature, top_p, stop, True)
+                # A streamed request asks for structure as often as a buffered one,
+                # and dropped it here even for json_object: the parameter was
+                # accepted and then never looked at again.
+                if response_format:
+                    payload["response_format"] = response_format
                 if tools:
                     payload["tools"] = tools
                 try:

codai/backends/vllm.py

@@ -24,6 +24,10 @@ from codai.backends.base import ModelBackend
 class VllmBackend(ModelBackend):
     """Thin OpenAI proxy to a managed vLLM server."""
 
+    #: Multipart (image) content goes to the server as it arrived — see the same flag
+    #: on RunpodBackend for why a proxy must not flatten it.
+    supports_vision = True
+
     _inflight = 0
     _inflight_lock = threading.Lock()
 
@@ -187,8 +191,14 @@ class VllmBackend(ModelBackend):
         self._enter_request()
         try:
             payload = self._chat_payload(messages, max_tokens, temperature, top_p, stop, False)
-            if response_format and response_format.get("type") == "json_object":
-                payload["response_format"] = {"type": "json_object"}
+            # Forward the structured-output request AS ASKED. This used to
+            # keep only {"type":"json_object"}, so a json_schema — the strict
+            # form every current client sends, Surya's guided decoding among
+            # them — was dropped on the way out and the server answered free
+            # text that the caller then failed to parse. The remote decides
+            # what it supports; a proxy that edits the request decides for it.
+            if response_format:
+                payload["response_format"] = response_format
             if tools:
                 payload["tools"] = tools
             r = requests.post(self._base() + "/v1/chat/completions", json=payload, timeout=3600)
@@ -205,6 +215,9 @@ class VllmBackend(ModelBackend):
         self._enter_request()
         try:
             payload = self._chat_payload(messages, max_tokens, temperature, top_p, stop, True)
+            # Same for the streamed path, which ignored it entirely.
+            if response_format:
+                payload["response_format"] = response_format
             if tools:
                 payload["tools"] = tools
             async for chunk in self._stream(self._base() + "/v1/chat/completions", payload,

codai/ocr/served_model.py

@@ -45,25 +45,38 @@ def _entry_backend(model_id: str):
     """
     from codai.models import manager as mm
 
+    # A model pinned to another host (service_url) already IS an endpoint. Asking
+    # first costs nothing, and a failure to even ask must not shadow the next
+    # candidate — hence the claim and the construction are separate steps.
+    claimed_remote = False
     try:
         from codai.backends import remote_openai
-        if remote_openai.remote_should_handle(model_id):
-            backend = remote_openai.RemoteOpenAIBackend()
+        claimed_remote = bool(remote_openai.remote_should_handle(model_id))
+    except Exception:
+        claimed_remote = False
+    if claimed_remote:
+        try:
+            from codai.backends.remote_openai import RemoteOpenAIBackend
+            backend = RemoteOpenAIBackend()
             backend.load_model(model_id)
             return backend
-    except OcrError:
-        raise
-    except Exception as exc:
-        raise OcrError(f"model {model_id!r}: remote endpoint unusable: {exc}", status=503)
+        except Exception as exc:
+            raise OcrError(f"model {model_id!r}: remote endpoint unusable: {exc}",
+                           status=503)
 
     try:
-        if mm.runpod_should_handle(model_id):
+        claimed_runpod = bool(mm.runpod_should_handle(model_id))
+    except Exception:
+        claimed_runpod = False
+    if claimed_runpod:
+        try:
             from codai.backends.runpod import RunpodBackend
             backend = RunpodBackend(mm.get_active_runpod_config())
             backend.load_model(model_id)
             return backend
-    except Exception as exc:
-        raise OcrError(f"model {model_id!r}: RunPod endpoint unusable: {exc}", status=503)
+        except Exception as exc:
+            raise OcrError(f"model {model_id!r}: RunPod endpoint unusable: {exc}",
+                           status=503)
 
     if mm.resolve_engine_backend(model_id) == "vllm":
         try:

tests/test_ocr_served_model.py

@@ -302,3 +302,38 @@ def test_the_incident_is_recorded_where_the_mode_is_defined():
     """-GPU-less is the case that found this; the next reader deserves the why."""
     assert "no GPU" in SERVED_SRC
     assert "NO_VLLM_VENV" in SERVED_SRC
+
+
+# ------------------------------------------------------ images through a proxy
+def test_a_proxy_backend_forwards_images_instead_of_flattening_them():
+    """The front turns multipart content into the literal text "[image_url content]"
+    unless the backend claims vision, and only the llama.cpp backend ever did — so
+    every page sent to a VLM served by RunPod or by vLLM arrived as that placeholder
+    and the model answered, fluently, about nothing.
+
+    A proxy is not the right place to make that judgement: forwarded, a text-only
+    model's server answers 400 and a VLM's server reads the page. This is the path a
+    client takes when it drives olmOCR itself."""
+    from codai.backends.runpod import RunpodBackend
+    from codai.backends.remote_openai import RemoteOpenAIBackend
+    from codai.backends.vllm import VllmBackend
+    for cls in (RunpodBackend, RemoteOpenAIBackend, VllmBackend):
+        assert getattr(cls, "supports_vision", False) is True, cls.__name__
+
+
+def test_the_front_asks_the_backend_and_nothing_else():
+    """If this gate moves, the flag above stops having any effect."""
+    text = (ROOT / "codai/api/text.py").read_text()
+    assert "'supports_vision', False" in text
+
+
+# ------------------------------------------- structured output through a proxy
+def test_a_proxy_forwards_the_schema_it_was_given():
+    """It used to keep only {"type":"json_object"} and drop everything else, so a
+    json_schema — the strict form current clients send, Surya's guided decoding among
+    them — never reached the server, which then answered free text the caller could
+    not parse. The streamed path dropped it outright, json_object included."""
+    for mod in ("codai/backends/runpod.py", "codai/backends/vllm.py"):
+        src = (ROOT / mod).read_text()
+        assert 'response_format.get("type") == "json_object"' not in src, mod
+        assert src.count('payload["response_format"] = response_format') == 2, mod

tests/test_runpod_pod_extra_args.py

+"""`extra_args` on a model's runpod block has to reach the pod's command line.
+
+It has been accepted there — and whitelisted per engine, so it survives a save —
+since pods shipped, and it went nowhere: parsed, stored, never printed. Setting it
+LOOKED like tuning and did nothing, which is worse than refusing it would have been,
+because the pod boots, serves, and quietly ignores the decision.
+
+The two Digesta OCR VLMs are what noticed. A page-reading model wants prefix caching
+(its prompt is identical on every page of every document) and the Qwen-VL pixel bounds
+that Surya's own launcher sets, and on a rented pod neither could be expressed: the
+only way in was `docker_args`, which replaces the entire generated command and so
+means re-deriving --model, --served-model-name, --max-model-len and the api key by
+hand, to be kept in step forever after.
+"""
+
+import pathlib
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.api import runpod_worker as rw                  # noqa: E402
+
+
+def _mcfg(**kw):
+    cfg = rw.RunpodModelConfig()
+    for k, v in kw.items():
+        setattr(cfg, k, v)
+    return cfg
+
+
+def test_a_vllm_pod_gets_the_operators_flags():
+    args = rw._vllm_docker_args(
+        _mcfg(ctx=18432, extra_args="--enable-prefix-caching"),
+        "datalab-to/surya-ocr-2", {})
+    assert "--model datalab-to/surya-ocr-2" in args
+    assert "--max-model-len 18432" in args
+    assert args.endswith("--enable-prefix-caching")
+
+
+def test_the_extra_flags_come_last_so_they_win():
+    """On any command line the later flag wins; a tuning knob that landed first
+    would be overridden by our own default and look like it had done nothing."""
+    args = rw._vllm_docker_args(
+        _mcfg(ctx=4096, extra_args="--max-model-len 9000"),
+        "org/model", {})
+    assert args.index("--max-model-len 4096") < args.index("--max-model-len 9000")
+
+
+def test_quoted_json_survives():
+    """--mm-processor-kwargs takes a JSON object; splitting and rejoining it on
+    whitespace would hand vLLM three broken fragments."""
+    kw = '--mm-processor-kwargs {"min_pixels":3136,"max_pixels":6291456}'
+    args = rw._vllm_docker_args(_mcfg(extra_args=kw), "org/model", {})
+    assert kw in args
+
+
+def test_a_llamacpp_pod_gets_them_too():
+    args = rw._llamacpp_docker_args(
+        _mcfg(ctx=8192, extra_args="--flash-attn", hf_gguf="org/repo:Q4_K_M"),
+        "org/repo:Q4_K_M", {})
+    assert args.endswith("--flash-attn")
+
+
+def test_nothing_configured_changes_nothing():
+    """The flag is additive: an install that never set it must get the exact command
+    line it got before."""
+    plain = rw._vllm_docker_args(_mcfg(ctx=1024), "org/model", {})
+    assert plain == plain.strip()
+    assert "  " not in plain
+
+
+def test_the_api_key_still_lands_on_a_tuned_pod():
+    """pod_plan appends --api-key only when it is not already there; a trailing
+    extra_args must not confuse that check into leaving the pod WIDE OPEN."""
+    plan = rw.pod_plan(_mcfg(ctx=1024, extra_args="--enable-prefix-caching"),
+                       "org/model", model_key="org/model", api_key="secrettoken",
+                       entry={})
+    assert "--api-key secrettoken" in plan["args"]
+    assert "--enable-prefix-caching" in plan["args"]
+
+
+def test_docker_args_still_replaces_everything():
+    """The full-override escape hatch is unchanged — and because it replaces the
+    line, extra_args is not appended to it either."""
+    plan = rw.pod_plan(_mcfg(docker_args="--model x --port 8000",
+                             extra_args="--enable-prefix-caching"),
+                       "org/model", model_key="org/model", entry={})
+    assert plan["args"].startswith("--model x --port 8000")
+    assert "--enable-prefix-caching" not in plan["args"]