a pod that terminates TLS could not reach itself (v0.4.5)

0.4.3 got placement working on a pod: it resolved the model and minted an
attach lease, which it had never managed before. Then surya said:

    SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1/ocr/attach/<lease>/v1
    is not reachable at /health

...about an address the pod was answering perfectly well. A pod on the
direct path serves HTTPS on its ONLY port, and an OCR engine there is a
SUBPROCESS: it attaches by URL and goes through the socket like any other
client. self_base() handed it `http://`, so it probed a plain request at a
TLS listener.

Fixed with verification ON end to end, not off:

* the leaf certificate now carries DNS:coderai-pod, DNS:localhost,
  IP:127.0.0.1 and IP:::1. It had one SAN, `coderai-pod`, that the comment
  said was "never checked" -- true for the orchestrator, which reaches a
  pod by a bare IP with the hostname check off. A pod reaching ITSELF has
  no such excuse, so it uses a name the certificate actually holds.
* boot.sh maps 127.0.0.1 coderai-pod and exports
  CODERAI_SELF_BASE=https://coderai-pod:8000; the plain-HTTP case (behind
  the RunPod proxy) exports the loopback http base explicitly instead of
  relying on a default.
* the CA's PUBLIC certificate now travels with the leaf, appended to
  CERTIFI'S ROOTS into one bundle. That order is not cosmetic: a bundle
  that lost the public roots would break every HuggingFace download on the
  pod, which is worse than the bug. Exported only once built and non-empty,
  and it says so plainly when it cannot be.

This covers chandra and olmOCR too -- in served mode they attach through
the same gateway URL from the main venv, so one trust store serves all
three.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.4"
+__version__ = "0.4.5"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/pod_tls.py

@@ -35,6 +35,17 @@ _lock = threading.Lock()
 #: Environment variables the pod reads (boot.sh). PEM text, not paths.
 ENV_CERT = "CODERAI_TLS_CERT"
 ENV_KEY = "CODERAI_TLS_KEY"
+#: The CA's PUBLIC certificate, sent so the pod can verify ITSELF.
+#:
+#: A pod that terminates TLS has to talk to its own API over loopback -- an OCR
+#: engine runs as a subprocess and attaches to the gateway by URL, so it goes
+#: through the socket like any other client. Without the CA it cannot verify the
+#: certificate the pod is serving, and surya reported the pod's own address as
+#: "not reachable at /health" while the pod was answering perfectly well.
+#:
+#: Public by definition: this is the certificate every user verifies releases
+#: against. The private key never leaves this machine.
+ENV_CA = "CODERAI_TLS_CA"
 
 
 def tls_dir() -> str:
@@ -92,7 +103,13 @@ def issue_pod_cert(pod_name: str, days: int = 30) -> dict:
         key, csr, crt = (os.path.join(tmp, n) for n in ("pod.key", "pod.csr", "pod.pem"))
         ext = os.path.join(tmp, "ext.cnf")
         with open(ext, "w") as f:
-            f.write("subjectAltName=DNS:coderai-pod\n"
+            # `coderai-pod` resolves to loopback inside the pod (boot.sh adds
+            # it to /etc/hosts), so the pod verifies ITS OWN certificate by a
+            # name the certificate actually carries -- no disabled hostname
+            # check anywhere on that path. The IP SANs are belt and braces for
+            # anything that reaches loopback numerically.
+            f.write("subjectAltName=DNS:coderai-pod,DNS:localhost,"
+                    "IP:127.0.0.1,IP:0:0:0:0:0:0:0:1\n"
                     "extendedKeyUsage=serverAuth\n"
                     "basicConstraints=CA:FALSE\n")
         _run(["req", "-newkey", "ec", "-pkeyopt", "ec_paramgen_curve:prime256v1",
@@ -109,4 +126,15 @@ def issue_pod_cert(pod_name: str, days: int = 30) -> dict:
 def pod_env(pod_name: str) -> dict:
     """The environment that turns TLS on in a coderai pod."""
     leaf = issue_pod_cert(pod_name)
-    return {ENV_CERT: leaf["cert"], ENV_KEY: leaf["key"]}
+    env = {ENV_CERT: leaf["cert"], ENV_KEY: leaf["key"]}
+    try:
+        with open(ensure_ca()) as f:
+            env[ENV_CA] = f.read()
+    except Exception as exc:
+        # Not fatal: without it the pod still serves TLS outward, it just
+        # cannot verify itself on loopback. Say so rather than let an OCR
+        # engine discover it as "/health is not reachable".
+        print(f"[runpod] could not read the pod TLS CA to send it ({exc}); "
+              f"an OCR pipeline ON this pod will not be able to verify the "
+              f"pod's own address", flush=True)
+    return env

packaging/runpod/boot.sh

@@ -80,6 +80,49 @@ if [ -n "$CODERAI_TLS_CERT" ] && [ -n "$CODERAI_TLS_KEY" ]; then
     unset CODERAI_TLS_KEY
     TLS_ARGS="--ssl-certfile /run/coderai-tls/cert.pem --ssl-keyfile /run/coderai-tls/key.pem"
     phase "tls: serving https with the certificate the renting coderai sent"
+
+    # This pod now speaks TLS on its ONLY port, so anything here that talks to
+    # the pod's own API goes through TLS too -- an OCR engine runs as a
+    # subprocess and attaches to the gateway by URL, over the socket like any
+    # other client. Handing it http://127.0.0.1:8000 made it probe a plain
+    # request at a TLS listener and report the pod's own address as "not
+    # reachable at /health" while the pod was answering fine.
+    #
+    # So: a name the certificate carries, resolving to loopback, and a trust
+    # store that holds BOTH the public roots and our CA. Verification stays on
+    # end to end -- no skipped hostname check, no disabled verify.
+    grep -q ' coderai-pod$' /etc/hosts 2>/dev/null \
+        || echo "127.0.0.1 coderai-pod" >> /etc/hosts 2>/dev/null \
+        || phase "tls: could not add coderai-pod to /etc/hosts"
+    if [ -n "${CODERAI_TLS_CA:-}" ]; then
+        printf '%s\n' "$CODERAI_TLS_CA" > /run/coderai-tls/ca.pem
+        # certifi FIRST, ours appended: a bundle that lost the public roots
+        # would break every HuggingFace download on this pod, which is a far
+        # worse failure than the one being fixed. Only export it once it is
+        # built and non-empty.
+        BASE_CA="$(python -c 'import certifi;print(certifi.where())' 2>/dev/null || true)"
+        if [ -n "$BASE_CA" ] && [ -s "$BASE_CA" ]; then
+            cat "$BASE_CA" /run/coderai-tls/ca.pem > /run/coderai-tls/bundle.pem 2>/dev/null
+        else
+            cat /etc/ssl/certs/ca-certificates.crt /run/coderai-tls/ca.pem \
+                > /run/coderai-tls/bundle.pem 2>/dev/null || true
+        fi
+        if [ -s /run/coderai-tls/bundle.pem ] \
+           && grep -qc "BEGIN CERTIFICATE" /run/coderai-tls/bundle.pem >/dev/null 2>&1; then
+            export SSL_CERT_FILE=/run/coderai-tls/bundle.pem
+            export REQUESTS_CA_BUNDLE=/run/coderai-tls/bundle.pem
+            export CURL_CA_BUNDLE=/run/coderai-tls/bundle.pem
+            export CODERAI_SELF_BASE="https://coderai-pod:8000"
+            phase "tls: loopback verified via coderai-pod with the public roots plus our CA"
+        else
+            phase "tls: trust bundle could not be built — an OCR pipeline on this pod cannot verify loopback"
+        fi
+    else
+        phase "tls: no CA was sent — an OCR pipeline on this pod cannot verify loopback"
+    fi
+else
+    # Plain HTTP (the RunPod proxy is in front): loopback is loopback.
+    export CODERAI_SELF_BASE="http://127.0.0.1:8000"
 fi
 
 # A ray WORKER instead of a server: the vllm image joins a multi-node vLLM

tests/test_pod_loopback_tls.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""A pod that terminates TLS must be able to reach ITSELF.
+
+A pod on the direct path serves HTTPS on its only port. An OCR engine running
+there is a SUBPROCESS: it attaches to the gateway by URL and goes through the
+socket like any other client. It was handed `http://127.0.0.1:8000`, probed a
+plain request at a TLS listener, and reported
+
+    SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1/ocr/attach/<lease>/v1
+    is not reachable at /health. Start the server or unset the variable.
+
+while the pod was answering that very address perfectly well over TLS.
+
+The fix keeps verification ON end to end, which is the part worth pinning:
+a name the certificate actually carries, resolved to loopback, and a trust
+store holding the public roots AND our CA. No skipped hostname check and no
+disabled verify anywhere on that path -- the orchestrator needs those to
+reach a pod by a bare IP, but a pod reaching itself has no such excuse.
+"""
+
+import os
+import pathlib
+import re
+import subprocess
+import tempfile
+
+import pytest
+
+from codai.api import pod_tls
+from codai.ocr import gateway
+
+
+BOOT = pathlib.Path("packaging/runpod/boot.sh")
+
+
+# ------------------------------------------------------- the certificate #
+def test_the_cert_names_loopback_so_nothing_has_to_skip_the_check():
+    leaf = pod_tls.issue_pod_cert("some-pod")
+    with tempfile.NamedTemporaryFile("w", suffix=".pem", delete=False) as f:
+        f.write(leaf["cert"])
+        path = f.name
+    try:
+        text = subprocess.run(["openssl", "x509", "-in", path, "-noout", "-text"],
+                              capture_output=True, text=True).stdout
+    finally:
+        os.unlink(path)
+    sans = " ".join(l for l in text.splitlines() if "DNS:" in l or "IP Address" in l)
+    assert "DNS:coderai-pod" in sans
+    assert "DNS:localhost" in sans
+    assert "IP Address:127.0.0.1" in sans
+
+
+def test_the_ca_travels_with_the_leaf():
+    """Without it the pod holds a certificate it cannot verify."""
+    env = pod_tls.pod_env("some-pod")
+    assert pod_tls.ENV_CA in env
+    assert env[pod_tls.ENV_CA].startswith("-----BEGIN")
+
+
+def test_the_private_key_is_still_the_only_secret_sent():
+    env = pod_tls.pod_env("some-pod")
+    assert set(env) == {pod_tls.ENV_CERT, pod_tls.ENV_KEY, pod_tls.ENV_CA}
+    # The CA's KEY must never be in there.
+    assert "ca.key" not in " ".join(env.values())
+
+
+# -------------------------------------------------------------- boot.sh #
+def test_boot_sets_a_self_base_for_both_schemes():
+    src = BOOT.read_text()
+    assert 'export CODERAI_SELF_BASE="https://coderai-pod:8000"' in src
+    assert 'export CODERAI_SELF_BASE="http://127.0.0.1:8000"' in src
+
+
+def test_boot_maps_the_cert_name_to_loopback():
+    assert "127.0.0.1 coderai-pod" in BOOT.read_text()
+
+
+def test_boot_keeps_the_public_roots_in_the_bundle():
+    """A bundle that lost them would break every model download on the pod --
+    a worse failure than the one being fixed."""
+    src = BOOT.read_text()
+    assert "certifi" in src
+    # ours is APPENDED to the public roots, not the other way round
+    assert re.search(r'cat "\$BASE_CA" /run/coderai-tls/ca\.pem', src)
+
+
+def test_boot_only_exports_a_bundle_it_actually_built():
+    src = BOOT.read_text()
+    block = src.split("BASE_CA=")[1].split("else\n    # Plain HTTP")[0]
+    assert "-s /run/coderai-tls/bundle.pem" in block
+    assert "export SSL_CERT_FILE" in block
+    assert "export REQUESTS_CA_BUNDLE" in block
+
+
+def test_boot_says_so_when_it_cannot():
+    """Silence here is how the last two defects stayed hidden for days."""
+    src = BOOT.read_text()
+    assert src.count("cannot verify loopback") >= 2
+
+
+def test_boot_is_valid_shell():
+    assert subprocess.run(["bash", "-n", str(BOOT)]).returncode == 0
+
+
+# ------------------------------------------------------------- the URL #
+def test_self_base_follows_the_environment(monkeypatch):
+    monkeypatch.setenv("CODERAI_SELF_BASE", "https://coderai-pod:8000")
+    assert gateway.self_base() == "https://coderai-pod:8000"
+
+
+def test_self_base_falls_back_to_loopback_http(monkeypatch):
+    monkeypatch.delenv("CODERAI_SELF_BASE", raising=False)
+    monkeypatch.delenv("CODERAI_PORT", raising=False)
+    assert gateway.self_base() == "http://127.0.0.1:8000"
+
+
+def test_the_engine_url_is_built_from_it(monkeypatch):
+    monkeypatch.setenv("CODERAI_SELF_BASE", "https://coderai-pod:8000")
+    url = gateway.engine_url("LEASE")
+    assert url == "https://coderai-pod:8000/v1/ocr/attach/LEASE/v1"
+    # surya derives /health by stripping the trailing /v1 -- see gateway.PREFIX
+    assert url.endswith("/v1")