front: forward-auth peers from env, and say why one was refused (v0.2.88)

The Digesta host reported SSO minting a session from inside the container
(:8776) but refusing from the host through the published port (:8777), with the
container's address in trusted_proxies and the front's own access log showing
that very address as the client. Both that and "enabled: true in config.json
did not switch the feature on" are one cause.

The peer _fa_trusted checks is not the socket peer: uvicorn installs
ProxyHeadersMiddleware by default with trusted_hosts="127.0.0.1", so a hop
arriving over loopback from the internal nginx has scope["client"] rewritten
from X-Forwarded-For before any of our middleware runs. On a container network
that is the address the proxy was SNATed to. So the log and the check agreed and
the peer really was 10.89.0.178 -- but the list it was compared against was the
DEFAULT ["127.0.0.1", "::1"], because their config.json sat at a path the app
does not read (the $CODERAI_CONFIG_DIR/coderai trap), which is equally why the
file's `enabled` did nothing and the env var was needed. Loopback is in that
default; the container address is not.

The loader is fine and `enabled` from the file does suffice -- verified, not
assumed. What was missing was a way to supply the peers the same way the switch
and the secret can already be supplied, so CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_
PROXIES now overrides the list; blank or unset falls through to config.json.
Enabling the feature from a unit file and then having it reject every request
from your own proxy was a gap in the contract.

And a refusal now says why, once per peer and reason per 60s, naming the list it
was checked against and never the secret. A refusal is a 302 to the login form,
indistinguishable from "no session yet"; the report had to ask for exactly this
instrumentation, so it is permanent now.

tests/test_forward_auth_peer_trust.py reproduces the peer resolution for all
three hops, so the table in the doc is executable rather than reasoned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.87"
+__version__ = "0.2.88"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/frontproxy/ui_pages.py

@@ -29,6 +29,12 @@ from fastapi.staticfiles import StaticFiles
 
 
 _FA_WARNED = False
+# Rejected forward-auth attempts already reported, as {(peer, reason): monotonic}.
+# A refusal is a 302 to the login form and nothing else, which is indistinguishable
+# from "no session yet" -- so the one thing an operator needs is WHY. Bounded and
+# time-windowed: a misconfigured proxy must not be able to flood the log.
+_FA_REJECTS: dict = {}
+_FA_REJECT_WINDOW = 60.0
 
 
 def register_ui_pages(app: FastAPI, config_dir) -> bool:
@@ -95,17 +101,59 @@ def register_ui_pages(app: FastAPI, config_dir) -> bool:
             return None
         return cfg, secret
 
+    def _fa_peer_allowlist(cfg) -> list:
+        """The trusted-proxy list, with an env override.
+
+        CODERAI_ADMIN_FORWARD_AUTH could already be turned on from a unit file and
+        the secret supplied the same way, but the peer list could only come from
+        config.json -- so a container whose config is mounted read-only (or, as on
+        the Digesta host, whose config was written to a path the app does not read)
+        could enable the feature by env and then reject every request from its own
+        proxy, because the list in effect was still the default
+        ["127.0.0.1", "::1"]. Comma- or space-separated.
+        """
+        import os
+        raw = os.environ.get("CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_PROXIES", "")
+        if raw.strip():
+            return [t for t in (x.strip() for x in raw.replace(",", " ").split()) if t]
+        return [str(t).strip() for t in (cfg.trusted_proxies or []) if str(t).strip()]
+
+    def _fa_reject(peer: str, reason: str) -> bool:
+        """Say once, per peer and reason, why a forward-auth attempt was refused."""
+        import time
+        now = time.monotonic()
+        key = (peer, reason)
+        last = _FA_REJECTS.get(key)
+        if last is not None and (now - last) < _FA_REJECT_WINDOW:
+            return False
+        if len(_FA_REJECTS) > 64:
+            _FA_REJECTS.clear()
+        _FA_REJECTS[key] = now
+        print(f"[front] admin forward-auth refused: {reason} "
+              f"(peer {peer or 'unknown'!r})", flush=True)
+        return False
+
     def _fa_trusted(request: Request, cfg, secret: str) -> bool:
-        """Both checks, or nothing: the immediate peer must be a trusted proxy
-        AND the shared secret must match. Either one alone is forgeable."""
+        """Both checks, or nothing: the peer must be a trusted proxy AND the shared
+        secret must match. Either one alone is forgeable.
+
+        The peer is ``request.client.host``, which uvicorn's ProxyHeadersMiddleware
+        has already rewritten from X-Forwarded-For -- so it is the address the proxy
+        reports, not the socket peer, and it is what the access log shows. That
+        matters for what belongs in trusted_proxies: on a container network the
+        value is the address the proxy is SNATed to, not 127.0.0.1, even though the
+        TCP connection into the front is over loopback.
+        """
         import hmac
         peer = (request.client.host if request.client else "") or ""
-        trusted = [str(t).strip() for t in (cfg.trusted_proxies or []) if str(t).strip()]
+        trusted = _fa_peer_allowlist(cfg)
         if peer not in trusted:
-            return False
+            return _fa_reject(peer, f"peer is not in trusted_proxies {trusted}")
         sent = request.headers.get(cfg.shared_secret_header.lower(), "") or ""
-        if not sent or not hmac.compare_digest(sent, secret):
-            return False
+        if not sent:
+            return _fa_reject(peer, f"no {cfg.shared_secret_header} header")
+        if not hmac.compare_digest(sent, secret):
+            return _fa_reject(peer, f"{cfg.shared_secret_header} does not match")
         return True
 
     def _fa_identity(request: Request):

docs/admin-forward-auth.md

@@ -146,6 +146,74 @@ request: engine#0 crash-looping GPU-less despite `backend.type=cpu` +
 enforced). This forward-auth feature is independent of both.
 
 
+---
+
+## Prod activation status — 08/10/2026 (code 0.2.87): one bug left
+
+Forward-auth has shipped (`admin.forward_auth` in config + `CODERAI_ADMIN_FORWARD_AUTH` /
+`CODERAI_ADMIN_FORWARD_AUTH_SECRET` env), `/v1/*` now enforces the bearer (401), and
+engine#0 starts clean GPU-less. The admin-GUI forward-auth is **almost working on the
+Digesta prod host**, with **one reproducible bug left, only on the rootless published-port
+path**.
+
+### Prod setup (rootless podman + quadlet)
+- Image `ghcr.io/nextime/coderai:latest`, in-image code upgraded to **0.2.87** with the
+  baked-in upgrader (`CODERAI_UPGRADE_REF=master` → fetched 0.2.87, `podman commit` back).
+  The upgrader also accepts **`CODERAI_UPGRADE_REF=production`** to track the `production`
+  branch — we used `master` only because `production` was lagging (0.2.65 at last check).
+  Say if prod should track `production` instead.
+- `admin.forward_auth`: `enabled`, `shared_secret` set, `admin_group=admins`,
+  `trusted_proxies=["127.0.0.1","::1","10.89.0.178"]`, `keep_local_login=true`.
+  **Note:** config-only `enabled:true` did **not** turn the feature on — setting the env
+  `CODERAI_ADMIN_FORWARD_AUTH=1` (+ `_SECRET`) in the quadlet was required. Worth checking
+  whether `enabled` from `config.json` alone is meant to suffice.
+- The container's own IP on `digesta-net` is pinned to `10.89.0.178`
+  (`Network=digesta-net:ip=10.89.0.178`), because rootless netavark SNATs host→published-port
+  traffic to the container's own eth0 IP.
+- Topology: host nginx → `127.0.0.1:8777` (published) → container internal nginx `:8776` →
+  front (uvicorn) `:18776`.
+
+### What works (0.2.87, tested from inside the container)
+- **Direct to the front** `127.0.0.1:18776/admin`, with `X-Coderai-Proxy-Secret` +
+  `X-Forwarded-User` + `X-Forwarded-Groups: admins` → **`200` + `Set-Cookie: session=…`**.
+  The front forward-auth logic is correct.
+- **Through the internal nginx** `127.0.0.1:8776/admin` (same headers, `X-Forwarded-Prefix:
+  /coderai`) → **also `200` + `Set-Cookie`**. So the internal nginx forwards the custom
+  headers to the front correctly (this hop was broken on 0.2.85/0.2.86; 0.2.87 fixed it).
+
+### The one bug
+- **Host → published port** `127.0.0.1:8777/admin` (same headers) → **`302` → `/coderai/login`,
+  no `Set-Cookie`**. SSO is not minted. This is the exact path the real host nginx uses, so
+  SSO cannot be switched on until it works here.
+
+### Why it's puzzling (for the maintainer)
+- The front's uvicorn **access log shows the client for host→8777 as `10.89.0.178`** — which
+  **is** in `trusted_proxies`.
+- The shared secret matches (sha256-verified; and the in-container `:8776` hop mints with the
+  same secret through the same internal nginx).
+- The internal nginx forwards the headers (proven by the minting `:8776` hop).
+- So the **only** difference between the minting `:8776` hop and the failing `:8777` hop is the
+  **source IP into the internal nginx**: `127.0.0.1` (in-container) vs `10.89.0.178` (host via
+  rootlessport). Both are in `trusted_proxies`, yet only the loopback one mints.
+
+### Likely area to check
+- How `_fa_trusted` resolves the **peer** for the published-port path vs what uvicorn logs as
+  the client. If the proxy-headers middleware (`--proxy-headers` / `--forwarded-allow-ips`)
+  computes `request.client.host` from `X-Forwarded-For` for the access log, but `_fa_trusted`
+  reads a **different** value (the raw socket peer, or a different XFF element), the two can
+  disagree — the log says `10.89.0.178` while the trust check sees something not in
+  `trusted_proxies`.
+- Less likely: the secret header dropped only on the rootlessport path (the `:8776` hop
+  forwards it fine).
+- Fastest confirmation: log, inside `_fa_trusted`, the actual `peer` value and whether the
+  secret header is present, for one host→8777 request.
+
+### Ops state
+GUI still behind nginx **basic-auth** on prod (not swapped to `auth_request`): swapping before
+host→8777 mints would lock everyone out (no coderai passwords; SSO would be the only way in).
+The Digesta side is ready — `GET /api/authz/coderai` returns `204` + `X-User` for a Digesta
+admin, `401` otherwise, for nginx `auth_request`.
+
 ---
 
 ## As shipped (0.2.85)
@@ -204,3 +272,97 @@ bearer at all** — `GET /v1/models` answered 200 with no header on 0.2.83. Fixe
 in 0.2.84; the front now refuses `/v1` without credentials instead of falling
 open. That was the second item in the "not part of this request" note, and it
 was real.
+
+---
+
+## Answer to the 08/10/2026 report: found, and it is one cause, not two
+
+Both symptoms — `enabled: true` in `config.json` not switching the feature on, and
+host→`:8777` refusing to mint while in-container→`:8776` minted — are the **same**
+problem. Nothing is wrong with the trust logic.
+
+### 1. The peer checked against `trusted_proxies` is not the socket peer
+
+`_fa_trusted` reads `request.client.host`. uvicorn installs
+`ProxyHeadersMiddleware` **by default**, with `trusted_hosts="127.0.0.1"`. The
+internal nginx connects to the front over loopback, so that middleware trusts the
+hop and **rewrites `scope["client"]` from `X-Forwarded-For`** before any CoderAI
+middleware runs. Reproduced exactly (`tests/test_forward_auth_peer_trust.py`):
+
+| hop | nginx sends `X-Forwarded-For` | peer the app sees |
+|---|---|---|
+| in-container → `:8776` | `127.0.0.1` | `127.0.0.1` |
+| host → `:8777` | `10.89.0.178` | `10.89.0.178` |
+| real nginx → `:8777` | `203.0.113.5, 10.89.0.178` | `10.89.0.178` |
+
+So the access log and the trust check **do** agree — that was a correct
+observation, and it is why the rejection looked impossible. The peer really is
+`10.89.0.178`.
+
+### 2. …but the list it was compared against was the default
+
+`admin.forward_auth.trusted_proxies` defaults to `["127.0.0.1", "::1"]`. The
+config loader is fine — a `config.json` carrying `admin.forward_auth` round-trips
+correctly, and `enabled: true` from the file alone **does** suffice (verified; there
+is no second bug there). Which means: **the front was not reading your
+`config.json` at all.** That is exactly what "config-only `enabled: true` did not
+work, the env var was required" tells us, and it is the known path trap:
+
+> `coderai-entrypoint` creates `$CODERAI_CONFIG_DIR/coderai` and symlinks
+> `~/.coderai` to it. The app resolves config from the HOME-style path, so a
+> `config.json` at **`$CODERAI_CONFIG_DIR/config.json`** is silently ignored and a
+> default is written alongside it.
+
+With the feature forced on by env and the secret supplied by env, everything came
+from env — except `trusted_proxies`, which stayed at the default. Loopback is in
+that default; `10.89.0.178` is not. Hence: `:8776` mints, `:8777` does not.
+
+### The fix on your side
+
+Move the file to **`$CODERAI_CONFIG_DIR/coderai/config.json`** and your whole
+`admin.forward_auth` block takes effect — then you can drop
+`CODERAI_ADMIN_FORWARD_AUTH` / `_SECRET` from the quadlet entirely if you like.
+Check which file is live before anything else:
+
+```bash
+podman exec coderai sh -lc 'ls -l ~/.coderai/config.json; \
+  python3 -c "import json;print(json.load(open(\"$HOME/.coderai/config.json\")).get(\"admin\"))"'
+```
+
+### The fix on ours (0.2.88)
+
+- **`CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_PROXIES`** (comma- or space-separated) now
+  overrides the list, so a unit file that can enable the feature and supply its
+  secret can also supply its peers. Enabling by env and then rejecting every
+  request from your own proxy was a gap in the contract, not a misuse. An unset or
+  blank value falls through to `config.json`.
+- **A refused attempt now says why**, once per peer and reason per 60s:
+
+  ```
+  [front] admin forward-auth refused: peer is not in trusted_proxies
+          ['127.0.0.1', '::1'] (peer '10.89.0.178')
+  [front] admin forward-auth refused: no X-Coderai-Proxy-Secret header (peer '10.89.0.178')
+  [front] admin forward-auth refused: X-Coderai-Proxy-Secret does not match (peer '10.89.0.178')
+  ```
+
+  The log never prints the secret or the value sent. This is the instrumentation
+  the report asked for, made permanent: a refusal is a 302 to the login form,
+  which is indistinguishable from "no session yet".
+- `_fa_trusted`'s docstring no longer says "immediate peer", because it is not —
+  believing it was is what made this look impossible.
+
+### What goes in `trusted_proxies`
+
+On a container network, the address the proxy is **SNATed to** — for you
+`10.89.0.178` — even though the TCP connection into the front is over loopback.
+Keeping `127.0.0.1` and `::1` as well is right: that is the in-container hop you
+test with.
+
+### Two answers to your questions
+
+- **Track `production`.** It was lagging when you looked; it now carries 0.2.88 and
+  will stay current. Use `CODERAI_UPGRADE_REF=production`.
+- **Switching nginx from basic-auth to `auth_request` is safe once host→8777
+  mints**, and you no longer have to choose: with `keep_local_login: true` the
+  password form stays as a loopback fallback, so a broken SSO cannot lock you out.
+  Verify the mint through the real path first, then swap.

tests/test_admin_forward_auth.py

@@ -68,11 +68,12 @@ def test_the_config_round_trips(tmp_path):
 # --------------------------------------------------------- the trust contract
 def test_both_checks_are_required_not_either():
     blk = UI.split("def _fa_trusted")[1].split("def _fa_identity")[0]
-    assert "peer not in trusted" in blk and "return False" in blk
+    assert "peer not in trusted" in blk
     assert "compare_digest" in blk, "the secret must not be compared with =="
-    # peer check and secret check are separate early returns, so neither alone
-    # can authenticate.
-    assert blk.count("return False") >= 2
+    # The peer check and the secret check are separate early returns, so neither
+    # alone can authenticate. Each refusal goes through _fa_reject, which returns
+    # False after saying why.
+    assert blk.count("_fa_reject(") >= 2
 
 
 def test_the_header_is_ignored_entirely_when_untrusted():

tests/test_forward_auth_peer_trust.py

+"""Which address forward-auth actually compares against trusted_proxies.
+
+Reported from the Digesta prod host on 2026-10-08: with forward-auth enabled, a
+request carrying the right secret and user headers minted a session when it came
+from inside the container (`127.0.0.1:8776`, the internal nginx) but 302'd to the
+login form when it came from the host through the published port (`:8777`) --
+the exact path the real nginx uses. The operator had 10.89.0.178 in
+trusted_proxies and the front's own access log showed that very address as the
+client, so the rejection looked impossible.
+
+Two things were true at once:
+
+1. The peer a trusted-header check sees is NOT the socket peer. uvicorn installs
+   ProxyHeadersMiddleware by default with trusted_hosts="127.0.0.1", so for a
+   connection arriving over loopback from the internal nginx it REWRITES
+   scope["client"] from X-Forwarded-For. On a container network that is the
+   address the proxy was SNATed to, not 127.0.0.1. The first class of tests here
+   pins that resolution down, because everything else follows from it.
+
+2. Their config.json was at a path the app does not read (it resolves config from
+   the HOME-style `$CODERAI_CONFIG_DIR/coderai`), which is also why `enabled:
+   true` alone did not switch the feature on and the env var was needed. So the
+   list actually in effect was the DEFAULT ["127.0.0.1", "::1"] -- loopback in,
+   container address out. One cause, both symptoms.
+
+Hence the env override: the feature could already be turned on, and its secret
+supplied, from a unit file, but the peer list could only come from the file. You
+could enable it by env and have it reject every request from your own proxy.
+"""
+import pathlib
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+UI = (ROOT / "codai/frontproxy/ui_pages.py").read_text()
+
+
+def _peer_seen_by_the_app(xff: str, socket_peer: str = "127.0.0.1",
+                          trusted_hosts: str = "127.0.0.1") -> str:
+    """What request.client.host is, behind uvicorn's proxy-headers middleware."""
+    from starlette.applications import Starlette
+    from starlette.responses import PlainTextResponse
+    from starlette.routing import Route
+    from starlette.testclient import TestClient
+    from uvicorn.middleware.proxy_headers import ProxyHeadersMiddleware
+
+    async def who(request):
+        return PlainTextResponse(request.client.host if request.client else "")
+
+    app = ProxyHeadersMiddleware(Starlette(routes=[Route("/", who)]),
+                                 trusted_hosts=trusted_hosts)
+    client = TestClient(app, client=(socket_peer, 40000))
+    return client.get("/", headers={"x-forwarded-for": xff} if xff else {}).text
+
+
+# ------------------------------------------------- what the peer actually resolves to
+
+def test_the_peer_is_the_forwarded_address_not_the_socket():
+    """The TCP connection is from loopback, but the peer reported is the proxy's."""
+    assert _peer_seen_by_the_app("10.89.0.178") == "10.89.0.178"
+
+
+def test_a_loopback_hop_still_looks_like_loopback():
+    """Which is why the in-container test minted and looked like proof."""
+    assert _peer_seen_by_the_app("127.0.0.1") == "127.0.0.1"
+
+
+def test_an_outer_proxy_adds_a_hop_and_the_peer_is_still_the_inner_one():
+    """host nginx -> published port -> internal nginx: $proxy_add_x_forwarded_for
+    appends, and uvicorn walks from the right to the first untrusted entry."""
+    assert _peer_seen_by_the_app("203.0.113.5, 10.89.0.178") == "10.89.0.178"
+
+
+def test_a_client_cannot_forge_loopback_by_sending_the_header():
+    """Our nginx APPENDS the real address, so a spoofed entry is never the one
+    picked -- the peer stays the proxy's own address, not the claimed 127.0.0.1."""
+    assert _peer_seen_by_the_app("127.0.0.1, 10.89.0.178") == "10.89.0.178"
+
+
+def test_the_default_trusted_proxies_reject_a_container_address():
+    """The failure in one line: the list in effect was the default."""
+    from codai.config import AdminForwardAuthConfig
+    default = AdminForwardAuthConfig().trusted_proxies
+    assert default == ["127.0.0.1", "::1"]
+    assert _peer_seen_by_the_app("127.0.0.1") in default
+    assert _peer_seen_by_the_app("10.89.0.178") not in default
+
+
+# --------------------------------------------------------------- the env override
+
+def test_the_trusted_proxy_list_has_an_env_override():
+    assert "CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_PROXIES" in UI
+
+
+def test_the_env_list_is_read_where_the_trust_decision_is_made():
+    blk = UI.split("def _fa_trusted(")[1].split("def _fa_identity")[0]
+    assert "_fa_peer_allowlist(cfg)" in blk, "the override must not be bypassed"
+
+
+def test_the_env_list_accepts_commas_or_spaces():
+    blk = UI.split("def _fa_peer_allowlist")[1].split("def _fa_reject")[0]
+    assert 'replace(",", " ").split()' in blk
+
+
+def test_an_empty_env_value_does_not_wipe_the_configured_list():
+    """An unset or blank variable must fall through to config.json, not trust
+    nothing -- that would be a second way to break a working deployment."""
+    blk = UI.split("def _fa_peer_allowlist")[1].split("def _fa_reject")[0]
+    assert "if raw.strip():" in blk
+    assert "cfg.trusted_proxies" in blk.split("if raw.strip():")[1]
+
+
+def test_the_override_is_documented_with_the_reason():
+    blk = UI.split("def _fa_peer_allowlist")[1].split("def _fa_reject")[0]
+    assert "read-only" in blk or "does not read" in blk
+
+
+# ------------------------------------------------------------------- diagnosability
+
+def test_a_refusal_says_why():
+    """A refusal is a 302 to the login form, which is indistinguishable from "no
+    session yet". Without a reason in the log this took a day to find."""
+    blk = UI.split("def _fa_reject")[1].split("def _fa_trusted(")[0]
+    assert "forward-auth refused" in blk
+
+
+def test_the_refusal_names_the_peer_and_the_list_it_was_checked_against():
+    blk = UI.split("def _fa_trusted(")[1].split("def _fa_identity")[0]
+    assert "trusted_proxies {trusted}" in blk, "print the list, not just the peer"
+    assert "_fa_reject(peer," in blk
+
+
+def test_a_missing_secret_and_a_wrong_secret_are_told_apart():
+    """"no header" and "does not match" are different operator mistakes."""
+    blk = UI.split("def _fa_trusted(")[1].split("def _fa_identity")[0]
+    assert "no {cfg.shared_secret_header} header" in blk
+    assert "does not match" in blk
+
+
+def test_the_refusal_log_never_prints_the_secret():
+    blk = UI.split("def _fa_trusted(")[1].split("def _fa_identity")[0]
+    for bad in ("{secret}", "{sent}"):
+        assert bad not in blk, f"{bad} would put a credential in the log"
+
+
+def test_the_refusal_log_is_rate_limited_and_bounded():
+    """A misconfigured proxy retries forever; it must not fill the disk."""
+    blk = UI.split("def _fa_reject")[1].split("def _fa_trusted(")[0]
+    assert "_FA_REJECT_WINDOW" in blk
+    assert "_FA_REJECTS.clear()" in blk, "the key set must be bounded"
+    assert "return False" in blk, "_fa_reject stands in for a refusal"
+
+
+def test_rejections_are_keyed_by_peer_and_reason():
+    blk = UI.split("def _fa_reject")[1].split("def _fa_trusted(")[0]
+    assert "key = (peer, reason)" in blk
+
+
+# ------------------------------------------------------------- the docstring is true
+
+def test_the_docstring_no_longer_claims_the_immediate_peer():
+    """It says "immediate peer" nowhere, because that is not what is checked --
+    believing it is what made the report look impossible."""
+    blk = UI.split("def _fa_trusted(")[1].split("def _fa_identity")[0]
+    assert "immediate peer" not in blk
+    assert "ProxyHeadersMiddleware" in blk, "say where the value comes from"