proxy+training: chain-aware nginx for double-proxy; report training model as loaded

Internal container nginx is now chain-aware: it prefers the outer proxy's
X-Forwarded-Proto/Host/Prefix and nests bundled sub-app prefixes under any
outer prefix (outer /ai + /township -> /ai/township). Fixes characters/
environments thumbnails and other absolute/sub-path URLs 404ing when the
all-in-one container runs behind a second reverse proxy. Documented the
required outer-proxy headers in docs/reverse-proxy-nginx.md.

LoRA training loads its base pipeline outside the model manager (and unloads
all manager models first), so the engine reported 0 loaded models mid-train.
Surface the active training base model via active_training_model() so the
engines card reflects the busy GPU.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.1.23"
+__version__ = "0.1.24"
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
 # expandable_segments lets the allocator return freed pages to the driver even

codai/api/app.py

@@ -303,6 +303,17 @@ async def internal_engine_state():
                     loaded.append(_mp)
     except Exception:
         pass
+    # A LoRA training job loads its base pipeline OUTSIDE the model manager (and
+    # unloads all manager models first), so without this the engine would report 0
+    # loaded models mid-training even though the GPU is fully busy. Surface the base
+    # model being trained so the engines card reflects what's actually resident.
+    try:
+        from codai.api.loras import active_training_model
+        _tm = active_training_model()
+        if _tm and _tm not in loaded:
+            loaded.append(_tm)
+    except Exception:
+        pass
     # VRAM is CACHED with a short TTL: this endpoint is polled every couple seconds
     # by the front's health monitor, and calling torch.cuda.mem_get_info /
     # get_device_name on EVERY poll can serialize behind the running generation on

codai/api/loras.py

@@ -75,6 +75,11 @@ _train_lock = threading.Lock()
 _jobs_lock = threading.Lock()
 _jobs: dict = {}                  # job_id -> record
 _active_job_id: Optional[str] = None
+# Base model of the job currently executing on the GPU. The trainer loads its base
+# pipeline OUTSIDE the model manager (and unloads all manager models first), so the
+# engine's loaded_models would otherwise read 0 mid-training even though the GPU is
+# busy. Surfaced via active_training_model() so the engine status reflects reality.
+_active_train_model: Optional[str] = None
 _bg_tasks: set = set()            # strong refs to detached train tasks
 # Job ids with a pending cancel. Survives the window before a queued job's task
 # is registered (the worker checks this set right after acquiring the GPU lock).
@@ -2044,6 +2049,17 @@ def _write_meta(name, req, base_path, n_images, arch, instance_prompt):
 _TRAIN_MODEL_KEY = "lora-train"
 
 
+def active_training_model() -> Optional[str]:
+    """The base model of the training job currently running on the GPU, or None.
+
+    The trainer holds its base pipeline outside the model manager, so this is the
+    only way the engine status can report that the GPU is busy with training rather
+    than showing 0 loaded models."""
+    if _train_lock.locked():
+        return _active_train_model
+    return None
+
+
 def _train_lora_blocking(req: LoraTrainRequest, job_id: Optional[str] = None) -> dict:
     """Run one training job to completion (called inside a worker thread).
 
@@ -2053,9 +2069,10 @@ def _train_lora_blocking(req: LoraTrainRequest, job_id: Optional[str] = None) ->
     `_active_job_id` is set so live progress mirrors into this job's record (and
     only this job's) for its owner to poll.
     """
-    global _active_job_id
+    global _active_job_id, _active_train_model
     _train_lock.acquire()
     _active_job_id = job_id
+    _active_train_model = getattr(req, "base_model", "") or None
     # Live cancellable task (id == job id). Registered here, when the job actually
     # starts on the GPU, so its progress mirrors via _set_progress.
     if job_id:
@@ -2113,6 +2130,7 @@ def _train_lora_blocking(req: LoraTrainRequest, job_id: Optional[str] = None) ->
         raise
     finally:
         _active_job_id = None
+        _active_train_model = None
         if job_id:
             _cancel_requested.discard(job_id)
             _force_resume_jobs.discard(job_id)

docs/reverse-proxy-nginx.md

@@ -141,3 +141,84 @@ server {
 
 Sub-path mounting for these three needs their client URLs made relative (the
 same change already applied to `video_editor.py`).
+
+> **Note:** inside the all-in-one Docker image the bundled nginx already mounts
+> `gen_township_fighters.py` at `/township/` (and the editor/videogen) and
+> rewrites its server-rendered asset URLs, so the township UI *does* work under
+> a sub-path **when reached through the container's own nginx**. The caveat above
+> applies only to running these tools standalone, directly behind your proxy.
+
+## Double proxy: the all-in-one container behind another reverse proxy
+
+This is the common production layout: the `coderai` Docker image already runs an
+**internal** nginx on `:8776` that fronts the API plus the bundled tool UIs
+(`/township/`, `/editor/`, `/videogen/`). You then put **your own** nginx in
+front of it (terminating TLS, on your real hostname) pointing at the container's
+LAN IP — two proxies in a chain.
+
+The internal nginx is **chain-aware**: it prefers the `X-Forwarded-Proto`,
+`X-Forwarded-Host`, and `X-Forwarded-Prefix` your outer proxy sends and only
+falls back to its own hop's values when they're absent. It also **nests** its
+sub-app prefixes under any outer prefix (outer `/ai` + bundled `/township` →
+`/ai/township`). So the *only* thing you have to get right is what your **outer**
+proxy advertises — if it doesn't tell the stack the public scheme/host/prefix,
+the container can only see the LAN IP + plain http on the inner leg, and
+absolute links (image/file URLs, redirects) and sub-path asset URLs break. That
+is exactly why the characters/environments thumbnails 404 in a misconfigured
+double proxy.
+
+**Outer proxy at the root** (`https://ai.example.com/` → container):
+
+```nginx
+server {
+    listen 443 ssl;
+    server_name ai.example.com;
+    # ssl_certificate ... ; ssl_certificate_key ... ;
+    client_max_body_size 4096m;
+
+    location / {
+        proxy_pass http://CONTAINER_LAN_IP:8776;
+        proxy_http_version 1.1;
+        proxy_set_header Host              $host;            # NOT the LAN IP
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;          # https, not http
+        proxy_set_header X-Forwarded-Host  $host;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        proxy_buffering off;                                  # SSE
+    }
+}
+```
+
+**Outer proxy under a sub-path** (`https://example.com/ai/` → container):
+
+```nginx
+location /ai/ {
+    proxy_pass http://CONTAINER_LAN_IP:8776/;   # trailing slash strips /ai
+    proxy_http_version 1.1;
+    proxy_set_header Host              $host;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_set_header X-Forwarded-Host  $host;
+    proxy_set_header X-Forwarded-Prefix /ai;     # <-- the key line; gets nested
+    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+    proxy_read_timeout 3600s;
+    proxy_buffering off;
+}
+```
+
+With the sub-path form, the township UI ends up correctly at
+`https://example.com/ai/township/` and its `/media/...` images resolve to
+`/ai/township/media/...`.
+
+The two most common double-proxy mistakes:
+
+1. **Omitting `proxy_set_header Host $host`** on the outer proxy. nginx then
+   sends `Host: CONTAINER_LAN_IP:8776` upstream, and CoderAI builds public file
+   URLs against the LAN IP — unreachable from the browser.
+2. **Omitting `X-Forwarded-Proto $scheme`** when the public side is HTTPS. The
+   inner leg is plain http, so links come back as `http://` and get blocked as
+   mixed content on an https page.
+
+If you'd rather not depend on headers at all, pin the API's public origin with
+`--url https://ai.example.com` (root mount only).

packaging/linux/nginx.conf

@@ -41,14 +41,41 @@ http {
     proxy_send_timeout   3600s;
     proxy_connect_timeout 75s;
 
+    # --- Chain-aware forwarded headers (works behind an OUTER proxy too) ------
+    # When this container sits behind another reverse proxy (a "double proxy"),
+    # the public scheme/host/sub-path are whatever the OUTER proxy already
+    # advertised. If we blindly set these to our own hop's values ($scheme is
+    # http on the outer->inner leg, $host is the LAN IP, prefix is just ours),
+    # CoderAI (codai/api/urlutils.py) and the bundled tools build public URLs
+    # against the wrong origin and images/links 404. So: prefer an incoming
+    # X-Forwarded-* value, fall back to our own only when the outer proxy didn't
+    # send one.
+    map $http_x_forwarded_proto $fwd_proto {
+        default $scheme;
+        "~."    $http_x_forwarded_proto;
+    }
+    map $http_x_forwarded_host $fwd_host {
+        default $host;
+        "~."    $http_x_forwarded_host;
+    }
+    # Incoming sub-path prefix from the outer proxy (trailing slash stripped),
+    # so we can NEST our own mount prefixes under it (e.g. outer "/ai" + our
+    # "/township" -> "/ai/township"). Empty when there is no outer prefix.
+    map $http_x_forwarded_prefix $fwd_prefix {
+        "~^(?<p>.*?)/?$" $p;
+    }
+
     # Shared proxy headers. CoderAI builds public URLs from these
     # (codai/api/urlutils.py); the tools honour X-Forwarded-Prefix for sub-paths.
     proxy_http_version 1.1;
-    proxy_set_header Host              $host;
+    proxy_set_header Host              $fwd_host;
     proxy_set_header X-Real-IP         $remote_addr;
     proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
-    proxy_set_header X-Forwarded-Proto $scheme;
-    proxy_set_header X-Forwarded-Host  $host;
+    proxy_set_header X-Forwarded-Proto $fwd_proto;
+    proxy_set_header X-Forwarded-Host  $fwd_host;
+    # Pass the outer prefix through unchanged at the root; sub-apps override this
+    # with their own nested value below.
+    proxy_set_header X-Forwarded-Prefix $http_x_forwarded_prefix;
 
     upstream coderai   { server 127.0.0.1:18776; }
     upstream editor    { server 127.0.0.1:8420;  }
@@ -63,7 +90,7 @@ http {
         # --- Video editor: https://host:8776/editor/ -------------------------
         location /editor/ {
             proxy_pass http://editor/;            # trailing slash strips the prefix
-            proxy_set_header X-Forwarded-Prefix /editor;
+            proxy_set_header X-Forwarded-Prefix "${fwd_prefix}/editor";  # nest under outer prefix
             proxy_request_buffering off;          # stream large uploads through
             proxy_buffering off;                  # SSE progress
         }
@@ -71,7 +98,7 @@ http {
         # --- Videogen studio: https://host:8776/videogen/ -------------------
         location /videogen/ {
             proxy_pass http://videogen/;
-            proxy_set_header X-Forwarded-Prefix /videogen;
+            proxy_set_header X-Forwarded-Prefix "${fwd_prefix}/videogen";  # nest under outer prefix
             proxy_request_buffering off;
             proxy_buffering off;
         }
@@ -79,7 +106,7 @@ http {
         # --- Township fighters: https://host:8776/township/ ----------------
         location /township/ {
             proxy_pass http://township/;
-            proxy_set_header X-Forwarded-Prefix /township;
+            proxy_set_header X-Forwarded-Prefix "${fwd_prefix}/township";  # nest under outer prefix
             proxy_request_buffering off;
             proxy_buffering off;
         }