Reporting VRAM via torch.cuda.mem_get_info lazily creates the CUDA primary
context (~256 MiB on an RTX 3090). An engine that never loads a torch model
(the GGUF/llama.cpp engine) therefore pinned ~256 MiB just to answer health
polls, the capability probe and the load-path eviction check — and that stray
context was enough to tip a borderline 4-bit Wan2.2 A14B video load into OOM.

New codai/models/gpu_query.py queries VRAM without a context: pynvml first,
nvidia-smi fallback, torch only if a context already exists. visible_gpu_memory()
scopes to the engine's cards via CUDA_VISIBLE_DEVICES (matched by index OR UUID;
empty value -> no CUDA cards, e.g. the Vulkan/Radeon engine).

Wired into the idle health poll (api/app.py), the capability probe
(broker/capabilities.py) and the load-path free-VRAM checks
(models/manager.py: _get_free_vram_gb / _free_vram_snapshot). Adds nvidia-ml-py
to requirements-nvidia.txt and the update overlay. Bumps to 0.1.28.

Net: the GGUF engine sits at 0 MiB while idle (its real context comes from
llama.cpp on model load and is freed on unload), returning the headroom that
makes the A14B load fit instead of OOM-ing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

The 0.1.24 nesting blindly appended the bundled sub-app name to the incoming
X-Forwarded-Prefix, which double-counted it when the outer proxy mounts the app
under the SAME name and proxies to our /<app>/ path (topology B): outer sends
"/township" -> container emitted "/township/township", so every township
fetch() 404'd ("Not found" -> JSON parse error in the UI).

Per-app maps now only append our name when the incoming prefix doesn't already
end in it, so both topologies work: outer-wraps-everything ("/ai" -> "/ai/town
ship") and outer-mounts-same-name ("/township" -> "/township"). Direct (no outer
proxy) still yields "/township".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

Internal container nginx is now chain-aware: it prefers the outer proxy's
X-Forwarded-Proto/Host/Prefix and nests bundled sub-app prefixes under any
outer prefix (outer /ai + /township -> /ai/township). Fixes characters/
environments thumbnails and other absolute/sub-path URLs 404ing when the
all-in-one container runs behind a second reverse proxy. Documented the
required outer-proxy headers in docs/reverse-proxy-nginx.md.

LoRA training loads its base pipeline outside the model manager (and unloads
all manager models first), so the engine reported 0 loaded models mid-train.
Surface the active training base model via active_training_model() so the
engines card reflects the busy GPU.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

Trained LoRAs, characters, environments and voices are resolved from the legacy
HOME-based dir ~/.coderai, not from --config. In the container the user's home
(e.g. /home/ubuntu) isn't where config is mounted ($CONFIG_DIR/coderai), so those
lookups missed the mounted data and image-gen failed with "LoRA '<name>' not found
on server". Symlink ~/.coderai -> $CODERAI_CONFIG_DIR/coderai so HOME-based and
--config lookups agree, removing the need for an extra per-deployment --map. Acts
only when safe (a symlink, missing, or an empty dir) so a user-mounted volume or
real data is never clobbered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

Loading an HF model (e.g. Qwen3.5-9B, 4-bit) failed with 'Permission denied:
./offload'. Cause: the model's per-model offload_dir was the relative './offload'
(a stale auto-saved default), which _cfg_or_global lets win over the global
config; './offload' resolves to the CWD = the READ-ONLY /opt/coderai/app tree in
the image. The config WAS respected — a relative offload path is just meaningless
where the CWD isn't writable.

* hf_loading.resolve_offload_dir(): an absolute offload_dir is respected as-is; a
  relative/empty one INHERITS the configured GLOBAL offload directory
  (global_args.offload_dir) when absolute, then CODERAI_OFFLOAD_DIR, then the user
  cache — never the CWD. Applied in the manager (both load sites, always passed),
  hf_loading, and defensively in the cuda backend.
* main.py + entrypoint: a container-writable CODERAI_OFFLOAD_DIR (=/cache/offload,
  created by the entrypoint) is used when the GLOBAL config is still the bare
  './offload' default; explicit config wins.
* run_oci.sh: forward HF_TOKEN / HUGGING_FACE_HUB_TOKEN from the host env so the
  engines authenticate to the HF Hub (the 'unauthenticated requests' warning) for
  higher rate limits + gated models. HF_HOME/cache dir was already honoured
  (main.py from config.models.hf_cache_dir).

Bump version to 0.1.8.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

* coderai-docker (run_oci.sh): new --tool-arg TOOL VAL (one token) and
  --tool-args TOOL STR (whitespace string), repeatable, for video-editor |
  videogen | township | parler. They accumulate into CODERAI_<TOOL>_ARGS env vars
  that supervisord appends to each tool's command line. Documented in --help
  (plus the --map dir-or-file note and bring-your-config examples), and shown on
  the startup banner (tool-args:).
* entrypoint: default+export the four CODERAI_*_ARGS (empty) so supervisord's
  %(ENV_...)s never fails on an undefined var; pre-create /cache/video_editor/sessions.
* supervisord: append %(ENV_CODERAI_*_ARGS)s to all four tool launchers; the video
  editor now runs with --session --session-dir /cache/video_editor/sessions so
  editor-state recovery is ON by default and persists on the cache volume.
* video_editor.py: new --session-dir to place session state/assets at a
  persistent/mapped path (defaults to ~/.cache/... when omitted).

Bump version to 0.1.6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

* pipelines.py: /v1/pipelines/* (image-to-video, story, video-dub, audio-dub)
  chained their steps by calling the image/video/text/TTS handlers DIRECTLY
  in-process, forcing every modality onto whichever engine received the pipeline
  request. In a multi-engine deployment each model may live on a different engine.
  Route every sub-step through the front via execute_api_request (same fix as
  characters/environments) so each lands on the engine that owns that model;
  in-process fallback for single-process mode. Drops the unused TestClient import.

* supervisord.conf: the bundled tools (video_editor, videogen, township) now use
  --base-url http://127.0.0.1:8776 — the PUBLIC nginx port — so they talk to
  coderai exactly like any external client (nginx already proxies the full API
  with 4G bodies / 1h timeouts). The front's own bind stays CODERAI_PORT=18776.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

Mirror the township config fix for the bundled video editor:
* supervisord: pass --config /cache/video_editor/video_editor.config.json so a
  bind-mounted (or cache-persisted) config auto-loads — previously it only
  auto-loaded video_editor.config.json from the baked cwd, which isn't mappable.
* video_editor.py: a MISSING --config file is no longer fatal (was raise
  SystemExit) — start with defaults; the web UI Save creates it.
* entrypoint: pre-create /cache/video_editor so the config dir exists for
  auto-load + web-UI Save (persists on the standard /cache volume by default).
* run_oci.sh (coderai-docker): --map now accepts a single FILE, not just a dir,
  so a tool config that lives loose on the host can be linked directly, e.g.
  --map /host/video_editor.config.json:/cache/video_editor/video_editor.config.json

Bump version to 0.1.5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

The bundled township tool (supervisord [program:township]) launched without
-c/--config, so a township_config.json bind-mounted via --map into
/cache/township_output was never read — the web UI came up with blank API key /
league upload credentials / options every restart.

* supervisord: pass --config /cache/township_output/township_config.json so a
  mapped config auto-loads. Map the bare-metal dir with
    coderai-docker --map /path/to/township_output:/cache/township_output
  and the saved settings + credentials come up automatically.
* gen_township_fighters.py: a MISSING --config file is no longer fatal (fresh
  install with no mapped dir) — fall back to defaults and start normally; a
  malformed config still errors. The web UI's Save already writes back to
  <out-dir>/township_config.json, so first-run saves persist into the mapped dir.

Bump version to 0.1.4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

`docker load -i` shows no useful progress and the installer captured its output
into a variable, so a ~12G load looked like a multi-minute hang. Stream the
tarball through a meter into `docker load` instead — the file read tracks load
progress closely. Prefer `pv` (bar + ETA), fall back to GNU `dd status=progress`
(bytes + throughput), else plain load with a tip to install pv. The meter draws
on stderr so the "Loaded image:" line is still captured. Pre-authenticate sudo
first so its prompt doesn't collide with the bar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

Three operational fixes for the distributable image:

run_oci.sh: default the container to the invoking user (uid:gid, SUDO_UID-aware)
instead of the image's root default, with a new --root opt-out. The image has no
USER directive and supervisord sets no user=, so a run without --user created
root-owned dirs (logs, coderai-tmp, hf cache) in the bind-mounted /cache; a later
--user run then couldn't write them ("cannot write /cache/logs ... logging to
stdout only"). Running as the user by default makes a fresh install safe
regardless of run order. --root restores the old behaviour (and throwaway config
copy) for shared root-managed data dirs.

nginx.conf: error_log crit (was info) + access_log off. nginx's [notice]
startup/worker lines and per-request access logs were piped to the container
console by supervisord and buried coderai's own output on attach / docker logs.
Real failures (crit/alert/emerg) still surface.

install.sh: print an extensive post-install guide (quick start, --local + --map,
where data lives, the run-as-you default, file logging, docker logs/stop, --help).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

A multi-stage build leaves untagged <none> intermediate layers behind that pile
up across rebuilds. After the build (and the dist export, which still needs the
final image), run `docker image prune -f` to drop them. Only DANGLING images are
removed — the final image, the pulled CUDA base, and any other tagged images are
kept. Default on; disable with --no-prune.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontproxy:
- _DROP_RESP now drops date/server when relaying engine responses. The front's
  own ASGI server adds Date/Server, so keeping the engine's too produced DUPLICATE
  header lines — which nginx logged as a warning on every request, flooding the
  terminal. Now each appears once.
- Brokered route logs the engine's actual status/bytes/preview (or "NO ENGINE ->
  503") so a brokered request that "doesn't get executed" (instant tiny reply) is
  diagnosable from the log.

coderai-oci:
- Use the host-tailable file log only when it's actually writable. When the
  container runs as --user but the log dir/file was created by an earlier root
  run, it's root-owned; tee then spammed "Permission denied" and the file stayed
  empty. Now we detect that and log to stdout only with a clear note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

launcher: image auto-resolution, --user in-place config persistence, --debug SPEC, install/uninstall image handling

run_oci.sh:
- Resolve the installed coderai image when none is given (single -> use it,
  several -> menu); no longer hardcodes a possibly-wrong tag.
- --debug accepts SPEC as the next token (not just --debug=SPEC) and won't
  swallow an image-looking arg, fixing "image: <debug spec>" mishaps.
- --local puts the runtime dir under ~/.config/coderai-runtime (override
  --data-dir).
- New --user[=UID[:GID]]: run as that user and switch a config dir to an
  IN-PLACE mount so the app's config edits persist (owned by you). Without
  --user, --local/--config-dir stay a throwaway copy. Banner shows user.

Non-destructive config persistence (no startup rewrite):
- codai/cli.py: new --host/--port that override config.server in memory only.
- codai/main.py: apply those overrides right after config load (never written
  to config.json).
- coderai-oci: pass --host/--port to the server and stop rewriting an existing
  config.json (only create one on true first run) — so an in-place-mounted
  config dir is never modified at startup.

install.sh / uninstall.sh:
- install: after loading, offer to remove OTHER coderai images (default no);
  no auto-launch.
- uninstall: resolve the installed coderai image(s) to remove (single ->
  confirm, several -> menu / all / none) instead of a hardcoded tag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderai-oci (in-image): append ${CODERAI_EXTRA_ARGS} to the server argv in both
exec paths. supervisord runs a fixed command, so arbitrary server flags arrive
via this env var rather than argv. (Takes effect after an image rebuild.)

run_oci.sh: --coderai-arg ARG (repeatable, one token each) and --coderai-args
"STR" build CODERAI_EXTRA_ARGS and inject it with -e; new cdr-args banner line.
`--` still goes to the container engine, not coderai.

Docs (AI.PROMPT, dist-bundle README.md/.txt) updated accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

build_oci_image.sh:
- --versioned derives a deterministic tag from codai.__version__ + build mode +
  GPU: local-venv -> coderai:full_<gpu>_<version>, from-scratch ->
  coderai:base_<gpu>_<version> (overrides -t/--tag).
- --gpu all|nvidia|vulkan (default all) sets the <gpu> token; label only — the
  image always bundles both CUDA and Vulkan.

run_oci.sh:
- --host ADDR binds the published port to a specific interface (-p ADDR:PORT:8776);
  default unchanged (all interfaces). CODERAI_HOST stays 0.0.0.0 in-container.
  Banner URL reflects the bind host.

Docs (AI.PROMPT, dist-bundle README.md/.txt, README-RUN.txt):
- Document --versioned/--gpu, additive --nvidia/--vulkan/--all, --vulkan
  auto-libcuda + --with-libcuda, graceful llama-cpp degradation, --host, and the
  new uninstall.sh + confirm gates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docker/backend: graceful llama-cpp load + additive GPU modes + libcuda mapping; admin GGUF batch/slots tuning

Backend robustness:
- vulkan.py catches Exception (not just ImportError) around the llama_cpp
  import: a CUDA-built llama-cpp missing libcuda.so.1 raised RuntimeError/OSError
  that crash-looped the whole server. Now it logs a warning and marks the
  Vulkan/GGUF backend unavailable; CUDA/CPU/ds4 keep working.
- detect_available_backends() reads LLAMA_CPP_AVAILABLE instead of re-importing
  (which re-raised the same error).

Docker launcher (run_oci.sh):
- GPU backends are now additive: --nvidia --vulkan enables both (maps libcuda via
  --gpus all AND /dev/dri). Added --all and --with-libcuda[=PATH].
- --vulkan auto bind-mounts the host's libcuda.so.1 (the bundled llama-cpp is a
  CUDA build), so Vulkan GGUF loads without full --gpus all. Banner shows mode set
  and libcuda status.

Dist bundle:
- New uninstall.sh (removes runner + optional image), wired into make_dist_bundle.
- install.sh + unins...

Optional CPU-temperature source for thermal control. Not essential — the
thermal monitor reads CPU temp from psutil and the kernel's /sys/class/thermal
zones first (both work in-container); `sensors` is only a last-resort text
parse for hosts whose sysfs doesn't expose a CPU zone. Added to both the
from-venv and from-scratch runtime stages for completeness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a GitHub-flavored README.md alongside the existing README.txt in the
all-in-one docker distribution bundle, and have make_dist_bundle.sh stage it
so future builds include both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The image's demo tool UIs (video editor :8420, videogen :7790, township :7788,
parler TTS :8123) are now individually toggleable at container start — no
rebuild. supervisord.conf drives each program's autostart from a CODERAI_TOOL_*
env var; the entrypoint seeds defaults (three UIs on, parler off) so the
%(ENV_...)s expansion always resolves. run_oci.sh (shipped as the dist
coderai-docker runner) gains:

  --no-tools            disable all three demo UIs
  --enable-tool NAME    force-enable one (also turns on parler)
  --disable-tool NAME   disable one; explicit toggles override --no-tools

NAME ∈ {video-editor, videogen, township, parler}. README + --help updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- make_dist_bundle.sh: assemble dist/coderai-docker-dist.tar containing the image
  tarball (docker save | pigz), the coderai-docker runner (run_oci.sh, image tag
  pinned), install.sh and README. Stages under dist/ (not tiny /tmp) and hardlinks
  the multi-GB image tarball instead of copying it.
- dist-bundle/install.sh: docker-load the image (sudo-fallback for daemon access)
  then install the coderai-docker runner to /usr/local/bin (root) or
  ~/.local/usr/bin (user, added to ~/.bashrc PATH if missing).
- build_oci_image.sh: after a successful build, export + bundle for distribution
  by default (--no-dist to skip).
- run_oci.sh: default image tag -> coderai:dist (matches what's shipped/loaded).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Web UI / models:
- "To download" wishlist: models known but not on disk and not configured show
  as non-configured to-download rows. Free-disk on an unconfigured model, Remove
  on a model with no files left, and a new "Add to list" button in the download
  window all record into models.json `to_download`; pruned on enable/download.
  New endpoints model-mark-download / model-unmark-download.
- mmproj multimodal components: mmproj GGUFs are classified as components (not
  models), selectable per-GGUF in the model config (auto-selected, enables vision
  capability). VulkanBackend loads them via llama.cpp's MTMDChatHandler (--mmproj
  equivalent), and the chat path now forwards image_url content end-to-end.
- All window.alert() replaced by a shared styled showAlert()/showConfirm() modal
  in base.html (used across every admin template).

Front proxy / broker:
- Fix engine model-assignment NameError (keep -> _keep).
- Brokered GET /coderai/capabilities now answers from the front (whole node) so
  multi-GPU hosts report every card, not a single engine's CUDA-visible one.
- Log a clear reason when the broker is disabled.

Packaging (distributable OCI image):
- Multi-stage venv image + smoke test; bundle ds4/wav2lip/sadtalker + parler;
  whisper-server etc. dereferenced (cp -aL) so no dangling symlinks.
- Dockerfile.update + update_oci_image.sh: ~30s incremental code-only rebuild on
  an immutable coderai:base (no 20GB bundle recopy).
- run_oci.sh: --local/--config-dir + --map to run against existing local config
  and data dirs without a rebuild; --debug[=flags] + --log-file for selectable
  debug flags and a host-tailable file log (launcher tees; supervisord kills the
  process group). tmp_janitor age-prunes the dedicated temp dir.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- frontproxy: torch-free front proxy + per-vendor engine supervisor with auth,
  localhost binding, model routing; Ctrl-C now force-kills engines (own session +
  PDEATHSIG, SIGKILL of engine process groups, watchdog on hung drain)
- gemma-4 tool calling: prompt via native tools= template, parse call:NAME{...}
  into tool_calls, honour generation_config EOS so it stops instead of looping
- ds4 external worker, parler/expressive TTS backends, video editor tooling
- --debug-requests: full client<->API request/response logging + live snapshots
- stop tracking runtime artifacts (video_editor/sessions/, tools/coderai_media/)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Township tool (tools/gen_township_fighters.py):
- Outcome videos now generate TWO keyframes per outcome (finish + victory),
  each anchoring its own clip; victory clip uses a dedicated referee shot.
- Referee characters: new role on create form, kept out of fighter pools,
  dressed as officials, attachable per-match and used in victory keyframes.
- Per-match referee selection (new-match form + match editor, persisted).
- Autogenerate buttons on character/referee, environment and new-match forms
  (LLM-filled, editable before create) via /profile/autogen + /matches/autogen.
- Single-worker generation queue: all coderai-bound jobs (create/regen/train/
  match/process) are serialised and surfaced as "queued", with one persistent
  match-detail monitor replacing the competing per-job pollers (fixes the
  blinking progress when two jobs were launched at once).

coderai: favicon.ico served at /favicon.ico + linked in admin/login templates;
bundled township favicon served at /favicon.ico.

Also gitignore large packaging/runtime artifact dirs (.packaging-cache/, tmp/).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>