0.4.16 fixed one blocking call on the loop by finding it the hard way: a
5,000-document run rented 127 pods and completed nothing for ten minutes,
and the only honest signal in the system was that /v1/runpod/status had
stopped answering. Nothing in the process said "the loop was blocked",
though it had been, for minutes at a time.

Both halves of that are addressed here.

MEASURED: codai/api/loop_watch.py samples the loop's own behaviour -- a
sleep that asks for 0.5 s and returns at 4 s says the loop was busy
elsewhere for 3.5 s and served NOTHING in that window. Installed in both
the engine and the front (the front is the process the client talks to, so
a stall there is the one it experiences), logged per stall with its
duration, and exported on /metrics as coderai_loop_lag_seconds,
_lag_worst_seconds, _stalls_total and _blocked_seconds_total. It cannot
name the culprit, but it turns "the server is slow" into "the server was
not running", which is a different investigation.

CHECKED: tests/test_no_blocking_calls_on_the_loop.py walks every coroutine
in codai/ for calls that block the calling thread. What makes it precise is
that it does not look inside a nested def or lambda -- `to_thread(lambda:
f())` and `run_on_pod_pool(f, x)` are both correct, so only a call reached
without crossing a function boundary is on the loop. Its own ten tests
prove it catches what it claims and ignores os.path.join, asyncio
primitives and coroutines handed to ensure_future.

It found 20 more, all fixed:

  * ocr/gateway.py: `placement.choose()` in async chat_completions -- the
    SAME 1200 s pool acquire as 0.4.16, on the chat path instead of the page
    path, and with no retry loop above it to make the wait visible.
  * api/ocr.py: a sync urlopen of a caller-supplied URL, so an OCR step was
    as slow as whoever served that URL, for every request in the process.
  * eleven `subprocess.run()` ffmpeg/ffprobe calls across faceswap, video,
    pipelines, voice_clone, custom_pipelines and the outfit video path. A
    media encode is seconds to minutes, and all of it was everyone's wait.
  * two HuggingFace API fetches in the admin routes.
  * two `thread.join()`s on the CUDA streaming path, and a `time.sleep(0.1)`
    per generated image.

The allowlist in that test is empty and should stay empty: an entry in it is
a request that cannot be served while the named call runs.

Also tools/loadprobe.py, for the outside view: /healthz does no work and
returns the instant the loop is free, so its latency IS loop lag as a client
sees it. p99 healthz flat while documents crawl is a capacity problem; p99
healthz in seconds is not, and no extra capacity would have helped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

* pipelines.py: /v1/pipelines/* (image-to-video, story, video-dub, audio-dub)
  chained their steps by calling the image/video/text/TTS handlers DIRECTLY
  in-process, forcing every modality onto whichever engine received the pipeline
  request. In a multi-engine deployment each model may live on a different engine.
  Route every sub-step through the front via execute_api_request (same fix as
  characters/environments) so each lands on the engine that owns that model;
  in-process fallback for single-process mode. Drops the unused TestClient import.

* supervisord.conf: the bundled tools (video_editor, videogen, township) now use
  --base-url http://127.0.0.1:8776 — the PUBLIC nginx port — so they talk to
  coderai exactly like any external client (nginx already proxies the full API
  with 4G bodies / 1h timeouts). The front's own bind stays CODERAI_PORT=18776.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

Tasks / queue management:
- Central in-memory task registry with cooperative cancel, pause/resume,
  and step progress across image/video/audio/text generation + LoRA training
- Tasks admin page (live 2s poll): cancel, interrupt, pause/resume, restart,
  remove; done jobs auto-drop from the list; bounded persisted job history
- Disable interrupted-training recovery via --no-resume-jobs + settings toggle

Quantization / acceleration:
- TurboQuant embedding vector quantization (data-free, inner-product
  preserving): built-in NumPy backend + optional turboquant-py library,
  selectable per embedding model; /v1/embeddings `quantization` param
- llama.cpp KV-cache quantization (cache_type_k/v) for GGUF text models,
  configurable in the Models UI

Hardware telemetry:
- Thermal cooldown state surfaced on the Tasks page (banner + per-task badge)
- Live CPU/GPU/RAM/VRAM usage + temperature panel via /admin/api/system-stats

Docs: API documentation gaps/accuracy pass + Swagger overhaul; DISTRIBUTION.md
implementation spec. Plus I2V LoRA training channel-mismatch fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>