-
Stefy Lanza (nextime / spora ) authored
The unattended upgrade measured idleness as requests in flight, and a LoRA training run has none: the request that starts it returns immediately and the work continues for hours, so coderai_engine_inflight is 0 throughout. An hourly upgrade would have restarted straight through one -- the standing "never restart during training" rule, reintroduced by automation. Nothing was missing in the training path. loras.py already registers a `training` task, the engine already ships active tasks in /internal/engine-state, and the front already stores them on EngineEntry.tasks. The gap was that /metrics never exposed them and the probe never looked, so this adds coderai_engine_jobs_active per engine and coderai_jobs_active across them, counting running/queued/paused -- paused on purpose, because a thermally paused generation has not finished and restarting loses it. The probe now reads both numbers from one scrape and takes the MAXIMUM, never the sum: a generation is both an in-flight request and a running task, so adding them would double-count and the numbers in the log would match nothing an operator can see. A front that predates the gauge reports "-" and the detail line says "front too old", rather than claiming zero and silently restoring the blind spot. tests/test_autoupgrade_sees_training.py renders the exposition from a fake engine carrying a training task, runs the probe's awk program as the script actually has it, and pins both halves of the premise -- the registration in loras.py and the engine's reporting of active states -- so the gauge cannot go quiet without a test failing. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com>
dadb0196
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| common | ||
| dinov2cpp | ||
| linux | ||
| runpod | ||
| windows | ||
| build-rpc-server.sh | ||
| cosign.pub | ||
| patch-colibri.py | ||
| patch-k3.py | ||
| setup-salad-venv.sh | ||
| sign-images.sh | ||
| versions.env |