• Stefy Lanza (nextime / spora )'s avatar
    front: expose active tasks on /metrics so training blocks an upgrade (v0.2.92) · dadb0196
    Stefy Lanza (nextime / spora ) authored
    The unattended upgrade measured idleness as requests in flight, and a LoRA
    training run has none: the request that starts it returns immediately and the
    work continues for hours, so coderai_engine_inflight is 0 throughout. An hourly
    upgrade would have restarted straight through one -- the standing "never restart
    during training" rule, reintroduced by automation.
    
    Nothing was missing in the training path. loras.py already registers a
    `training` task, the engine already ships active tasks in
    /internal/engine-state, and the front already stores them on
    EngineEntry.tasks. The gap was that /metrics never exposed them and the probe
    never looked, so this adds coderai_engine_jobs_active per engine and
    coderai_jobs_active across them, counting running/queued/paused -- paused on
    purpose, because a thermally paused generation has not finished and restarting
    loses it.
    
    The probe now reads both numbers from one scrape and takes the MAXIMUM, never
    the sum: a generation is both an in-flight request and a running task, so adding
    them would double-count and the numbers in the log would match nothing an
    operator can see. A front that predates the gauge reports "-" and the detail
    line says "front too old", rather than claiming zero and silently restoring the
    blind spot.
    
    tests/test_autoupgrade_sees_training.py renders the exposition from a fake
    engine carrying a training task, runs the probe's awk program as the script
    actually has it, and pins both halves of the premise -- the registration in
    loras.py and the engine's reporting of active states -- so the gauge cannot go
    quiet without a test failing.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    dadb0196
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...