fix: front-driven thermal pause indicator no longer erased by engine poll

On a global CPU cooldown the thermal supervisor already pauses ALL engines
(want_pause = gpu_hot or cpu_hot, applied per-engine), and the logs confirm
every engine gets a pause. But the Tasks page often showed only ONE engine
as cooling while the others looked like they were still running.

Cause: two threads write engine.cooling. The thermal loop sets it when it
pauses an engine; the health poll loop overwrites it every tick with the
engine's OWN self-reported cooldown (cooling=d.get("cooling")). An engine
only self-reports cooling while sitting in its own wait_until_safe loop (it
has an in-flight request). An engine the front paused while IDLE reports
cooling=None, so the poll loop cleared the front's pause indicator — the UI
then showed that engine as running mid-cooldown.

Fix: while the front holds an engine paused (engine.therm_paused), pass the
update_state sentinel (cooling=False, "don't touch") instead of the engine's
self-report, so the front's indicator survives. Once the front resumes the
engine, the engine's self-report flows through again as before.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoSpEthysqmseCc6Geizty
parent 1126bdbc
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.60" __version__ = "0.1.61"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
...@@ -840,13 +840,26 @@ class EngineSupervisor: ...@@ -840,13 +840,26 @@ class EngineSupervisor:
d = r.json() d = r.json()
healthy = True healthy = True
loaded = d.get("loaded_models") or [] loaded = d.get("loaded_models") or []
# Cooling indicator: the FRONT's thermal supervisor owns the
# pause state (it's what pauses ALL engines on a global CPU
# cooldown). The engine only self-reports cooling while it's
# sitting in its OWN wait_until_safe loop (i.e. it has an
# in-flight request). An engine the front paused while idle
# reports cooling=None — so blindly taking the engine's report
# here would ERASE the front's pause indicator and the Tasks
# page would show that engine as running mid-cooldown. While
# the front holds this engine paused, keep the front's
# indicator (sentinel False = don't touch); only let the
# engine's self-report through once the front isn't pausing it.
_cooling = (False if getattr(engine, "therm_paused", False)
else d.get("cooling"))
self.registry.update_state( self.registry.update_state(
engine.id, healthy=True, engine.id, healthy=True,
loaded_models=loaded, loaded_models=loaded,
loaded_info=d.get("loaded_info") or [], loaded_info=d.get("loaded_info") or [],
vram=d.get("vram"), vram=d.get("vram"),
tasks=d.get("tasks") or [], tasks=d.get("tasks") or [],
cooling=d.get("cooling"), cooling=_cooling,
) )
# Backstop: the engine answered, so it's no longer GIL-blocked # Backstop: the engine answered, so it's no longer GIL-blocked
# loading. Clear the log-parsed loading state (the real task, # loading. Clear the log-parsed loading state (the real task,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment