-
Stefy Lanza (nextime / spora ) authored
The thermal guard was checkpoint-based and ran inside each engine, so it went blind during a single long native call (llama.cpp prefill / image encode): the engine can't reach a between-token checkpoint, and the [thermal][debug] heartbeat stops exactly when the GPU is hottest. Move the AUTHORITATIVE monitor to the front, which stays responsive regardless of what an engine is doing: * Front (engine_supervisor): a thermal thread reads per-card temps via gpu_stats() + CPU temp, maps each card to the owning engine by its CODERAI_ENGINE_GPUS selectors, and drives pause/resume with hysteresis (pause at *_high, resume only back at *_resume). A hot GPU pauses just its engine; a hot CPU pauses all. * Engines stop cooperatively, as before, but triggered remotely: the front POSTs /internal/thermal-pause; thermal.set_external_pause() makes wait_until_safe()/ checkpoint() block at the next safe point (publishing cooldown state so the Tasks page shows it), until /internal/thermal-resume. * Escalation: if a paused engine keeps generating (inflight > 0) — stuck in a native call it can't interrupt — for stop_escalate_checks (default 3) consecutive checks, the front SIGSTOPs the engine's process group; SIGCONT on cooldown. Both signals target the session group so children freeze too. * stop_all()/restart_engine() SIGCONT a frozen engine first (a stopped process ignores SIGTERM until continued); _spawn() resets the thermal flags. * Config: thermal.supervisor_enabled (default on), thermal.stop_escalate_checks. * UI: per-engine temp + pause/frozen state in engines_list and the Tasks cooldown banner (covers a SIGSTOPped engine that can't report its own cooldown). Also: exclude coderai-runtime/*-runtime from the Docker build context (.dockerignore) — a root-owned runtime temp file was breaking the image build. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
707c89bb