• Stefy Lanza (nextime / spora )'s avatar
    thermal: front-driven central supervisor with cooperative pause + SIGSTOP escalation · 707c89bb
    Stefy Lanza (nextime / spora ) authored
    The thermal guard was checkpoint-based and ran inside each engine, so it went
    blind during a single long native call (llama.cpp prefill / image encode): the
    engine can't reach a between-token checkpoint, and the [thermal][debug] heartbeat
    stops exactly when the GPU is hottest.
    
    Move the AUTHORITATIVE monitor to the front, which stays responsive regardless of
    what an engine is doing:
    
    * Front (engine_supervisor): a thermal thread reads per-card temps via gpu_stats()
      + CPU temp, maps each card to the owning engine by its CODERAI_ENGINE_GPUS
      selectors, and drives pause/resume with hysteresis (pause at *_high, resume only
      back at *_resume). A hot GPU pauses just its engine; a hot CPU pauses all.
    * Engines stop cooperatively, as before, but triggered remotely: the front POSTs
      /internal/thermal-pause; thermal.set_external_pause() makes wait_until_safe()/
      checkpoint() block at the next safe point (publishing cooldown state so the Tasks
      page shows it), until /internal/thermal-resume.
    * Escalation: if a paused engine keeps generating (inflight > 0) — stuck in a
      native call it can't interrupt — for stop_escalate_checks (default 3) consecutive
      checks, the front SIGSTOPs the engine's process group; SIGCONT on cooldown. Both
      signals target the session group so children freeze too.
    * stop_all()/restart_engine() SIGCONT a frozen engine first (a stopped process
      ignores SIGTERM until continued); _spawn() resets the thermal flags.
    * Config: thermal.supervisor_enabled (default on), thermal.stop_escalate_checks.
    * UI: per-engine temp + pause/frozen state in engines_list and the Tasks cooldown
      banner (covers a SIGSTOPped engine that can't report its own cooldown).
    
    Also: exclude coderai-runtime/*-runtime from the Docker build context (.dockerignore)
    — a root-owned runtime temp file was breaking the image build.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    707c89bb
app.py 21.6 KB