• Stefy Lanza (nextime / spora )'s avatar
    thermal: front-driven central supervisor with cooperative pause + SIGSTOP escalation · 707c89bb
    Stefy Lanza (nextime / spora ) authored
    The thermal guard was checkpoint-based and ran inside each engine, so it went
    blind during a single long native call (llama.cpp prefill / image encode): the
    engine can't reach a between-token checkpoint, and the [thermal][debug] heartbeat
    stops exactly when the GPU is hottest.
    
    Move the AUTHORITATIVE monitor to the front, which stays responsive regardless of
    what an engine is doing:
    
    * Front (engine_supervisor): a thermal thread reads per-card temps via gpu_stats()
      + CPU temp, maps each card to the owning engine by its CODERAI_ENGINE_GPUS
      selectors, and drives pause/resume with hysteresis (pause at *_high, resume only
      back at *_resume). A hot GPU pauses just its engine; a hot CPU pauses all.
    * Engines stop cooperatively, as before, but triggered remotely: the front POSTs
      /internal/thermal-pause; thermal.set_external_pause() makes wait_until_safe()/
      checkpoint() block at the next safe point (publishing cooldown state so the Tasks
      page shows it), until /internal/thermal-resume.
    * Escalation: if a paused engine keeps generating (inflight > 0) — stuck in a
      native call it can't interrupt — for stop_escalate_checks (default 3) consecutive
      checks, the front SIGSTOPs the engine's process group; SIGCONT on cooldown. Both
      signals target the session group so children freeze too.
    * stop_all()/restart_engine() SIGCONT a frozen engine first (a stopped process
      ignores SIGTERM until continued); _spawn() resets the thermal flags.
    * Config: thermal.supervisor_enabled (default on), thermal.stop_escalate_checks.
    * UI: per-engine temp + pause/frozen state in engines_list and the Tasks cooldown
      banner (covers a SIGSTOPped engine that can't report its own cooldown).
    
    Also: exclude coderai-runtime/*-runtime from the Docker build context (.dockerignore)
    — a root-owned runtime temp file was breaking the image build.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    707c89bb
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...