launcher: reap stale front/engine trees before start

Engines run in their own sessions, so supervisord's killasgroup cannot
reach them; teardown relies on the front's shutdown handler + the
engines' PDEATHSIG. That chain leaked a full front+engine tree across a
restart — two trees then fought over one GPU (phantom VRAM, evictions
freeing nothing, duplicated model workers, requests served by one tree
while logs came from the other). The launcher now pkills any surviving
coderai-front/engine/dinov2-embed processes before starting: at that
point none can legitimately exist.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
parent e9131f38
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.48" __version__ = "0.1.49"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
...@@ -86,6 +86,18 @@ set -- --config "$CONFIG_DIR" --host "$CODERAI_HOST" --port "$CODERAI_PORT" "$@" ...@@ -86,6 +86,18 @@ set -- --config "$CONFIG_DIR" --host "$CODERAI_HOST" --port "$CODERAI_PORT" "$@"
CODERAI_BIN="/opt/coderai/python/bin/python3 /opt/coderai/app/coderai" CODERAI_BIN="/opt/coderai/python/bin/python3 /opt/coderai/app/coderai"
# Stale-tree sweep: engines run in their OWN sessions (setsid), so supervisord's
# killasgroup cannot reach them — teardown relies on the front's shutdown handler
# and the engines' parent-death signal. That chain has been observed to leak a
# whole front+engine tree across a restart (two trees then fight over one GPU:
# phantom VRAM usage, evictions that free nothing, duplicated model workers).
# At THIS point no coderai server process can legitimately exist yet, so reap
# any survivor from a previous incarnation before starting.
for _pat in 'coderai-front' 'coderai-nvidia' 'coderai-nvidia-gguf' \
'coderai-radeon' 'coderai-system' 'dinov2-embed'; do
pkill -9 -x "$_pat" 2>/dev/null || true
done
# Optional host-tailable file log. CODERAI_LOG_FILE should point under a mounted # Optional host-tailable file log. CODERAI_LOG_FILE should point under a mounted
# volume (e.g. /cache/logs/coderai.log) so it's visible + tailable on the host. # volume (e.g. /cache/logs/coderai.log) so it's visible + tailable on the host.
# We tee so output still reaches `docker logs` too. (supervisord runs this script # We tee so output still reaches `docker logs` too. (supervisord runs this script
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment