packaging: hourly unattended upgrade that will not interrupt a request (v0.2.89)

Two production installs now follow this repo: zeiss (nexlab) and the Digesta
host in Aruba. The Aruba one should track `production` without anyone logging
in, so this adds coderai-autoupgrade plus a systemd user service and an hourly
timer.

It skips the upgrade outright while CoderAI is serving -- checked BEFORE the
image is touched, not merely before the restart, so a busy host never reaches
the state where new code waits in the image. Two probes and the higher count
wins: the front's coderai_engine_inflight (every request in flight, pod-served
ones included) and the pods' own summary.inflight from /v1/runpod/status. The
RunPod one carries the weight on an orchestrator where every model is remote:
the local engines are idle by definition, so the pods' count is the only sign
work is happening, and killing a pod-served request loses the work AND keeps
paying for the pod that was doing it. The check fails CLOSED -- no token, wrong
URL or an unreachable endpoint skips the run, because an unreadable probe is
nearly always a misconfiguration rather than an idle server.

It decides by reading the image, not by an exit code: run_oci.sh exits 0 both
when it upgraded and when the image was already current (the rc==10 branch
prints its message and falls through to the same exit), so an exit-code-driven
loop would have restarted the service every hour forever. The script compares
__version__ baked in the image before and after, and a change is the only thing
that triggers a restart. By default MAX_DEFERRALS=0, so a request in flight is
never interrupted: the new code waits and every run says so.

And it rolls back -- the pre-upgrade image is tagged first, and a service that
is not healthy within HEALTH_TIMEOUT gets the old tag back, another restart and
a non-zero exit that says whether it recovered.

Writing the example config surfaced a trap worth the test that now guards it:
the file is sourced, so `RESTART_CMD=systemctl --user restart coderai.service`
unquoted assigns "systemctl" and RUNS the rest -- restarting nothing, failing
the health check and rolling back a good upgrade. Everything in the example is
quoted and each run echoes the restart command it resolved.

Verified against the live server on zeiss: already-current does nothing; an
overlapping run is a no-op; a missing token skips; three in-flight engine
requests skip; and engines idle with a pod serving two still skips.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.88"
+__version__ = "0.2.89"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

docs/orchestrator-runpod-guide.md

@@ -996,6 +996,92 @@ For an orchestrator fleet, pulling the signed image is the auditable path and
 Restart after an upgrade (`systemctl restart coderai`) — the code is swapped in
 the image, not in the running process.
 
+### Upgrading unattended, hourly
+
+Two installs follow this repo: **zeiss** (nexlab) and the **Digesta production
+host in Aruba**. The Aruba one upgrades itself, hourly, from `production`, with
+nobody logged in. That is `packaging/linux/launcher/coderai-autoupgrade` plus the
+two units in `packaging/linux/systemd/`.
+
+```bash
+install -D -m755 packaging/linux/launcher/coderai-autoupgrade ~/.local/bin/
+install -D -m644 packaging/linux/systemd/coderai-autoupgrade.service \
+                 packaging/linux/systemd/coderai-autoupgrade.timer \
+                 ~/.config/systemd/user/
+install -D -m600 packaging/linux/systemd/autoupgrade.conf.example \
+                 ~/.config/coderai/autoupgrade.conf
+$EDITOR ~/.config/coderai/autoupgrade.conf      # see the two traps below
+systemctl --user daemon-reload
+systemctl --user enable --now coderai-autoupgrade.timer
+sudo loginctl enable-linger "$USER"             # NOT optional on a server
+```
+
+**Run it once by hand first** and read the output — `~/.local/bin/coderai-autoupgrade`.
+It prints the configuration it actually resolved, and whether the idle check is
+on. An unattended job that was never watched once is a guess.
+
+**It skips the upgrade entirely while CoderAI is serving.** Checked before the
+image is touched, not merely before the restart, so a busy host never reaches the
+state where new code waits in the image. Two probes, the higher count wins:
+
+| probe | what it sees |
+|---|---|
+| `<base>/metrics` → `coderai_engine_inflight` | every request the front has in flight, pod-served included |
+| `<base>/v1/runpod/status` → `summary.inflight` | what the rented pods themselves report |
+
+The RunPod one matters most on an orchestrator where **every** model is remote:
+the local engines are idle by definition, so the pods' own count is the only
+signal that work is happening. Interrupting a pod-served request loses the work
+*and* keeps paying for the pod that was doing it.
+
+The check **fails closed**. No token, wrong URL, unreachable endpoint → it skips
+and says so, because an unreadable probe almost always means a misconfiguration
+rather than an idle server. `REQUIRE_IDLE_CHECK="0"` opts out, and then it will
+interrupt live requests.
+
+By default a live request is **never** interrupted (`MAX_DEFERRALS="0"`): if the
+image was upgraded and a request arrives before the restart, the new code waits
+and every run says so. Set it above zero to allow a restart after that many
+hourly ticks.
+
+**It decides by reading the image, not by an exit code.** `coderai-docker
+--upgrade` exits `0` both when it upgraded and when the image was already
+current, so an exit-code-driven loop would restart the service every hour
+forever. The script reads `__version__` out of the image before and after; a
+change is the only thing that triggers a restart.
+
+**It rolls back.** The pre-upgrade image is tagged first (a tag, so it costs
+nothing). If the service is not healthy within `HEALTH_TIMEOUT` after the
+restart, the tag is put back, the service is restarted again, and the run exits
+non-zero with one line saying whether it recovered. A service left down says
+"Needs a human".
+
+Two traps worth knowing before they cost you an evening:
+
+- **The config file is sourced as shell, so quote anything with a space.**
+  `RESTART_CMD=systemctl --user restart coderai.service` assigns only
+  `systemctl` and then *runs* `--user restart coderai.service`. The restart then
+  restarts nothing, the health check fails, and a perfectly good upgrade gets
+  rolled back. Hence the quoting in the example, and hence the resolved
+  `restart=[…]` line in every run's log.
+- **`loginctl enable-linger`.** A systemd *user* timer only fires while the user
+  has a session. Without lingering it is silently inert after a reboot — the one
+  failure mode here that leaves no trace anywhere.
+
+`HEALTH_URL` is the only address to get right; `/metrics` and
+`/v1/runpod/status` are derived from it. On the Digesta host the container's
+8776 is published on 8777, so it is `http://127.0.0.1:8777/healthz`.
+
+State and history live in `${XDG_STATE_HOME:-~/.local/state}/coderai`:
+`autoupgrade.log` (trimmed, since this runs 24 times a day), `restart-pending`
+and `deferrals`. The journal has the per-run copy: `journalctl --user -u
+coderai-autoupgrade`.
+
+One thing to keep in mind: each real upgrade commits another layer onto the tag,
+so after many of them the image has drifted from anything in the registry. Pull
+the signed image occasionally to re-base it — `--upgrade` is for carrying fixes
+between releases, not a substitute for them.
+
 ## 10. Reference
 
 | | |
@@ -1007,4 +1093,5 @@ the image, not in the running process.
 | Remote execution / engines | `docs/remote-execution.md` |
 | Other capability images | `docs/install-from-packages.md` |
 | In-image upgrader | `packaging/linux/launcher/coderai-upgrade`, driven by `coderai-docker --upgrade` |
+| Unattended upgrade | `packaging/linux/launcher/coderai-autoupgrade` + `packaging/linux/systemd/` (hourly, skips while serving, rolls back) |
 | Host runner | `packaging/linux/run_oci.sh` (installed as `coderai-docker`) |

packaging/linux/launcher/coderai-autoupgrade

+#!/usr/bin/env bash
+# Unattended in-place upgrade for a CoderAI orchestrator, for a host that should
+# follow the `production` branch without someone logging in.
+#
+# Driven by a systemd timer (see packaging/linux/systemd/) or cron. One run:
+#
+#   1. take a lock, so two timers can never upgrade the same image at once;
+#   2. remember the version baked into the image, and tag that image as the
+#      rollback point;
+#   3. run the host runner's `--upgrade`;
+#   4. read the baked version again. CHANGED is the signal to restart -- not the
+#      runner's exit code, which reports 0 both when it upgraded and when the
+#      image was already current, so an exit-code-driven loop would restart the
+#      service on every tick;
+#   5. restart only if the service is idle, else defer to the next tick (the new
+#      code sits in the image until then, which is safe);
+#   6. verify health after the restart, and ROLL BACK to the remembered image if
+#      it does not come up.
+#
+# Exit 0: nothing to do, upgraded and healthy, or deliberately deferred.
+# Exit 1: the upgrade or the restart failed -- and then the message says whether
+#         a rollback happened, because that is the only thing an operator who
+#         reads one line needs to know.
+set -uo pipefail
+
+CONF="${CODERAI_AUTOUPGRADE_CONF:-${XDG_CONFIG_HOME:-$HOME/.config}/coderai/autoupgrade.conf}"
+# shellcheck disable=SC1090
+[ -f "$CONF" ] && . "$CONF"
+
+# Which container engine. A rootless podman host is the common case for this
+# script, so prefer podman when both are installed.
+if [ -z "${ENGINE:-}" ]; then
+    if command -v podman >/dev/null 2>&1; then ENGINE=podman
+    elif command -v docker >/dev/null 2>&1; then ENGINE=docker
+    else echo "autoupgrade: neither podman nor docker on PATH" >&2; exit 1; fi
+fi
+IMAGE_TAG="${IMAGE_TAG:-ghcr.io/nextime/coderai:latest}"
+UPGRADE_REF="${UPGRADE_REF:-production}"
+RUNNER="${RUNNER:-$(command -v coderai-docker || true)}"
+RESTART_CMD="${RESTART_CMD:-systemctl --user restart coderai.service}"
+HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:8777/healthz}"
+HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-300}"
+# The idle check. An upgrade is SKIPPED outright while CoderAI is serving
+# anything -- a request on a local engine or, the case that matters most here,
+# one being served by a rented RunPod pod, because interrupting that both loses
+# the work and keeps paying for the pod that was doing it.
+#
+# Both probes are derived from HEALTH_URL so there is one address to get right:
+#   <base>/metrics            -> coderai_engine_inflight, every request the front
+#                                has in flight, pod-served ones included
+#   <base>/v1/runpod/status   -> summary.inflight, the pods' own count
+# The higher of the two wins. They overlap; neither is trusted to be complete.
+ADMIN_TOKEN="${ADMIN_TOKEN:-}"
+_base="${HEALTH_URL%/healthz}"; _base="${_base%/}"
+METRICS_URL="${METRICS_URL:-$_base/metrics}"
+STATUS_URL="${STATUS_URL:-$_base/v1/runpod/status}"
+# Fail CLOSED: if neither probe can be read, assume it is serving and do nothing.
+# A silent upgrade that interrupts production is worse than a skipped one, and an
+# unreadable probe usually means a wrong token or URL, not an idle server.
+REQUIRE_IDLE_CHECK="${REQUIRE_IDLE_CHECK:-1}"
+# Once the image HAS been upgraded, the restart can be deferred this many ticks
+# before going ahead anyway. 0 (the default) means never interrupt a request:
+# the new code waits in the image and every run says so.
+MAX_DEFERRALS="${MAX_DEFERRALS:-0}"
+STATE_DIR="${STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/coderai}"
+LOG="${LOG:-$STATE_DIR/autoupgrade.log}"
+ROLLBACK_TAG="${ROLLBACK_TAG:-coderai-autoupgrade:rollback}"
+
+mkdir -p "$STATE_DIR" || { echo "autoupgrade: cannot create $STATE_DIR" >&2; exit 1; }
+mkdir -p "$(dirname "$LOG")" 2>/dev/null
+PENDING="$STATE_DIR/restart-pending"
+DEFERRALS="$STATE_DIR/deferrals"
+
+say(){ printf '%s %s\n' "$(date -Is)" "$*" | tee -a "$LOG"; }
+
+# Running hourly for years, this is the one file that grows without bound. Keep
+# the tail and nothing else: the journal has the per-run copy, this log exists so
+# the HISTORY of upgrades survives a journal rotation.
+LOG_MAX_LINES="${LOG_MAX_LINES:-5000}"
+if [ -f "$LOG" ] && [ "$(wc -l < "$LOG" 2>/dev/null || echo 0)" -gt "$LOG_MAX_LINES" ]; then
+    tail -n "$((LOG_MAX_LINES / 2))" "$LOG" > "$LOG.trim" 2>/dev/null \
+        && mv "$LOG.trim" "$LOG"
+fi
+
+[ -n "$RUNNER" ] || { say "FAIL: no coderai-docker on PATH (set RUNNER= in $CONF)"; exit 1; }
+[ -x "$RUNNER" ] || { say "FAIL: RUNNER '$RUNNER' is not executable"; exit 1; }
+
+# Echo the settings that were actually resolved, every run. The config file is
+# SOURCED, so an unquoted multi-word value silently keeps only its first word --
+# `RESTART_CMD=systemctl --user restart coderai.service` assigns "systemctl" and
+# runs the rest. That mangling is invisible until it restarts nothing, the health
+# check fails and a perfectly good upgrade gets rolled back. One line makes it
+# obvious instead.
+IDLE_CHECK=off; [ -n "$ADMIN_TOKEN" ] && IDLE_CHECK=on
+say "config: engine=$ENGINE image=$IMAGE_TAG ref=$UPGRADE_REF"
+say "config: restart=[$RESTART_CMD] health=$HEALTH_URL idle-check=$IDLE_CHECK"
+
+# ---------------------------------------------------------------------- the lock
+# flock on a dedicated fd, non-blocking: a run that overlaps the previous one is
+# a no-op, not a queue. The timer fires again soon enough.
+exec 9>"$STATE_DIR/autoupgrade.lock"
+if ! flock -n 9; then
+    say "another autoupgrade run holds the lock; nothing done"
+    exit 0
+fi
+
+baked_version(){
+    "$ENGINE" run --rm --entrypoint grep "$IMAGE_TAG" -h "__version__" \
+        /opt/coderai/app/codai/__init__.py 2>/dev/null \
+        | tr -d ' "' | cut -d= -f2 | head -1
+}
+
+_probe_metrics(){
+    # Sum of coderai_engine_inflight across engines. A request routed to a RunPod
+    # pod is still in flight here, so this covers remote serving as well as local.
+    curl -fsS --max-time 10 -H "Authorization: Bearer $ADMIN_TOKEN" "$METRICS_URL" 2>/dev/null \
+        | awk '/^coderai_engine_inflight[ {]/ { n += $NF; seen = 1 }
+               END { if (seen) printf "%d\n", n; else print "unknown" }'
+}
+
+_probe_runpod(){
+    # What the pods themselves report. Present even if the front's counter is not.
+    curl -fsS --max-time 10 -H "Authorization: Bearer $ADMIN_TOKEN" "$STATUS_URL" 2>/dev/null \
+        | python3 -c 'import json,sys
+try: print(int((json.load(sys.stdin).get("summary") or {}).get("inflight") or 0))
+except Exception: print("unknown")' 2>/dev/null
+}
+
+# Echoes "<count> <detail>", or "unknown <why>" when nothing could be read.
+inflight(){
+    local m r best detail
+    [ -n "$ADMIN_TOKEN" ] || { echo "unknown no ADMIN_TOKEN configured"; return; }
+    m="$(_probe_metrics)"; r="$(_probe_runpod)"
+    case "$m" in ''|unknown) m=unknown ;; *[!0-9]*) m=unknown ;; esac
+    case "$r" in ''|unknown) r=unknown ;; *[!0-9]*) r=unknown ;; esac
+    if [ "$m" = unknown ] && [ "$r" = unknown ]; then
+        echo "unknown neither $METRICS_URL nor $STATUS_URL could be read"
+        return
+    fi
+    best=0; detail=""
+    [ "$m" != unknown ] && { best="$m"; detail="engines=$m"; }
+    if [ "$r" != unknown ]; then
+        detail="${detail:+$detail }runpod=$r"
+        [ "$r" -gt "$best" ] && best="$r"
+    fi
+    echo "$best $detail"
+}
+
+# Serving anything, or we cannot tell? Then this run does nothing.
+guard_idle(){
+    local out count detail
+    out="$(inflight)"; count="${out%% *}"; detail="${out#* }"
+    if [ "$count" = unknown ]; then
+        if [ "$REQUIRE_IDLE_CHECK" = "1" ]; then
+            say "SKIP: cannot tell whether CoderAI is serving ($detail) — refusing to"
+            say "      upgrade blind. Fix the probe, or set REQUIRE_IDLE_CHECK=\"0\" to"
+            say "      upgrade without the check (it will interrupt live requests)."
+            return 1
+        fi
+        say "WARN: idle check unavailable ($detail) and REQUIRE_IDLE_CHECK=0 — continuing"
+        return 0
+    fi
+    if [ "$count" -gt 0 ]; then
+        say "SKIP: serving $count request(s) [$detail] — leaving the image alone"
+        return 1
+    fi
+    say "idle ($detail)"
+    return 0
+}
+
+healthy(){
+    local deadline=$(( $(date +%s) + HEALTH_TIMEOUT ))
+    while [ "$(date +%s)" -lt "$deadline" ]; do
+        if curl -fsS --max-time 10 "$HEALTH_URL" 2>/dev/null | grep -q '"ok"[[:space:]]*:[[:space:]]*true'; then
+            return 0
+        fi
+        sleep 10
+    done
+    return 1
+}
+
+restart_and_verify(){
+    say "restarting: $RESTART_CMD"
+    if ! sh -c "$RESTART_CMD" >>"$LOG" 2>&1; then
+        say "WARN: the restart command returned non-zero"
+    fi
+    if healthy; then return 0; fi
+    return 1
+}
+
+# ------------------------------------------------------------------- the upgrade
+BEFORE="$(baked_version)"
+[ -n "$BEFORE" ] || { say "FAIL: cannot read the version from $IMAGE_TAG (wrong tag? engine '$ENGINE'?)"; exit 1; }
+
+if [ -f "$PENDING" ]; then
+    # A previous run upgraded the image but could not restart. Do not upgrade
+    # again on top of that; land what is already there first.
+    say "a restart is still pending for $(cat "$PENDING" 2>/dev/null) — not upgrading further"
+else
+    # Nothing is touched while CoderAI is serving. Checked BEFORE the upgrade and
+    # not only before the restart, so a busy host never even reaches the state
+    # where new code sits in the image waiting for a window.
+    guard_idle || exit 0
+    say "image $IMAGE_TAG carries $BEFORE; checking '$UPGRADE_REF'"
+    # The rollback point is a TAG on the current image, so it costs nothing and
+    # survives the commit the upgrade makes onto $IMAGE_TAG.
+    "$ENGINE" tag "$IMAGE_TAG" "$ROLLBACK_TAG" >/dev/null 2>&1 \
+        || say "WARN: could not tag the rollback point; continuing without one"
+
+    CODERAI_UPGRADE_REF="$UPGRADE_REF" "$RUNNER" --upgrade --upgrade-ref "$UPGRADE_REF" \
+        "$IMAGE_TAG" >>"$LOG" 2>&1
+    rc=$?
+    [ "$rc" -eq 0 ] || { say "FAIL: the upgrade run exited $rc; image left as it was"; exit 1; }
+
+    AFTER="$(baked_version)"
+    if [ "$AFTER" = "$BEFORE" ]; then
+        say "already current at $BEFORE; nothing to restart"
+        rm -f "$DEFERRALS"
+        exit 0
+    fi
+    say "image upgraded $BEFORE -> $AFTER"
+    printf '%s -> %s\n' "$BEFORE" "$AFTER" > "$PENDING"
+fi
+
+# ------------------------------------------------------------- restart, or defer
+# The upgrade itself takes a minute, so ask again: a request may have arrived
+# while it ran, and the restart is the part that would kill it.
+N=$(cat "$DEFERRALS" 2>/dev/null || echo 0)
+OUT="$(inflight)"; BUSY="${OUT%% *}"; WHAT="${OUT#* }"
+if [ "$BUSY" = unknown ] || { [ "$BUSY" -gt 0 ] 2>/dev/null; }; then
+    if [ "$N" -lt "$MAX_DEFERRALS" ]; then
+        echo $((N + 1)) > "$DEFERRALS"
+        say "serving [$WHAT] — deferring the restart ($((N + 1))/$MAX_DEFERRALS)"
+        exit 0
+    fi
+    if [ "$MAX_DEFERRALS" -le 0 ]; then
+        echo $((N + 1)) > "$DEFERRALS"
+        say "serving [$WHAT] — the new code is in the image and waits for an idle"
+        say "      moment ($((N + 1)) run(s) so far). Set MAX_DEFERRALS>0 to restart anyway."
+        exit 0
+    fi
+    say "serving [$WHAT] but $N deferrals already — restarting anyway"
+fi
+
+if restart_and_verify; then
+    say "OK: running $(cat "$PENDING" 2>/dev/null | sed 's/.*-> //') and healthy"
+    rm -f "$PENDING" "$DEFERRALS"
+    exit 0
+fi
+
+# ------------------------------------------------------------------- the rollback
+say "FAIL: the service did not become healthy within ${HEALTH_TIMEOUT}s after the upgrade"
+if "$ENGINE" image inspect "$ROLLBACK_TAG" >/dev/null 2>&1; then
+    say "rolling back $IMAGE_TAG to the pre-upgrade image"
+    "$ENGINE" tag "$ROLLBACK_TAG" "$IMAGE_TAG" >/dev/null 2>&1
+    if restart_and_verify; then
+        say "FAIL: upgrade reverted — back on $BEFORE and healthy. The upgrade is NOT applied."
+        rm -f "$PENDING" "$DEFERRALS"
+        exit 1
+    fi
+    say "FAIL: the service is down and the rollback did not bring it back. Needs a human."
+    exit 1
+fi
+say "FAIL: no rollback image available; the service is down. Needs a human."
+exit 1

packaging/linux/systemd/autoupgrade.conf.example

+# CoderAI unattended upgrade — copy to ~/.config/coderai/autoupgrade.conf
+# Sourced by coderai-autoupgrade as shell, so it is `KEY=value`, no export needed.
+# QUOTE any value containing spaces: unquoted, `RESTART_CMD=systemctl --user
+# restart coderai.service` assigns only "systemctl" and then RUNS the rest as a
+# command. Every value below is a default made explicit; delete what you do not
+# change.
+
+# podman (preferred when installed) or docker.
+ENGINE="podman"
+
+# The tag the service actually runs. --upgrade COMMITS onto this tag, so it must
+# be the same one the quadlet/unit names, or the restart picks up the old code.
+IMAGE_TAG="ghcr.io/nextime/coderai:latest"
+
+# The branch to follow. `production` is the released line; `master` is ahead of it.
+UPGRADE_REF="production"
+
+# The host runner. Found on PATH when unset.
+#RUNNER="/usr/local/bin/coderai-docker"
+
+# How the service is restarted once the image carries new code. For a rootless
+# quadlet this is the generated unit; for a system install, `sudo systemctl
+# restart coderai`; on a host without systemd, `/etc/init.d/coderai restart`.
+RESTART_CMD="systemctl --user restart coderai.service"
+
+# Where to confirm the service came back. On the Digesta host the container's
+# 8776 is published on 8777. If this is wrong the script will roll back a
+# perfectly good upgrade, so check it once by hand.
+HEALTH_URL="http://127.0.0.1:8777/healthz"
+HEALTH_TIMEOUT="300"
+
+# Optional idle check: an admin-scoped API token lets the script ask how many
+# requests are in flight and defer the restart instead of interrupting them.
+# Without it the restart happens whenever the timer says so.
+#STATUS_URL="http://127.0.0.1:8777/v1/runpod/status"
+#ADMIN_TOKEN="cra-..."
+
+# After this many deferrals the restart happens anyway, so a fleet that is never
+# idle cannot pin an old version indefinitely.
+MAX_DEFERRALS="8"

packaging/linux/systemd/coderai-autoupgrade.service

+# Unattended in-place upgrade for a CoderAI orchestrator.
+#
+# A rootless-podman host runs this as a USER unit, next to the quadlet that runs
+# the orchestrator itself:
+#
+#   install -D -m755 packaging/linux/launcher/coderai-autoupgrade ~/.local/bin/
+#   install -D -m644 packaging/linux/systemd/coderai-autoupgrade.* \
+#       ~/.config/systemd/user/
+#   systemctl --user daemon-reload
+#   systemctl --user enable --now coderai-autoupgrade.timer
+#
+# A user timer only fires while the user has a session, so on a server:
+#   sudo loginctl enable-linger $USER
+# Without it the timer is silently inert after a reboot, which is the one
+# failure mode of this setup that leaves no trace anywhere.
+[Unit]
+Description=CoderAI unattended in-place upgrade (production branch)
+Documentation=file:/opt/coderai/app/docs/orchestrator-runpod-guide.md
+
+[Service]
+Type=oneshot
+ExecStart=%h/.local/bin/coderai-autoupgrade
+# A shallow clone plus a pip sync inside the container, then a restart and a
+# health wait. Generous: being killed half way through is worse than being slow.
+TimeoutStartSec=2700
+# The script already writes its own timestamped log; this keeps the journal copy.
+StandardOutput=journal
+StandardError=journal

packaging/linux/systemd/coderai-autoupgrade.timer

+# How often to look for a new production build. HOURLY: the Digesta production
+# host follows `production` closely, so a fix should land within the hour rather
+# than waiting for the night.
+#
+# An hourly tick is cheap when there is nothing to do -- the check is a shallow
+# clone of one branch and a version compare, and the script exits without
+# touching the service unless the baked version actually changed. A restart only
+# happens on a real upgrade, and only when the fleet is idle (or after
+# MAX_DEFERRALS hours of never being idle).
+#
+# RandomizedDelaySec spreads a fleet of orchestrators so they do not all clone
+# and restart on the same minute. Persistent=true runs a missed check once after
+# a reboot, so a host that was off does not wait for the next full hour.
+#
+# For a host that should only move in a maintenance window, replace OnCalendar
+# with e.g. `*-*-* 04:30:00` and keep everything else.
+[Unit]
+Description=Hourly check for a new CoderAI production build
+
+[Timer]
+# `hourly` is systemd's shorthand for `*-*-* *:00:00`.
+OnCalendar=hourly
+RandomizedDelaySec=5m
+Persistent=true
+AccuracySec=1m
+
+[Install]
+WantedBy=timers.target

tests/test_autoupgrade.py

+"""The unattended upgrade for a production orchestrator.
+
+Two production installs now follow this repo: zeiss (nexlab) and the Digesta
+host in Aruba. The Aruba one upgrades itself hourly from `production`, with no
+one logged in -- so the failure modes are the ones nobody is watching for:
+
+- restarting on every tick, because the host runner exits 0 BOTH when it
+  upgraded and when the image was already current (run_oci.sh: the rc==10 branch
+  prints "no upgrade applied" and then falls through to the same `exit 0`). The
+  script must therefore decide by reading the version baked into the image
+  before and after, never by the exit code.
+- upgrading while CoderAI is serving. A request being served by a rented RunPod
+  pod is the worst case: killing it loses the work AND keeps paying for the pod
+  that was doing it. The check covers local engines and pods, and fails CLOSED.
+- a mangled config. The file is SOURCED, so an unquoted
+  `RESTART_CMD=systemctl --user restart coderai.service` assigns "systemctl" and
+  runs the rest -- restarting nothing, failing the health check, and rolling back
+  a perfectly good upgrade. Caught in testing; hence the echoed config line.
+- a timer that never fires, because a user unit needs `loginctl enable-linger`.
+"""
+import pathlib
+import re
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+SH = (ROOT / "packaging/linux/launcher/coderai-autoupgrade").read_text()
+SERVICE = (ROOT / "packaging/linux/systemd/coderai-autoupgrade.service").read_text()
+TIMER = (ROOT / "packaging/linux/systemd/coderai-autoupgrade.timer").read_text()
+CONF = (ROOT / "packaging/linux/systemd/autoupgrade.conf.example").read_text()
+RUNNER = (ROOT / "packaging/linux/run_oci.sh").read_text()
+
+
+# ------------------------------------------------- why the exit code is not usable
+def test_the_runner_really_does_collapse_both_outcomes():
+    """The premise of the whole design. If this ever changes, the script could be
+    simplified -- and if it changes silently, this test says so."""
+    blk = RUNNER.split('if [[ "$UPGRADE" -eq 1 ]]; then')[1]
+    assert 'elif [[ "$rc" -eq 10 ]]; then' in blk
+    nothing = blk.split('elif [[ "$rc" -eq 10 ]]; then')[1].split("else")[0]
+    assert "no upgrade applied" in nothing
+    assert "exit" not in nothing, "a distinct exit code here would be observable"
+
+
+def test_the_decision_comes_from_the_image_not_the_exit_code():
+    assert "baked_version()" in SH
+    assert 'BEFORE="$(baked_version)"' in SH
+    assert 'AFTER="$(baked_version)"' in SH
+    assert 'if [ "$AFTER" = "$BEFORE" ]' in SH
+
+
+def test_nothing_to_do_means_no_restart():
+    tail = SH.split('if [ "$AFTER" = "$BEFORE" ]; then')[1].split("\n    fi")[0]
+    assert "already current" in tail
+    assert "exit 0" in tail
+    # The message may SAY "nothing to restart"; what must not appear is any way
+    # of actually restarting.
+    assert "restart_and_verify" not in tail
+    assert "$RESTART_CMD" not in tail
+
+
+# --------------------------------------------------------------- the idle guard
+def test_the_upgrade_is_gated_on_idle_not_only_the_restart():
+    """A busy host must never even reach the state where new code sits in the
+    image waiting for a window."""
+    body = SH.split("# ------------------------------------------------------------------- the upgrade")[1]
+    gate = body.index("guard_idle || exit 0")
+    upgrade = body.index('"$RUNNER" --upgrade')
+    assert gate < upgrade, "the idle check must come before the upgrade runs"
+
+
+def test_runpod_serving_counts_even_when_the_engines_are_idle():
+    """The case the requirement names: all models remote, so the only in-flight
+    work is on the pods."""
+    assert "_probe_runpod()" in SH
+    assert '"inflight"' in SH
+    blk = SH.split("inflight(){")[1].split("guard_idle()")[0]
+    assert '[ "$r" -gt "$best" ] && best="$r"' in blk, "the higher of the two must win"
+
+
+def test_local_serving_counts_too():
+    assert "coderai_engine_inflight" in SH
+
+
+def test_an_unreadable_probe_skips_rather_than_upgrades():
+    """Fail closed: an unreadable probe usually means a wrong token or URL, not
+    an idle server."""
+    blk = SH.split("guard_idle(){")[1].split("\n}")[0]
+    assert 'if [ "$count" = unknown ]' in blk
+    first = blk.split('if [ "$count" = unknown ]')[1]
+    assert "SKIP" in first and "return 1" in first
+
+
+def test_the_fail_closed_default_is_on():
+    assert 'REQUIRE_IDLE_CHECK="${REQUIRE_IDLE_CHECK:-1}"' in SH
+
+
+def test_a_missing_token_is_not_treated_as_idle():
+    blk = SH.split("inflight(){")[1].split("guard_idle()")[0]
+    assert 'echo "unknown no ADMIN_TOKEN configured"' in blk
+
+
+def test_idle_is_re_checked_immediately_before_the_restart():
+    """The upgrade takes a minute; a request can arrive while it runs, and the
+    restart is the part that would kill it."""
+    after = SH.split('printf \'%s -> %s\\n\' "$BEFORE" "$AFTER" > "$PENDING"')[1]
+    assert "inflight)" in after
+    assert after.index("inflight)") < after.index("restart_and_verify")
+
+
+def test_by_default_a_live_request_is_never_interrupted():
+    assert 'MAX_DEFERRALS="${MAX_DEFERRALS:-0}"' in SH
+    blk = SH.split('if [ "$MAX_DEFERRALS" -le 0 ]; then')[1].split("fi")[0]
+    assert "exit 0" in blk, "it must wait, not restart"
+
+
+# ------------------------------------------------------------- safety on failure
+def test_a_failed_upgrade_rolls_back():
+    assert "ROLLBACK_TAG" in SH
+    assert '"$ENGINE" tag "$ROLLBACK_TAG" "$IMAGE_TAG"' in SH
+
+
+def test_the_rollback_point_is_taken_before_the_upgrade():
+    body = SH.split("# ------------------------------------------------------------------- the upgrade")[1]
+    assert body.index('"$ENGINE" tag "$IMAGE_TAG" "$ROLLBACK_TAG"') < body.index('"$RUNNER" --upgrade')
+
+
+def test_health_is_verified_after_the_restart():
+    assert "healthy()" in SH
+    assert "HEALTH_TIMEOUT" in SH
+    blk = SH.split("restart_and_verify(){")[1].split("\n}")[0]
+    assert "if healthy; then return 0; fi" in blk
+
+
+def test_a_service_left_down_is_said_plainly_and_exits_nonzero():
+    tail = SH.split("# ------------------------------------------------------------------- the rollback")[1]
+    assert "Needs a human" in tail
+    assert "exit 1" in tail
+
+
+def test_concurrent_runs_cannot_overlap():
+    assert "flock -n 9" in SH
+    blk = SH.split("flock -n 9")[1].split("fi")[0]
+    assert "exit 0" in blk, "an overlapping run is a no-op, not a queue"
+
+
+# --------------------------------------------------------- the config-file trap
+def test_every_multi_word_value_in_the_example_is_quoted():
+    """Unquoted, a sourced `KEY=two words` assigns one word and runs the rest."""
+    for line in CONF.splitlines():
+        line = line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        key, _, val = line.partition("=")
+        if " " in val.strip():
+            assert val.strip().startswith('"') and val.strip().endswith('"'), line
+
+
+def test_the_example_warns_about_the_quoting_trap():
+    assert "QUOTE any value containing spaces" in CONF
+
+
+def test_the_resolved_config_is_logged_every_run():
+    """So a mangled value is visible in the log instead of causing a rollback."""
+    assert 'say "config: restart=[$RESTART_CMD]' in SH
+
+
+# ------------------------------------------------------------------ the schedule
+def test_the_timer_runs_hourly():
+    assert re.search(r"^OnCalendar=hourly$", TIMER, re.M)
+
+
+def test_a_missed_check_is_caught_up_after_a_reboot():
+    assert re.search(r"^Persistent=true$", TIMER, re.M)
+
+
+def test_a_fleet_does_not_all_restart_on_the_same_minute():
+    assert re.search(r"^RandomizedDelaySec=", TIMER, re.M)
+
+
+def test_the_unit_is_a_oneshot_with_room_to_finish():
+    assert re.search(r"^Type=oneshot$", SERVICE, re.M)
+    m = re.search(r"^TimeoutStartSec=(\d+)$", SERVICE, re.M)
+    assert m and int(m.group(1)) >= 1800, "a pip sync plus a health wait needs room"
+
+
+def test_the_linger_requirement_is_documented_where_it_bites():
+    """A user timer is silently inert after a reboot without it -- the one
+    failure mode of this setup that leaves no trace anywhere."""
+    assert "enable-linger" in SERVICE
+
+
+# --------------------------------------------------------------------- hygiene
+def test_an_hourly_job_does_not_grow_a_log_for_ever():
+    assert "LOG_MAX_LINES" in SH
+    assert "tail -n" in SH
+
+
+def test_the_token_is_never_written_to_the_log():
+    for line in SH.splitlines():
+        if "say " in line or "tee -a" in line:
+            assert "$ADMIN_TOKEN" not in line, line