packaging: hourly unattended upgrade that will not interrupt a request (v0.2.89)

Two production installs now follow this repo: zeiss (nexlab) and the Digesta
host in Aruba. The Aruba one should track `production` without anyone logging
in, so this adds coderai-autoupgrade plus a systemd user service and an hourly
timer.

It skips the upgrade outright while CoderAI is serving -- checked BEFORE the
image is touched, not merely before the restart, so a busy host never reaches
the state where new code waits in the image. Two probes and the higher count
wins: the front's coderai_engine_inflight (every request in flight, pod-served
ones included) and the pods' own summary.inflight from /v1/runpod/status. The
RunPod one carries the weight on an orchestrator where every model is remote:
the local engines are idle by definition, so the pods' count is the only sign
work is happening, and killing a pod-served request loses the work AND keeps
paying for the pod that was doing it. The check fails CLOSED -- no token, wrong
URL or an unreachable endpoint skips the run, because an unreadable probe is
nearly always a misconfiguration rather than an idle server.

It decides by reading the image, not by an exit code: run_oci.sh exits 0 both
when it upgraded and when the image was already current (the rc==10 branch
prints its message and falls through to the same exit), so an exit-code-driven
loop would have restarted the service every hour forever. The script compares
__version__ baked in the image before and after, and a change is the only thing
that triggers a restart. By default MAX_DEFERRALS=0, so a request in flight is
never interrupted: the new code waits and every run says so.

And it rolls back -- the pre-upgrade image is tagged first, and a service that
is not healthy within HEALTH_TIMEOUT gets the old tag back, another restart and
a non-zero exit that says whether it recovered.

Writing the example config surfaced a trap worth the test that now guards it:
the file is sourced, so `RESTART_CMD=systemctl --user restart coderai.service`
unquoted assigns "systemctl" and RUNS the rest -- restarting nothing, failing
the health check and rolling back a good upgrade. Everything in the example is
quoted and each run echoes the restart command it resolved.

Verified against the live server on zeiss: already-current does nothing; an
overlapping run is a no-op; a missing token skips; three in-flight engine
requests skip; and engines idle with a pod serving two still skips.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.88"
+__version__ = "0.2.89"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

docs/orchestrator-runpod-guide.md

@@ -996,6 +996,92 @@ For an orchestrator fleet, pulling the signed image is the auditable path and
 Restart after an upgrade (`systemctl restart coderai`) — the code is swapped in
 the image, not in the running process.
 
+### Upgrading unattended, hourly
+
+Two installs follow this repo: **zeiss** (nexlab) and the **Digesta production
+host in Aruba**. The Aruba one upgrades itself, hourly, from `production`, with
+nobody logged in. That is `packaging/linux/launcher/coderai-autoupgrade` plus the
+two units in `packaging/linux/systemd/`.
+
+```bash
+install -D -m755 packaging/linux/launcher/coderai-autoupgrade ~/.local/bin/
+install -D -m644 packaging/linux/systemd/coderai-autoupgrade.service \
+                 packaging/linux/systemd/coderai-autoupgrade.timer \
+                 ~/.config/systemd/user/
+install -D -m600 packaging/linux/systemd/autoupgrade.conf.example \
+                 ~/.config/coderai/autoupgrade.conf
+$EDITOR ~/.config/coderai/autoupgrade.conf      # see the two traps below
+systemctl --user daemon-reload
+systemctl --user enable --now coderai-autoupgrade.timer
+sudo loginctl enable-linger "$USER"             # NOT optional on a server
+```
+
+**Run it once by hand first** and read the output — `~/.local/bin/coderai-autoupgrade`.
+It prints the configuration it actually resolved, and whether the idle check is
+on. An unattended job that was never watched once is a guess.
+
+**It skips the upgrade entirely while CoderAI is serving.** Checked before the
+image is touched, not merely before the restart, so a busy host never reaches the
+state where new code waits in the image. Two probes, the higher count wins:
+
+| probe | what it sees |
+|---|---|
+| `<base>/metrics` → `coderai_engine_inflight` | every request the front has in flight, pod-served included |
+| `<base>/v1/runpod/status` → `summary.inflight` | what the rented pods themselves report |
+
+The RunPod one matters most on an orchestrator where **every** model is remote:
+the local engines are idle by definition, so the pods' own count is the only
+signal that work is happening. Interrupting a pod-served request loses the work
+*and* keeps paying for the pod that was doing it.
+
+The check **fails closed**. No token, wrong URL, unreachable endpoint → it skips
+and says so, because an unreadable probe almost always means a misconfiguration
+rather than an idle server. `REQUIRE_IDLE_CHECK="0"` opts out, and then it will
+interrupt live requests.
+
+By default a live request is **never** interrupted (`MAX_DEFERRALS="0"`): if the
+image was upgraded and a request arrives before the restart, the new code waits
+and every run says so. Set it above zero to allow a restart after that many
+hourly ticks.
+
+**It decides by reading the image, not by an exit code.** `coderai-docker
+--upgrade` exits `0` both when it upgraded and when the image was already
+current, so an exit-code-driven loop would restart the service every hour
+forever. The script reads `__version__` out of the image before and after; a
+change is the only thing that triggers a restart.
+
+**It rolls back.** The pre-upgrade image is tagged first (a tag, so it costs
+nothing). If the service is not healthy within `HEALTH_TIMEOUT` after the
+restart, the tag is put back, the service is restarted again, and the run exits
+non-zero with one line saying whether it recovered. A service left down says
+"Needs a human".
+
+Two traps worth knowing before they cost you an evening:
+
+- **The config file is sourced as shell, so quote anything with a space.**
+  `RESTART_CMD=systemctl --user restart coderai.service` assigns only
+  `systemctl` and then *runs* `--user restart coderai.service`. The restart then
+  restarts nothing, the health check fails, and a perfectly good upgrade gets
+  rolled back. Hence the quoting in the example, and hence the resolved
+  `restart=[…]` line in every run's log.
+- **`loginctl enable-linger`.** A systemd *user* timer only fires while the user
+  has a session. Without lingering it is silently inert after a reboot — the one
+  failure mode here that leaves no trace anywhere.
+
+`HEALTH_URL` is the only address to get right; `/metrics` and
+`/v1/runpod/status` are derived from it. On the Digesta host the container's
+8776 is published on 8777, so it is `http://127.0.0.1:8777/healthz`.
+
+State and history live in `${XDG_STATE_HOME:-~/.local/state}/coderai`:
+`autoupgrade.log` (trimmed, since this runs 24 times a day), `restart-pending`
+and `deferrals`. The journal has the per-run copy: `journalctl --user -u
+coderai-autoupgrade`.
+
+One thing to keep in mind: each real upgrade commits another layer onto the tag,
+so after many of them the image has drifted from anything in the registry. Pull
+the signed image occasionally to re-base it — `--upgrade` is for carrying fixes
+between releases, not a substitute for them.
+
 ## 10. Reference
 
 | | |
@@ -1007,4 +1093,5 @@ the image, not in the running process.
 | Remote execution / engines | `docs/remote-execution.md` |
 | Other capability images | `docs/install-from-packages.md` |
 | In-image upgrader | `packaging/linux/launcher/coderai-upgrade`, driven by `coderai-docker --upgrade` |
+| Unattended upgrade | `packaging/linux/launcher/coderai-autoupgrade` + `packaging/linux/systemd/` (hourly, skips while serving, rolls back) |
 | Host runner | `packaging/linux/run_oci.sh` (installed as `coderai-docker`) |

packaging/linux/systemd/autoupgrade.conf.example

+# CoderAI unattended upgrade — copy to ~/.config/coderai/autoupgrade.conf
+# Sourced by coderai-autoupgrade as shell, so it is `KEY=value`, no export needed.
+# QUOTE any value containing spaces: unquoted, `RESTART_CMD=systemctl --user
+# restart coderai.service` assigns only "systemctl" and then RUNS the rest as a
+# command. Every value below is a default made explicit; delete what you do not
+# change.
+
+# podman (preferred when installed) or docker.
+ENGINE="podman"
+
+# The tag the service actually runs. --upgrade COMMITS onto this tag, so it must
+# be the same one the quadlet/unit names, or the restart picks up the old code.
+IMAGE_TAG="ghcr.io/nextime/coderai:latest"
+
+# The branch to follow. `production` is the released line; `master` is ahead of it.
+UPGRADE_REF="production"
+
+# The host runner. Found on PATH when unset.
+#RUNNER="/usr/local/bin/coderai-docker"
+
+# How the service is restarted once the image carries new code. For a rootless
+# quadlet this is the generated unit; for a system install, `sudo systemctl
+# restart coderai`; on a host without systemd, `/etc/init.d/coderai restart`.
+RESTART_CMD="systemctl --user restart coderai.service"
+
+# Where to confirm the service came back. On the Digesta host the container's
+# 8776 is published on 8777. If this is wrong the script will roll back a
+# perfectly good upgrade, so check it once by hand.
+HEALTH_URL="http://127.0.0.1:8777/healthz"
+HEALTH_TIMEOUT="300"
+
+# Optional idle check: an admin-scoped API token lets the script ask how many
+# requests are in flight and defer the restart instead of interrupting them.
+# Without it the restart happens whenever the timer says so.
+#STATUS_URL="http://127.0.0.1:8777/v1/runpod/status"
+#ADMIN_TOKEN="cra-..."
+
+# After this many deferrals the restart happens anyway, so a fleet that is never
+# idle cannot pin an old version indefinitely.
+MAX_DEFERRALS="8"

packaging/linux/systemd/coderai-autoupgrade.service

+# Unattended in-place upgrade for a CoderAI orchestrator.
+#
+# A rootless-podman host runs this as a USER unit, next to the quadlet that runs
+# the orchestrator itself:
+#
+#   install -D -m755 packaging/linux/launcher/coderai-autoupgrade ~/.local/bin/
+#   install -D -m644 packaging/linux/systemd/coderai-autoupgrade.* \
+#       ~/.config/systemd/user/
+#   systemctl --user daemon-reload
+#   systemctl --user enable --now coderai-autoupgrade.timer
+#
+# A user timer only fires while the user has a session, so on a server:
+#   sudo loginctl enable-linger $USER
+# Without it the timer is silently inert after a reboot, which is the one
+# failure mode of this setup that leaves no trace anywhere.
+[Unit]
+Description=CoderAI unattended in-place upgrade (production branch)
+Documentation=file:/opt/coderai/app/docs/orchestrator-runpod-guide.md
+
+[Service]
+Type=oneshot
+ExecStart=%h/.local/bin/coderai-autoupgrade
+# A shallow clone plus a pip sync inside the container, then a restart and a
+# health wait. Generous: being killed half way through is worse than being slow.
+TimeoutStartSec=2700
+# The script already writes its own timestamped log; this keeps the journal copy.
+StandardOutput=journal
+StandardError=journal

packaging/linux/systemd/coderai-autoupgrade.timer

+# How often to look for a new production build. HOURLY: the Digesta production
+# host follows `production` closely, so a fix should land within the hour rather
+# than waiting for the night.
+#
+# An hourly tick is cheap when there is nothing to do -- the check is a shallow
+# clone of one branch and a version compare, and the script exits without
+# touching the service unless the baked version actually changed. A restart only
+# happens on a real upgrade, and only when the fleet is idle (or after
+# MAX_DEFERRALS hours of never being idle).
+#
+# RandomizedDelaySec spreads a fleet of orchestrators so they do not all clone
+# and restart on the same minute. Persistent=true runs a missed check once after
+# a reboot, so a host that was off does not wait for the next full hour.
+#
+# For a host that should only move in a maintenance window, replace OnCalendar
+# with e.g. `*-*-* 04:30:00` and keep everything else.
+[Unit]
+Description=Hourly check for a new CoderAI production build
+
+[Timer]
+# `hourly` is systemd's shorthand for `*-*-* *:00:00`.
+OnCalendar=hourly
+RandomizedDelaySec=5m
+Persistent=true
+AccuracySec=1m
+
+[Install]
+WantedBy=timers.target

tests/test_autoupgrade.py

+"""The unattended upgrade for a production orchestrator.
+
+Two production installs now follow this repo: zeiss (nexlab) and the Digesta
+host in Aruba. The Aruba one upgrades itself hourly from `production`, with no
+one logged in -- so the failure modes are the ones nobody is watching for:
+
+- restarting on every tick, because the host runner exits 0 BOTH when it
+  upgraded and when the image was already current (run_oci.sh: the rc==10 branch
+  prints "no upgrade applied" and then falls through to the same `exit 0`). The
+  script must therefore decide by reading the version baked into the image
+  before and after, never by the exit code.
+- upgrading while CoderAI is serving. A request being served by a rented RunPod
+  pod is the worst case: killing it loses the work AND keeps paying for the pod
+  that was doing it. The check covers local engines and pods, and fails CLOSED.
+- a mangled config. The file is SOURCED, so an unquoted
+  `RESTART_CMD=systemctl --user restart coderai.service` assigns "systemctl" and
+  runs the rest -- restarting nothing, failing the health check, and rolling back
+  a perfectly good upgrade. Caught in testing; hence the echoed config line.
+- a timer that never fires, because a user unit needs `loginctl enable-linger`.
+"""
+import pathlib
+import re
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+SH = (ROOT / "packaging/linux/launcher/coderai-autoupgrade").read_text()
+SERVICE = (ROOT / "packaging/linux/systemd/coderai-autoupgrade.service").read_text()
+TIMER = (ROOT / "packaging/linux/systemd/coderai-autoupgrade.timer").read_text()
+CONF = (ROOT / "packaging/linux/systemd/autoupgrade.conf.example").read_text()
+RUNNER = (ROOT / "packaging/linux/run_oci.sh").read_text()
+
+
+# ------------------------------------------------- why the exit code is not usable
+def test_the_runner_really_does_collapse_both_outcomes():
+    """The premise of the whole design. If this ever changes, the script could be
+    simplified -- and if it changes silently, this test says so."""
+    blk = RUNNER.split('if [[ "$UPGRADE" -eq 1 ]]; then')[1]
+    assert 'elif [[ "$rc" -eq 10 ]]; then' in blk
+    nothing = blk.split('elif [[ "$rc" -eq 10 ]]; then')[1].split("else")[0]
+    assert "no upgrade applied" in nothing
+    assert "exit" not in nothing, "a distinct exit code here would be observable"
+
+
+def test_the_decision_comes_from_the_image_not_the_exit_code():
+    assert "baked_version()" in SH
+    assert 'BEFORE="$(baked_version)"' in SH
+    assert 'AFTER="$(baked_version)"' in SH
+    assert 'if [ "$AFTER" = "$BEFORE" ]' in SH
+
+
+def test_nothing_to_do_means_no_restart():
+    tail = SH.split('if [ "$AFTER" = "$BEFORE" ]; then')[1].split("\n    fi")[0]
+    assert "already current" in tail
+    assert "exit 0" in tail
+    # The message may SAY "nothing to restart"; what must not appear is any way
+    # of actually restarting.
+    assert "restart_and_verify" not in tail
+    assert "$RESTART_CMD" not in tail
+
+
+# --------------------------------------------------------------- the idle guard
+def test_the_upgrade_is_gated_on_idle_not_only_the_restart():
+    """A busy host must never even reach the state where new code sits in the
+    image waiting for a window."""
+    body = SH.split("# ------------------------------------------------------------------- the upgrade")[1]
+    gate = body.index("guard_idle || exit 0")
+    upgrade = body.index('"$RUNNER" --upgrade')
+    assert gate < upgrade, "the idle check must come before the upgrade runs"
+
+
+def test_runpod_serving_counts_even_when_the_engines_are_idle():
+    """The case the requirement names: all models remote, so the only in-flight
+    work is on the pods."""
+    assert "_probe_runpod()" in SH
+    assert '"inflight"' in SH
+    blk = SH.split("inflight(){")[1].split("guard_idle()")[0]
+    assert '[ "$r" -gt "$best" ] && best="$r"' in blk, "the higher of the two must win"
+
+
+def test_local_serving_counts_too():
+    assert "coderai_engine_inflight" in SH
+
+
+def test_an_unreadable_probe_skips_rather_than_upgrades():
+    """Fail closed: an unreadable probe usually means a wrong token or URL, not
+    an idle server."""
+    blk = SH.split("guard_idle(){")[1].split("\n}")[0]
+    assert 'if [ "$count" = unknown ]' in blk
+    first = blk.split('if [ "$count" = unknown ]')[1]
+    assert "SKIP" in first and "return 1" in first
+
+
+def test_the_fail_closed_default_is_on():
+    assert 'REQUIRE_IDLE_CHECK="${REQUIRE_IDLE_CHECK:-1}"' in SH
+
+
+def test_a_missing_token_is_not_treated_as_idle():
+    blk = SH.split("inflight(){")[1].split("guard_idle()")[0]
+    assert 'echo "unknown no ADMIN_TOKEN configured"' in blk
+
+
+def test_idle_is_re_checked_immediately_before_the_restart():
+    """The upgrade takes a minute; a request can arrive while it runs, and the
+    restart is the part that would kill it."""
+    after = SH.split('printf \'%s -> %s\\n\' "$BEFORE" "$AFTER" > "$PENDING"')[1]
+    assert "inflight)" in after
+    assert after.index("inflight)") < after.index("restart_and_verify")
+
+
+def test_by_default_a_live_request_is_never_interrupted():
+    assert 'MAX_DEFERRALS="${MAX_DEFERRALS:-0}"' in SH
+    blk = SH.split('if [ "$MAX_DEFERRALS" -le 0 ]; then')[1].split("fi")[0]
+    assert "exit 0" in blk, "it must wait, not restart"
+
+
+# ------------------------------------------------------------- safety on failure
+def test_a_failed_upgrade_rolls_back():
+    assert "ROLLBACK_TAG" in SH
+    assert '"$ENGINE" tag "$ROLLBACK_TAG" "$IMAGE_TAG"' in SH
+
+
+def test_the_rollback_point_is_taken_before_the_upgrade():
+    body = SH.split("# ------------------------------------------------------------------- the upgrade")[1]
+    assert body.index('"$ENGINE" tag "$IMAGE_TAG" "$ROLLBACK_TAG"') < body.index('"$RUNNER" --upgrade')
+
+
+def test_health_is_verified_after_the_restart():
+    assert "healthy()" in SH
+    assert "HEALTH_TIMEOUT" in SH
+    blk = SH.split("restart_and_verify(){")[1].split("\n}")[0]
+    assert "if healthy; then return 0; fi" in blk
+
+
+def test_a_service_left_down_is_said_plainly_and_exits_nonzero():
+    tail = SH.split("# ------------------------------------------------------------------- the rollback")[1]
+    assert "Needs a human" in tail
+    assert "exit 1" in tail
+
+
+def test_concurrent_runs_cannot_overlap():
+    assert "flock -n 9" in SH
+    blk = SH.split("flock -n 9")[1].split("fi")[0]
+    assert "exit 0" in blk, "an overlapping run is a no-op, not a queue"
+
+
+# --------------------------------------------------------- the config-file trap
+def test_every_multi_word_value_in_the_example_is_quoted():
+    """Unquoted, a sourced `KEY=two words` assigns one word and runs the rest."""
+    for line in CONF.splitlines():
+        line = line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        key, _, val = line.partition("=")
+        if " " in val.strip():
+            assert val.strip().startswith('"') and val.strip().endswith('"'), line
+
+
+def test_the_example_warns_about_the_quoting_trap():
+    assert "QUOTE any value containing spaces" in CONF
+
+
+def test_the_resolved_config_is_logged_every_run():
+    """So a mangled value is visible in the log instead of causing a rollback."""
+    assert 'say "config: restart=[$RESTART_CMD]' in SH
+
+
+# ------------------------------------------------------------------ the schedule
+def test_the_timer_runs_hourly():
+    assert re.search(r"^OnCalendar=hourly$", TIMER, re.M)
+
+
+def test_a_missed_check_is_caught_up_after_a_reboot():
+    assert re.search(r"^Persistent=true$", TIMER, re.M)
+
+
+def test_a_fleet_does_not_all_restart_on_the_same_minute():
+    assert re.search(r"^RandomizedDelaySec=", TIMER, re.M)
+
+
+def test_the_unit_is_a_oneshot_with_room_to_finish():
+    assert re.search(r"^Type=oneshot$", SERVICE, re.M)
+    m = re.search(r"^TimeoutStartSec=(\d+)$", SERVICE, re.M)
+    assert m and int(m.group(1)) >= 1800, "a pip sync plus a health wait needs room"
+
+
+def test_the_linger_requirement_is_documented_where_it_bites():
+    """A user timer is silently inert after a reboot without it -- the one
+    failure mode of this setup that leaves no trace anywhere."""
+    assert "enable-linger" in SERVICE
+
+
+# --------------------------------------------------------------------- hygiene
+def test_an_hourly_job_does_not_grow_a_log_for_ever():
+    assert "LOG_MAX_LINES" in SH
+    assert "tail -n" in SH
+
+
+def test_the_token_is_never_written_to_the_log():
+    for line in SH.splitlines():
+        if "say " in line or "tee -a" in line:
+            assert "$ADMIN_TOKEN" not in line, line