autoupgrade: asking for the upgrade NOW was an undocumented file edit

The unattended policy has two states: wait for an idle moment, and — on a host
that is never idle — book a forced upgrade for the next midnight. Neither is an
operator saying "take it now". There was no way to say that at all. The only
lever was the booking file:

    echo 1 > ~/.local/state/coderai/force-upgrade-at

which works by accident (epoch 1 is in the past, so the next tick fires), is
written nowhere, and is NOT self-clearing when the run finds nothing to upgrade.
Observed on a production orchestrator: a `1` left from one forced release turned
every hourly tick for the next day into "a FORCED upgrade is booked", silently
disarming the idle check that protects live requests. Nobody asked for that; it
was the residue of a one-off.

So the three things an operator actually wants are options now:

  --force          upgrade and restart on THIS run, idle or not, and leave no
                   booking behind. It sets FORCED before the upgrade block, so
                   it also lands an upgrade already sitting in the image behind
                   deferrals — that host is precisely who types this.
  --force-at WHEN  book one (anything date -d takes). No second timer: the
                   hourly tick at or after WHEN takes it.
  --cancel-force   drop a booking, whether an operator made it or the script
                   armed it after MAX_DEFERRALS busy ticks. The arming message
                   now names this instead of telling you to delete a file.

The two booking edits return before the runner check and before anything touches
a container engine, so they work on a host where the upgrade itself could not
run, and they are written to the same log as the runs they change — a schedule
change belongs in the history of the schedule.

--help exists, and an unknown option is a usage error rather than a silent
unattended run under a typo.

Also drops one install's name from a sentence in the guide that did not need it.

13 tests. Suite: 3105 passed, the same 13 known-stale failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

docs/orchestrator-runpod-guide.md

@@ -1255,6 +1255,24 @@ restart, the tag is put back, the service is restarted again, and the run exits
 non-zero with one line saying whether it recovered. A service left down says
 "Needs a human".
 
+**Taking an upgrade now, on purpose.** The policy above is for the unattended
+run. An operator who wants the upgrade this minute says so on the command line,
+and nothing has to be edited by hand:
+
+```bash
+coderai-autoupgrade --force                  # upgrade + restart on this run,
+                                             # busy or not; no booking left behind
+coderai-autoupgrade --force-at 'tomorrow 02:00'   # book one; the hourly timer fires it
+coderai-autoupgrade --cancel-force           # drop a booking, whoever made it
+```
+
+`--force` interrupts whatever is in flight — that is what it is for. It also
+reaches the *restart* step, so it lands an upgrade that has been sitting in the
+image behind deferrals. `--force-at` takes anything `date -d` understands
+(`'+2 hours'`, `'now'`, an absolute time) and needs no second timer: the ordinary
+hourly tick at or after that moment takes it. `--cancel-force` clears the booking
+whether an operator made it or the script armed it after a run of busy ticks.
+
 Two traps worth knowing before they cost you an evening:
 
 - **The config file is sourced as shell, so quote anything with a space.**
@@ -1268,8 +1286,9 @@ Two traps worth knowing before they cost you an evening:
   failure mode here that leaves no trace anywhere.
 
 `HEALTH_URL` is the only address to get right; `/metrics` and
-`/v1/runpod/status` are derived from it. On the Digesta host the container's
-8776 is published on 8777, so it is `http://127.0.0.1:8777/healthz`.
+`/v1/runpod/status` are derived from it. When the container's 8776 is published on another
+port, that is the one to name — `http://127.0.0.1:8777/healthz` for a host
+publishing it on 8777.
 
 State and history live in `${XDG_STATE_HOME:-~/.local/state}/coderai`:
 `autoupgrade.log` (trimmed, since this runs 24 times a day), `restart-pending`

packaging/linux/launcher/coderai-autoupgrade

@@ -26,12 +26,59 @@
 # and never book a forced restart. Anything above 0 is a promise that the upgrade
 # WILL land, at midnight at the latest.
 #
+# An operator overrides all of that with `--force` (take the upgrade on this run,
+# busy or not), `--force-at WHEN` (book one) or `--cancel-force`. See --help.
+#
 # Exit 0: nothing to do, upgraded and healthy, or deliberately deferred.
 # Exit 1: the upgrade or the restart failed -- and then the message says whether
 #         a rollback happened, because that is the only thing an operator who
 #         reads one line needs to know.
 set -uo pipefail
 
+# ------------------------------------------------------------------ the operator
+# Everything above describes the UNATTENDED policy: wait for an idle moment, and
+# book a forced upgrade only when the host never has one. An operator who wants
+# the upgrade NOW is not covered by that policy, and the only way to say so used
+# to be to write an epoch into the booking file by hand -- undocumented, easy to
+# get wrong, and it leaves a booking behind that forces every later run too.
+FORCE_NOW=0
+FORCE_AT_SPEC=""
+CANCEL_FORCE=0
+usage(){
+    cat <<'EOF'
+coderai-autoupgrade [options]
+
+Unattended in-place upgrade for a CoderAI orchestrator. With no options it
+follows the configured policy: upgrade when the host is idle, defer when it is
+busy. See the comments at the top of this script, and the conf file it sources.
+
+  -f, --force         upgrade and restart on THIS run, idle or not. Work in
+                      flight WILL be interrupted. Leaves no booking behind.
+      --force-at WHEN book a forced upgrade for WHEN (anything `date -d` takes,
+                      e.g. 'tomorrow 02:00', '+2 hours', 'now'). The ordinary
+                      timer fires it; nothing else has to be installed.
+      --cancel-force  cancel a booked forced upgrade, whether this script booked
+                      it after a run of busy ticks or an operator did.
+  -h, --help          this text.
+
+Exit 0: nothing to do, upgraded and healthy, or deliberately deferred.
+Exit 1: the upgrade or the restart failed (the message says whether it rolled
+        back).
+EOF
+}
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -f|--force)     FORCE_NOW=1 ;;
+        --force-at)     [ $# -ge 2 ] || { echo "autoupgrade: --force-at needs a time" >&2; exit 2; }
+                        FORCE_AT_SPEC="$2"; shift ;;
+        --force-at=*)   FORCE_AT_SPEC="${1#*=}" ;;
+        --cancel-force) CANCEL_FORCE=1 ;;
+        -h|--help)      usage; exit 0 ;;
+        *)              echo "autoupgrade: unknown option '$1' (try --help)" >&2; exit 2 ;;
+    esac
+    shift
+done
+
 CONF="${CODERAI_AUTOUPGRADE_CONF:-${XDG_CONFIG_HOME:-$HOME/.config}/coderai/autoupgrade.conf}"
 
 # Every knob below is `VAR="${VAR:-default}"`, which reads as "the environment may
@@ -117,6 +164,32 @@ if [ -f "$LOG" ] && [ "$(wc -l < "$LOG" 2>/dev/null || echo 0)" -gt "$LOG_MAX_LI
         && mv "$LOG.trim" "$LOG"
 fi
 
+# --cancel-force and --force-at only touch the booking and leave: they are how an
+# operator edits the schedule, not a run of it. Done here, where $FORCE_AT and the
+# log exist, so the change is recorded in the same log as the runs it affects.
+if [ "$CANCEL_FORCE" = 1 ]; then
+    if [ -f "$FORCE_AT" ]; then
+        rm -f "$FORCE_AT"; say "booked forced upgrade CANCELLED by --cancel-force"
+    else
+        say "no forced upgrade was booked; nothing to cancel"
+    fi
+    exit 0
+fi
+if [ -n "$FORCE_AT_SPEC" ]; then
+    _at="$(date -d "$FORCE_AT_SPEC" +%s 2>/dev/null || true)"
+    case "${_at:-}" in
+        ''|*[!0-9]*) say "FAIL: --force-at '$FORCE_AT_SPEC' is not a time this host's date(1) understands"; exit 1 ;;
+    esac
+    echo "$_at" > "$FORCE_AT"
+    say "FORCED upgrade booked for $(date -d "@$_at" '+%F %T' 2>/dev/null || echo "$_at") by --force-at"
+    say "      The next timer tick at or after that moment upgrades and restarts"
+    say "      with work in flight. Cancel it with --cancel-force."
+    exit 0
+fi
+if [ "$FORCE_NOW" = 1 ]; then
+    say "--force: this run ignores the idle check and the deferral policy"
+fi
+
 [ -n "$RUNNER" ] || { say "FAIL: no coderai-docker on PATH (set RUNNER= in $CONF)"; exit 1; }
 [ -x "$RUNNER" ] || { say "FAIL: RUNNER '$RUNNER' is not executable"; exit 1; }
 
@@ -139,6 +212,7 @@ if [ -f "$FORCE_AT" ]; then
     say "config: a FORCED upgrade is booked for $(date -d "@$_at" '+%F %T' 2>/dev/null || echo "$_at")"
 fi
 
+
 # ---------------------------------------------------------------------- the lock
 # flock on a dedicated fd, non-blocking: a run that overlaps the previous one is
 # a no-op, not a queue. The timer fires again soon enough.
@@ -259,7 +333,7 @@ arm_midnight_force(){
     say "NEVER IDLE for $n run(s) — booking a FORCED upgrade for $(date -d "@$at" '+%F %T' 2>/dev/null || echo "$at")."
     say "      It will restart the service even with work in flight, because an"
     say "      install that is busy every hour otherwise never upgrades at all."
-    say "      Cancel it by deleting $FORCE_AT."
+    say "      Cancel it with: coderai-autoupgrade --cancel-force"
 }
 
 # Is a booked forced upgrade due? Clears the booking either way it is used, so a
@@ -340,6 +414,11 @@ restart_and_verify(){
 BEFORE="$(baked_version)"
 [ -n "$BEFORE" ] || { say "FAIL: cannot read the version from $IMAGE_TAG (wrong tag? engine '$ENGINE'?)"; exit 1; }
 
+# --force reaches the restart step too. A host whose upgrade has been sitting in
+# the image for days behind a deferral is exactly the case an operator types
+# --force for, and that path skips the block below entirely.
+FORCED="$FORCE_NOW"
+
 if [ -f "$PENDING" ]; then
     # A previous run upgraded the image but could not restart. Do not upgrade
     # again on top of that; land what is already there first.
@@ -348,7 +427,13 @@ else
     # Nothing is touched while CoderAI is serving. Checked BEFORE the upgrade and
     # not only before the restart, so a busy host never even reaches the state
     # where new code sits in the image waiting for a window.
-    if force_due; then
+    if [ "$FORCE_NOW" = 1 ]; then
+        # Asked for on the command line. Still clear any booking: the operator
+        # is taking the upgrade now, so a deadline made for the same upgrade has
+        # nothing left to fire for.
+        rm -f "$FORCE_AT"
+        say "FORCED UPGRADE (--force) — proceeding with work possibly in flight."
+    elif force_due; then
         FORCED=1
         say "FORCED UPGRADE (booked because this host was never idle) — proceeding"
         say "      with work possibly in flight. Requests being served WILL be"

tests/test_autoupgrade_force_options.py

+"""An operator has to be able to say "upgrade now", without editing state files.
+
+The unattended policy waits for an idle moment and, on a host that never has
+one, books a forced upgrade for the next midnight. Neither branch is an operator
+asking for the upgrade NOW. Before these options the only way to ask was to
+write an epoch into $STATE_DIR/force-upgrade-at by hand: undocumented, easy to
+get wrong, and it leaves a booking behind that forces every LATER run too --
+which is exactly what happened on a production orchestrator, where a hand-made
+`1` turned every hourly tick into a forced one for a day.
+
+These tests drive the real script. --help, --force-at and --cancel-force all
+return before the runner check and before anything touches a container engine,
+so they are safe to run anywhere.
+"""
+
+import os
+import pathlib
+import subprocess
+import time
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+SCRIPT = ROOT / "packaging/linux/launcher/coderai-autoupgrade"
+
+
+def _run(state_dir, *args):
+    env = dict(os.environ)
+    env["CODERAI_AUTOUPGRADE_CONF"] = str(state_dir / "no-such.conf")
+    env["STATE_DIR"] = str(state_dir)
+    return subprocess.run([str(SCRIPT), *args], capture_output=True, text=True,
+                          env=env, timeout=60)
+
+
+def _booking(state_dir):
+    f = state_dir / "force-upgrade-at"
+    return int(f.read_text().strip()) if f.exists() else None
+
+
+# ------------------------------------------------------------------- help #
+def test_help_lists_every_force_option():
+    r = subprocess.run([str(SCRIPT), "--help"], capture_output=True, text=True,
+                       timeout=60)
+    assert r.returncode == 0, r.stderr
+    for flag in ("--force", "--force-at", "--cancel-force"):
+        assert flag in r.stdout, flag
+
+
+def test_an_unknown_option_is_refused_rather_than_ignored():
+    """Silently ignoring it would run the UNATTENDED policy under a typo."""
+    r = subprocess.run([str(SCRIPT), "--forse"], capture_output=True, text=True,
+                       timeout=60)
+    assert r.returncode == 2
+    assert "--forse" in r.stderr
+
+
+# ---------------------------------------------------------------- booking #
+def test_force_at_books_the_time_it_was_given(tmp_path):
+    r = _run(tmp_path, "--force-at", "tomorrow 02:00")
+    assert r.returncode == 0, r.stderr
+    at = _booking(tmp_path)
+    assert at is not None
+    lt = time.localtime(at)
+    assert (lt.tm_hour, lt.tm_min) == (2, 0), time.ctime(at)
+    assert time.time() < at <= time.time() + 86400 + 60
+
+
+def test_force_at_takes_a_relative_time(tmp_path):
+    r = _run(tmp_path, "--force-at", "+2 hours")
+    assert r.returncode == 0, r.stderr
+    at = _booking(tmp_path)
+    assert abs(at - (time.time() + 7200)) < 120
+
+
+def test_force_at_refuses_a_time_it_cannot_parse(tmp_path):
+    """Writing garbage would book a deadline nothing can read -- and force_due
+    silently DROPS an unparsable booking, so the upgrade would never land and
+    nothing would say why."""
+    r = _run(tmp_path, "--force-at", "two tuesdays hence")
+    assert r.returncode == 1
+    assert _booking(tmp_path) is None
+    assert "not a time" in r.stdout
+
+
+def test_force_at_needs_an_argument(tmp_path):
+    r = _run(tmp_path, "--force-at")
+    assert r.returncode == 2
+    assert _booking(tmp_path) is None
+
+
+def test_force_at_accepts_the_equals_form(tmp_path):
+    assert _run(tmp_path, "--force-at=+1 hour").returncode == 0
+    assert _booking(tmp_path) is not None
+
+
+# ----------------------------------------------------------------- cancel #
+def test_cancel_force_removes_a_booking(tmp_path):
+    _run(tmp_path, "--force-at", "+3 hours")
+    assert _booking(tmp_path) is not None
+    r = _run(tmp_path, "--cancel-force")
+    assert r.returncode == 0, r.stderr
+    assert _booking(tmp_path) is None
+    assert "CANCELLED" in r.stdout
+
+
+def test_cancel_force_on_a_host_with_no_booking_is_not_an_error(tmp_path):
+    """An operator clearing a booking they are not sure exists must not get a
+    failure -- a timer wrapper would report it as a broken upgrade."""
+    r = _run(tmp_path, "--cancel-force")
+    assert r.returncode == 0
+    assert "nothing to cancel" in r.stdout
+
+
+def test_cancel_force_clears_a_booking_this_script_made(tmp_path):
+    """The midnight booking and an operator's booking are the same file, so one
+    command cancels either."""
+    (tmp_path / "force-upgrade-at").write_text(str(int(time.time()) + 600))
+    assert _run(tmp_path, "--cancel-force").returncode == 0
+    assert _booking(tmp_path) is None
+
+
+# ---------------------------------------------- the booking edits are logged #
+def test_a_schedule_change_is_written_to_the_upgrade_log(tmp_path):
+    """The log is the history of what this host did. A booking made or cancelled
+    by hand belongs in the same place as the runs it changes."""
+    _run(tmp_path, "--force-at", "+4 hours")
+    _run(tmp_path, "--cancel-force")
+    log = (tmp_path / "autoupgrade.log").read_text()
+    assert "booked" in log
+    assert "CANCELLED" in log
+
+
+# --------------------------------------------------- what --force wires into #
+def test_force_sets_the_same_flag_the_booking_sets():
+    """--force must reach the RESTART step too, not only the upgrade step. A
+    host whose upgrade has sat in the image behind deferrals is the case the
+    operator types --force for, and that path skips the upgrade block entirely.
+    """
+    src = SCRIPT.read_text()
+    assert 'FORCED="$FORCE_NOW"' in src
+
+
+def test_force_leaves_no_booking_behind():
+    """Taking the upgrade now settles any deadline made for the same upgrade;
+    leaving it would force the next run as well, with nobody having asked."""
+    src = SCRIPT.read_text()
+    i = src.index('FORCED UPGRADE (--force)')
+    # the clear sits just above the announcement, inside the same branch
+    assert 'rm -f "$FORCE_AT"' in src[i - 400:i]

tests/test_autoupgrade_midnight_force.py

@@ -83,8 +83,7 @@ def test_arming_says_when_and_how_to_cancel(tmp_path):
     r = _harness(tmp_path, 'arm_midnight_force 3')
     out = r.stdout
     assert "FORCED upgrade" in out
-    assert "Cancel it by deleting" in out
-    assert "force-upgrade-at" in out
+    assert "--cancel-force" in out
     assert "restart the service even with work in flight" in out