zeiss came back from this morning's hard reset with no container at all, and the
hourly upgrade then skipped every tick — so the host sat on old code and, worse,
stayed DOWN, with nothing in the loop that would ever have recovered it. Both
idle probes are HTTP, so an absent server is indistinguishable from a wrong
token: inflight() said "unknown" and the fail-closed guard refused to act. The
Digesta host, whose container was up, upgraded itself fourteen seconds after the
push; zeiss needed hands.

container_state() answers the one question the probes cannot — is there a server
at all — by name, then by any running container of the image being upgraded.
Only a POSITIVE "not there" counts as idle: an engine that cannot be asked
(daemon down, wrong socket) stays unknown and still fails closed, because the
point is to recognise an absent server, not to assume one. Checked before the
token, since a host with no server is in the same position as one with no token.
Both installs run their engine's ps as the service user and name the container
`coderai`, so the default matches; CONTAINER_NAME covers an install that does not.

The second half: every knob reads `VAR="${VAR:-default}"`, which looks like the
environment may override it, but the conf is sourced first and assigns
unconditionally — so `MAX_DEFERRALS=1 coderai-autoupgrade` did nothing at all,
with no line in the log to say why. I hit that landing this morning's restart and
had to source the conf from a wrapper to get round it. The environment is now
snapshotted before the conf and restored after, for that run only, and a test
checks the snapshot list against every knob the script resolves so the next one
added cannot quietly fall out. require-idle and max-deferrals are also logged
now: "it skipped again" and "it was told to skip" used to look identical.

The conf example claimed that without a token "the restart happens whenever the
timer says so". It is the opposite — the check fails closed, so a running server
is then never upgraded. Corrected, and REQUIRE_IDLE_CHECK documented next to it.

tests/test_autoupgrade_recovers_a_down_server.py RUNS the script against a stubbed
engine, runner, restart command and health file: the env restore is eval-based and
the container check branches on an exit code, neither of which reading the source
can confirm. 12 new tests, 56 across the three autoupgrade files.

No version bump: nothing in the image changed, and bumping would send both
production installs through an image upgrade and a restart for a host-side script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019sRvvg5WSdLgcFUM96pmE4