Two production installs now follow this repo: zeiss (nexlab) and the Digesta
host in Aruba. The Aruba one should track `production` without anyone logging
in, so this adds coderai-autoupgrade plus a systemd user service and an hourly
timer.

It skips the upgrade outright while CoderAI is serving -- checked BEFORE the
image is touched, not merely before the restart, so a busy host never reaches
the state where new code waits in the image. Two probes and the higher count
wins: the front's coderai_engine_inflight (every request in flight, pod-served
ones included) and the pods' own summary.inflight from /v1/runpod/status. The
RunPod one carries the weight on an orchestrator where every model is remote:
the local engines are idle by definition, so the pods' count is the only sign
work is happening, and killing a pod-served request loses the work AND keeps
paying for the pod that was doing it. The check fails CLOSED -- no token, wrong
URL or an unreachable endpoint skips the run, because an unreadable probe is
nearly always a misconfiguration rather than an idle server.

It decides by reading the image, not by an exit code: run_oci.sh exits 0 both
when it upgraded and when the image was already current (the rc==10 branch
prints its message and falls through to the same exit), so an exit-code-driven
loop would have restarted the service every hour forever. The script compares
__version__ baked in the image before and after, and a change is the only thing
that triggers a restart. By default MAX_DEFERRALS=0, so a request in flight is
never interrupted: the new code waits and every run says so.

And it rolls back -- the pre-upgrade image is tagged first, and a service that
is not healthy within HEALTH_TIMEOUT gets the old tag back, another restart and
a non-zero exit that says whether it recovered.

Writing the example config surfaced a trap worth the test that now guards it:
the file is sourced, so `RESTART_CMD=systemctl --user restart coderai.service`
unquoted assigns "systemctl" and RUNS the rest -- restarting nothing, failing
the health check and rolling back a good upgrade. Everything in the example is
quoted and each run echoes the restart command it resolved.

Verified against the live server on zeiss: already-current does nothing; an
overlapping run is a no-op; a missing token skips; three in-flight engine
requests skip; and engines idle with a pod serving two still skips.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>