-
Stefy Lanza (nextime / spora ) authored
The timer's own comment promised "a restart only happens on a real upgrade, and only when the fleet is idle (or after MAX_DEFERRALS hours of never being idle)". The second half could not happen: guard_idle || exit 0 # a busy host exits HERE ...upgrade... N=$(cat "$DEFERRALS" ...) # the counter only ever incremented HERE So on a busy host the script left before the image was even pulled, the counter the force depends on never advanced, and MAX_DEFERRALS was unreachable. Measured on the Digesta orchestrator 2026-10-14: 24 skips in one day, `deferrals` still 0, two versions behind, and 0.3.68 landed only because the host happened to be idle at 17:01. Now a busy skip COUNTS, and at MAX_DEFERRALS the script books a forced upgrade for the next midnight. The 00:00 tick of the existing hourly timer fires it, so there is no second unit to install and nothing stranded by a reboot. Midnight because it is the cheapest hour to interrupt a business-hours workload and a time an operator can predict without reading the script. Four things that each had a way to go wrong: * booked ONCE, not re-booked on every busy tick -- otherwise the deadline walks forward forever and never arrives; * a corrupt deadline file is discarded, not obeyed: garbage must not read as "the time has passed" and restart the service, nor sit there blocking a fresh booking; * a forced run does not defer at the restart step either -- deferring there would put it back where the booking was made; * MAX_DEFERRALS=0 keeps its old meaning (wait for an idle moment, however long, never force), so an install that did not opt in is unchanged. The booking is announced in the config banner of every later run and cancelled by deleting one file, because a forced restart that nobody saw coming is worse than a late upgrade. 16 tests, and the ones that matter are behavioural rather than source-matching: they extract the real arm_midnight_force and force_due from the script, run them under bash against a temp state dir, and assert the timestamp really is local midnight and that force_due flips at the right moment. A test that restates the logic only proves it agrees with itself. Suite at the 25 pre-existing failures. NOT installed on Aruba: that copy is byte-identical to this one and runs hourly, so overwriting it mid-run would be editing a running script. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
7a6234ce