When zeiss ran out of memory this morning the OOM killer picked coderai-nvidia.
Not because it was at fault — a runaway test process was — but because oom_score
is essentially "how much memory does this task hold", and an engine with a model
loaded is always near the top of that list. The leak carried on; the thing that
was serving died; the box needed a hard reset.

Linux offers one lever, /proc/<pid>/oom_score_adj, and two rules that decide
where it can be pulled from. Both were verified on the two production hosts
rather than assumed:

  raising is free, lowering needs CAP_SYS_RESOURCE — and a docker container does
  not have it (tested: in-container `echo -500` is denied, `echo 200` succeeds).
  So the protected base has to come from the engine daemon, via
  --oom-score-adj on the run, and everything inside can only step back UP from it.

  rootless podman cannot protect at all: a negative value is clamped to 0. 0 is
  still worth having on the Digesta host, where coderai currently sits at +200 —
  inherited from systemd's user manager, which makes it a PREFERRED victim with a
  kernel score of 800 against zeiss's 670.

So: the container gets -500 (CODERAI_OOM_SCORE_ADJ, 0 disables), and inside it
codai/util/oom.py states the order as three tiers relative to that base —
engines +100, workers +300, training +500. Relative, because the base differs by
host and a tier should keep its distance whatever it turns out to be. Engines are
marked in the existing _engine_preexec hook, which covers the isolated model
workers too: they are spawned by the engine, not the front, so they inherit it.
Training is marked from the parent after Popen — a preexec_fn would run Python
between fork and exec in a threaded server, which is the documented way to
deadlock.

The ordering is the point. Protecting everything equally would just move the
bullet to an innocent bystander; this way the kernel sheds a training run (hours,
restartable) before an engine (loses its in-flight work, and the front notices
and respawns it) before the front (loses everything, including the ability to
respawn anything).

Every helper can only ever make a process MORE killable: nudge() refuses a
negative delta rather than attempting a privileged write, because a process that
believes it is protected and is not is worse than one that knows it is exposed.
All of it is best-effort — a process that cannot adjust its score still serves.

16 tests, including that the number reaches the kernel and moves the score, that
a child can be marked from the parent, and that the engine mark sits inside a try
(it runs between fork and exec, so anything that raises there fails the spawn).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019sRvvg5WSdLgcFUM96pmE4