oom: the server should be the last thing the kernel kills, not the first

When zeiss ran out of memory this morning the OOM killer picked coderai-nvidia.
Not because it was at fault — a runaway test process was — but because oom_score
is essentially "how much memory does this task hold", and an engine with a model
loaded is always near the top of that list. The leak carried on; the thing that
was serving died; the box needed a hard reset.

Linux offers one lever, /proc/<pid>/oom_score_adj, and two rules that decide
where it can be pulled from. Both were verified on the two production hosts
rather than assumed:

  raising is free, lowering needs CAP_SYS_RESOURCE — and a docker container does
  not have it (tested: in-container `echo -500` is denied, `echo 200` succeeds).
  So the protected base has to come from the engine daemon, via
  --oom-score-adj on the run, and everything inside can only step back UP from it.

  rootless podman cannot protect at all: a negative value is clamped to 0. 0 is
  still worth having on the Digesta host, where coderai currently sits at +200 —
  inherited from systemd's user manager, which makes it a PREFERRED victim with a
  kernel score of 800 against zeiss's 670.

So: the container gets -500 (CODERAI_OOM_SCORE_ADJ, 0 disables), and inside it
codai/util/oom.py states the order as three tiers relative to that base —
engines +100, workers +300, training +500. Relative, because the base differs by
host and a tier should keep its distance whatever it turns out to be. Engines are
marked in the existing _engine_preexec hook, which covers the isolated model
workers too: they are spawned by the engine, not the front, so they inherit it.
Training is marked from the parent after Popen — a preexec_fn would run Python
between fork and exec in a threaded server, which is the documented way to
deadlock.

The ordering is the point. Protecting everything equally would just move the
bullet to an innocent bystander; this way the kernel sheds a training run (hours,
restartable) before an engine (loses its in-flight work, and the front notices
and respawns it) before the front (loses everything, including the ability to
respawn anything).

Every helper can only ever make a process MORE killable: nudge() refuses a
negative delta rather than attempting a privileged write, because a process that
believes it is protected and is not is worse than one that knows it is exposed.
All of it is best-effort — a process that cannot adjust its score still serves.

16 tests, including that the number reaches the kernel and moves the score, that
a child can be marked from the parent, and that the engine mark sits inside a try
(it runs between fork and exec, so anything that raises there fails the spawn).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019sRvvg5WSdLgcFUM96pmE4

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.5"
+__version__ = "0.3.6"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/loras.py

@@ -1846,6 +1846,17 @@ def _train_via_overlay(arch, req, base_path, images, instance_prompt,
     proc = subprocess.Popen([py, script, "--job", job_path],
                             stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                             text=True, bufsize=1)
+    # The most expendable thing on the box, and one of the fattest: a training run
+    # holds tens of GB for hours, and losing it costs a restart of one job rather
+    # than every request in flight. Tell the OOM killer to come here first. Set
+    # from the parent (raising another process's score needs no privilege) rather
+    # than in a preexec_fn, which runs Python between fork and exec in a threaded
+    # server. Best-effort: a training job that cannot be marked still trains.
+    try:
+        from codai.util import oom
+        oom.mark("training", proc.pid)
+    except Exception:
+        pass
 
     def _pump():
         for line in proc.stdout:

codai/frontproxy/engine_supervisor.py

@@ -54,6 +54,17 @@ def _engine_preexec():
         ctypes.CDLL("libc.so.6", use_errno=True).prctl(1, 9, 0, 0, 0)
     except Exception:
         pass
+    # Step this engine, and everything it goes on to spawn, one tier ABOVE the
+    # front in the OOM killer's order. An engine holding a model is always near the
+    # top of the kernel's badness list — that is how coderai-nvidia came to be the
+    # victim of a leak in an unrelated process — and the front is the one process
+    # whose death loses everything, including the ability to respawn an engine.
+    # Raising is unprivileged; see codai/util/oom.py for why it can only go this way.
+    try:
+        from codai.util import oom
+        oom.mark("engine")
+    except Exception:
+        pass
 
 
 def _port_is_free(port: int, host: str = "127.0.0.1") -> bool:

codai/util/__init__.py

+"""Small host-level helpers that are not tied to one subsystem."""

codai/util/oom.py

+"""Who the kernel should kill first when the host runs out of memory.
+
+On 2026-10-10 a runaway test process on zeiss took the host to 46 GB RSS plus 20 GB
+of swap. The kernel's OOM killer then picked `coderai-nvidia` — not because it was
+at fault, but because `oom_score` is essentially "how much memory does this task
+hold", and an engine holding a model is always near the top of that list. The thing
+that leaked survived; the thing that was serving died, and the box needed a hard
+reset.
+
+So the ordering has to be stated explicitly. Linux offers exactly one lever,
+``/proc/<pid>/oom_score_adj`` (-1000..1000), added to the score the kernel computes:
+
+    front / supervisor   protected   the one process whose death loses everything
+    engines              neutral     losing one loses its in-flight work; the front
+                                     notices and respawns it
+    model workers,       expendable  minutes of work at most, restartable, and the
+    training                         usual suspects for holding tens of GB
+
+Two kernel rules shape what is possible, and both were verified on the two
+production hosts rather than assumed:
+
+* **Raising is free, lowering is privileged.** Any process may make itself a more
+  attractive victim; moving the other way needs CAP_SYS_RESOURCE. A container
+  started with ``--oom-score-adj`` inherits the protected value from the engine
+  daemon, and in-container processes can then only step UP from it — which is
+  exactly the direction this module needs.
+* **Rootless cannot protect at all.** Under rootless podman a negative value is
+  clamped to 0. 0 still beats the +200 a user service inherits from systemd's user
+  manager, so the flag is worth passing there too, but the real protection on such
+  a host comes from making the expendable processes expendable.
+
+Everything here is best-effort and silent on failure: a process that cannot adjust
+its score must still start and serve.
+"""
+
+from __future__ import annotations
+
+import os
+
+# Steps, relative to whatever the container was given. Deliberately coarse: the
+# point is a strict order between the three tiers, not a tuned number. Engines sit
+# one step above the front so that shedding an engine (recoverable) always comes
+# before shedding the front (not recoverable).
+TIERS = {
+    "front": 0,
+    "engine": 100,
+    "worker": 300,
+    "training": 500,
+}
+
+ADJ_MIN, ADJ_MAX = -1000, 1000
+
+
+def _path(pid=None) -> str:
+    return f"/proc/{'self' if pid is None else int(pid)}/oom_score_adj"
+
+
+def current_adj(pid=None):
+    """This (or another) process's oom_score_adj, or None where /proc has none."""
+    try:
+        with open(_path(pid)) as handle:
+            return int(handle.read().strip())
+    except Exception:
+        return None
+
+
+def current_score(pid=None):
+    """The kernel's own badness figure, for logging. Higher = killed sooner."""
+    try:
+        with open(f"/proc/{'self' if pid is None else int(pid)}/oom_score") as handle:
+            return int(handle.read().strip())
+    except Exception:
+        return None
+
+
+def nudge(delta: int, pid=None) -> bool:
+    """Make a process MORE killable by ``delta``, relative to where it is now.
+
+    Relative, because the base is set outside this process (``--oom-score-adj`` on
+    the container) and a tier should keep its distance from the front whatever that
+    base turns out to be: -500 on a docker host, 0 on a rootless one.
+
+    Never lowers. A negative ``delta`` would need a privilege the server does not
+    have and should not want, and silently failing to protect something is worse
+    than not trying: returns False instead.
+    """
+    if delta <= 0:
+        return False
+    base = current_adj(pid)
+    if base is None:
+        return False
+    want = max(ADJ_MIN, min(ADJ_MAX, base + int(delta)))
+    if want <= base:
+        return False                      # already at the ceiling
+    try:
+        with open(_path(pid), "w") as handle:
+            handle.write(f"{want}\n")
+    except Exception:
+        return False
+    return current_adj(pid) == want
+
+
+def mark(tier: str, pid=None) -> bool:
+    """Put a process in one of the TIERS, relative to the front's base."""
+    return nudge(TIERS.get(tier, 0), pid)
+
+
+def preexec(tier: str):
+    """A ``preexec_fn`` that marks the CHILD, after fork and before exec.
+
+    Done in the child so the parent's own score is untouched, and so the value is
+    already in place for whatever the child goes on to spawn: workers inherit it
+    from the engine that started them, which is how one call here covers a tree.
+    """
+    delta = TIERS.get(tier, 0)
+
+    def _apply():
+        if delta > 0:
+            try:
+                nudge(delta)
+            except Exception:
+                pass
+
+    return _apply
+
+
+def describe(pid=None) -> str:
+    """One line for a log: where this process sits and how exposed it is."""
+    adj, score = current_adj(pid), current_score(pid)
+    if adj is None:
+        return "oom: /proc/self/oom_score_adj unavailable"
+    where = "protected" if adj < 0 else "neutral" if adj == 0 else "expendable"
+    return f"oom: adj={adj:+d} ({where}), kernel score={score}"

packaging/linux/run_oci.sh

@@ -224,6 +224,16 @@ Bring your bare-metal config + data into the container with --map (dir OR file):
   # Video editor (config file + the media_dir/output_dir it points at):
   ...  --map /host/video_editor.config.json:/cache/video_editor/video_editor.config.json \
        --map /host/coderai_media --map /host/video_editor_output
+
+Environment:
+  CODERAI_OOM_SCORE_ADJ  How hard the kernel should avoid killing this container
+                         when the HOST runs out of memory (-1000..1000, default
+                         -500; 0 leaves the kernel's own ordering alone). A server
+                         holding a model is one of the fattest tasks on the box, so
+                         the OOM killer reaches for it first even when the leak is
+                         somewhere else. Rootless podman clamps a negative value
+                         to 0. Inside the container, engines and training runs step
+                         back UP from this base so they are shed first.
 EOF
 }
 
@@ -474,6 +484,25 @@ if [[ "$HOST_NETWORK" == "1" ]]; then
 else
   args=(run --rm --name "$NAME" --ipc=host -p "$PUBLISH" -e CODERAI_HOST=0.0.0.0 -e CODERAI_PORT=8776)
 fi
+
+# Who the kernel kills when the HOST runs out of memory. oom_score is essentially
+# "how much memory does this task hold", so an orchestrator with a model loaded is
+# always near the top of that list and gets reached for first — even when the leak
+# is somewhere else entirely. That is what happened on 2026-10-10: a runaway test
+# process on the host filled swap, and the kernel killed coderai-nvidia, which was
+# serving at the time, while the process at fault carried on.
+#
+# -500 of the -1000..1000 range is a strong preference, not immunity. It has to be
+# set HERE because the container engine runs as root and has CAP_SYS_RESOURCE,
+# while nothing inside the container does: in-container processes can only step UP
+# from this base, which is exactly what the engine and worker tiers do
+# (codai/util/oom.py). Rootless podman clamps a negative value to 0 — still an
+# improvement on the +200 a user service inherits from systemd's user manager.
+# Set CODERAI_OOM_SCORE_ADJ=0 to leave the kernel's own ordering alone.
+OOM_SCORE_ADJ="${CODERAI_OOM_SCORE_ADJ:--500}"
+if [[ "$OOM_SCORE_ADJ" != "0" ]]; then
+  args+=(--oom-score-adj "$OOM_SCORE_ADJ")
+fi
 # Forward a HuggingFace token from the host env so the engines authenticate to the
 # HF Hub (higher rate limits + gated models) instead of sending unauthenticated
 # requests. huggingface_hub auto-reads these; no-op when unset.

tests/test_oom_priority.py

+"""Who the kernel kills when the host runs out of memory.
+
+On 2026-10-10 a runaway test process took zeiss to 46 GB RSS plus 20 GB of swap.
+The OOM killer then picked `coderai-nvidia` — not because it was at fault, but
+because `oom_score` is essentially "how much memory does this task hold", and an
+engine with a model loaded is always near the top of that list. The leak survived;
+the server died; the box needed a hard reset.
+
+The fix states the order explicitly, in three places that have to agree:
+
+* the container gets a protected base (`--oom-score-adj`, set by the engine daemon
+  because nothing inside the container has CAP_SYS_RESOURCE);
+* engines step one tier ABOVE the front, so shedding an engine (which the front
+  notices and respawns) always comes before shedding the front (which loses
+  everything, including the ability to respawn anything);
+* a training run — hours long, tens of GB, restartable — goes to the top.
+
+The direction matters and is not symmetric: raising is unprivileged, lowering is
+not. Every helper here can only ever make a process MORE killable, so a missing
+privilege can never silently turn protection into exposure.
+"""
+
+import pathlib
+import re
+import subprocess
+import sys
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.util import oom
+
+RUNNER = (ROOT / "packaging/linux/run_oci.sh").read_text()
+SUPERVISOR = (ROOT / "codai/frontproxy/engine_supervisor.py").read_text()
+LORAS = (ROOT / "codai/api/loras.py").read_text()
+
+HAS_PROC = pathlib.Path("/proc/self/oom_score_adj").exists()
+needs_proc = pytest.mark.skipif(not HAS_PROC, reason="no /proc/self/oom_score_adj")
+
+
+# --------------------------------------------------------------------- the order
+def test_the_front_is_the_most_protected_tier():
+    """It is the one process whose death loses everything."""
+    assert oom.TIERS["front"] == 0
+    assert oom.TIERS["engine"] > oom.TIERS["front"]
+    assert oom.TIERS["worker"] > oom.TIERS["engine"]
+    assert oom.TIERS["training"] >= oom.TIERS["worker"]
+
+
+def test_the_tiers_stay_inside_the_kernels_range():
+    """A base of -500 plus the largest tier must still be a legal value."""
+    assert oom.ADJ_MIN == -1000 and oom.ADJ_MAX == 1000
+    assert -500 + max(oom.TIERS.values()) <= oom.ADJ_MAX
+
+
+# ------------------------------------------------------------- only ever upwards
+@needs_proc
+def test_a_tier_is_relative_to_whatever_base_the_container_was_given():
+    """The base is -500 under docker and 0 under rootless podman, so a tier has to
+    keep its DISTANCE from the front rather than name an absolute number."""
+    script = """
+import sys; sys.path.insert(0, %r)
+from codai.util import oom
+print(oom.current_adj()); oom.mark("engine"); print(oom.current_adj())
+""" % str(ROOT)
+    out = subprocess.run([sys.executable, "-c", script], capture_output=True,
+                         text=True, timeout=60).stdout.split()
+    base, after = int(out[0]), int(out[1])
+    assert after - base == oom.TIERS["engine"]
+
+
+@needs_proc
+def test_lowering_is_refused_rather_than_attempted():
+    """Lowering needs CAP_SYS_RESOURCE. Asking and failing would leave a process
+    that believes it is protected and is not."""
+    assert oom.nudge(-100) is False
+    assert oom.nudge(0) is False
+
+
+@needs_proc
+def test_a_nudge_that_would_pass_the_ceiling_is_clamped_not_an_error():
+    script = """
+import sys; sys.path.insert(0, %r)
+from codai.util import oom
+oom.nudge(900); oom.nudge(900); print(oom.current_adj())
+""" % str(ROOT)
+    out = subprocess.run([sys.executable, "-c", script], capture_output=True,
+                         text=True, timeout=60).stdout.strip()
+    assert int(out) == oom.ADJ_MAX
+
+
+@needs_proc
+def test_the_tier_actually_reaches_the_kernel():
+    """Not just bookkeeping: the number has to land in /proc and move the score."""
+    script = """
+import sys; sys.path.insert(0, %r)
+from codai.util import oom
+before = oom.current_score()
+oom.mark("training")
+print(oom.current_adj(), before, oom.current_score())
+""" % str(ROOT)
+    adj, before, after = subprocess.run(
+        [sys.executable, "-c", script], capture_output=True, text=True,
+        timeout=60).stdout.split()
+    assert int(adj) == oom.TIERS["training"]
+    assert int(after) > int(before), "a raised adj must raise the kernel's badness"
+
+
+@needs_proc
+def test_a_child_can_be_marked_from_the_parent():
+    """How the training job is marked: a preexec_fn would run Python between fork
+    and exec in a threaded server, which is the documented way to deadlock."""
+    child = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(30)"])
+    try:
+        assert oom.mark("worker", child.pid) is True
+        assert oom.current_adj(child.pid) == oom.TIERS["worker"]
+    finally:
+        child.kill()
+        child.wait(timeout=10)
+
+
+def test_an_unreadable_proc_is_not_fatal():
+    """Not Linux, or a hardened /proc: the server still has to start."""
+    assert oom.current_adj(pid=2**30) is None
+    assert oom.nudge(100, pid=2**30) is False
+    assert "unavailable" in oom.describe(pid=2**30)
+
+
+@needs_proc
+def test_describe_says_which_side_of_neutral_a_process_is_on():
+    assert "neutral" in oom.describe() or "expendable" in oom.describe()
+
+
+# ----------------------------------------------------------- where it is applied
+def test_the_container_gets_a_protected_base():
+    """It has to be set by the ENGINE: the container has no CAP_SYS_RESOURCE, so
+    nothing inside it could ever lower its own score."""
+    assert 'OOM_SCORE_ADJ="${CODERAI_OOM_SCORE_ADJ:--500}"' in RUNNER
+    assert 'args+=(--oom-score-adj "$OOM_SCORE_ADJ")' in RUNNER
+
+
+def test_the_protection_can_be_turned_off():
+    block = RUNNER.split('OOM_SCORE_ADJ="${CODERAI_OOM_SCORE_ADJ:--500}"')[1][:200]
+    assert 'if [[ "$OOM_SCORE_ADJ" != "0" ]]' in block
+
+
+def test_the_flag_is_on_the_run_not_the_upgrade():
+    """The upgrade runs a throwaway container; protecting it would be meaningless."""
+    assert "up_args+=(--oom-score-adj" not in RUNNER
+
+
+def test_engines_are_marked_one_tier_above_the_front():
+    """In the preexec hook, so the engine AND everything it spawns inherit it —
+    the isolated model workers come from the engine, not from the front."""
+    hook = SUPERVISOR.split("def _engine_preexec():")[1].split("\ndef ")[0]
+    assert 'oom.mark("engine")' in hook
+
+
+def test_the_engine_mark_cannot_break_a_spawn():
+    """This runs between fork and exec. Anything that raises here fails the spawn,
+    so the whole fleet would stop starting over a /proc write."""
+    hook = SUPERVISOR.split("def _engine_preexec():")[1].split("\ndef ")[0]
+    before, after = hook.split('oom.mark("engine")')
+    assert "try:" in before and "except" not in before.rsplit("try:", 1)[1]
+    assert after.lstrip().startswith("except Exception:")
+
+
+def test_training_is_the_first_choice_of_victim():
+    blk = LORAS.split('subprocess.Popen([py, script, "--job", job_path]')[1][:900]
+    assert 'oom.mark("training", proc.pid)' in blk
+
+
+def test_the_incident_is_recorded_where_the_number_is_chosen():
+    """-500 is a judgement call; the next person deserves to know what it is for."""
+    assert re.search(r"oom_score is essentially", RUNNER)
+    assert "coderai-nvidia" in RUNNER