a refused connection is backpressure, not a dead pod (v0.4.37)

`discard()` terminated the pod, on the reasoning that a connection can
only be refused by a pod that no longer exists. A pod whose listen
backlog is full refuses connections while being perfectly alive and busy
-- and refuses them on /healthz too, so the "is anything still there?"
probe added to protect against exactly this agrees it has vanished.

Observed 2026-10-19, mid-campaign, ~58 concurrent connections per pod:

    08:36:13  pod 2n69g0vt47h0hl terminated (connection refused)
    08:36:30  pod c79lv31ayh28l1 terminated (connection refused)
    ... eight healthy A40s in 69 seconds ...
    08:37:22  pod n9fpbskd7owl2h terminated (connection refused)

522 pages were in flight on them. The warm floor then re-rented eight
more, which booted ~12 minutes, began refusing, and were killed again:
$3.72 of pods billed -- more than the campaign's whole spend -- on boots
that never produced a page, while throughput read 0.577 pages/s against
a measured 0.557 PER POD.

A pod is now QUARANTINED instead: out of rotation, still ours, still
counted against max_pods because it is still billing. `maintain`
re-probes it when the quarantine expires (45s) and

  * it answers -> straight back into rotation, which is the common case
    and the whole reason this stopped terminating;
  * it has work in flight -> left alone whatever the probe says, and it
    accrues no strikes while excused, so it does not die the moment the
    last page drains. Killing a pod with pages on it loses them AND pays
    for them;
  * it fails 3 probes with nothing in flight -> terminated. ~2 minutes of
    a pod that exists, bills and will not answer while carrying no load
    is dead in every sense that matters, so this does not trade a
    terminating bug for a leaking one.

A quarantined pod also counts toward the warm-floor deficit. Without
that, eight set-aside pods read as a deficit of eight and the pool rents
~700s of boot to cover a 45s gap, ending with two cards where the floor
asked for one -- the re-rent loop in one line.

The snapshot reports `quarantined` as its own state with the seconds
remaining, and `pods_quarantined` in the summary. It is NOT counted as
booting, which told an operator a settled fleet was still starting and
told a client sizing from pods_ready to wait for pods already present.

`discard` has exactly one caller (RunpodBackend._unreachable, reason
"connection refused"), so nothing else loses a termination it needed.

19 tests. Full suite 3777 passed, 15 pre-existing failures unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.36"
+__version__ = "0.4.37"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/backends/runpod.py

@@ -332,11 +332,17 @@ class RunpodBackend(ModelBackend):
             release()
 
     def _unreachable(self) -> None:
-        """The pod refused the connection: drop it and let the caller retry.
+        """The pod refused the connection: set it aside and let the caller retry.
 
         A pod terminated behind the pool's back (the test harness reaping by
         API, a reclaimed spot machine) stays "healthy" until the next timed
         probe; without this, one request in that window fails for nothing.
+
+        `discard` takes it out of rotation and does NOT terminate it: a refused
+        connection is as likely to mean the pod's listen backlog is full as that
+        the pod is gone, and the pool re-probes it before deciding. Reporting a
+        refusal here is therefore cheap and safe, which it was not when this
+        call destroyed the card.
         """
         pod = getattr(self, "_last_pod", None)
         if pod is not None and self._pool is not None:

tests/test_runpod_client_scale.py

@@ -359,8 +359,8 @@ def test_the_warm_top_up_does_not_stack_on_a_booting_pod():
     """
     src = __import__("pathlib").Path(rw.__file__).read_text()
     blk = src.split("def maintain")[1].split("def close")[0]
-    assert ("healthy_in_hand = len([p for p in self.pods if p.healthy]) "
-            "+ self._booting") in blk
+    assert "if p.healthy or p.quarantined_until])" in blk
+    assert "healthy_in_hand" in blk
     assert "billing_in_hand = len(self.pods) + self._booting" in blk
     assert "want = min(keep_min - healthy_in_hand," in blk
     # the cap must be the billing count, or the regression returns

tests/test_runpod_warm_floor_respects_max_pods.py

@@ -104,7 +104,12 @@ def test_an_unhealthy_pod_does_not_satisfy_the_floor_when_there_is_room():
 def test_worker_uses_two_distinct_counts():
     src = open("codai/api/runpod_worker.py").read()
     assert "billing_in_hand = len(self.pods) + self._booting" in src
-    assert "healthy_in_hand = len([p for p in self.pods if p.healthy]) + self._booting" in src
+    # The deficit count also credits a QUARANTINED pod since v0.4.37: it is
+    # ours and billing and due back in seconds, so renting over it buys ~700s
+    # of boot to cover a 45s gap. Still a count of pods that can SERVE soon,
+    # as distinct from pods that BILL.
+    assert "if p.healthy or p.quarantined_until])" in src
+    assert "healthy_in_hand" in src and "+ self._booting" in src
     assert "max(0, self.effective_max_pods() - billing_in_hand)" in src, \
         "the max_pods cap must be tested against pods that bill"
     assert "want = min(keep_min - healthy_in_hand," in src, \