a refused connection no longer halves a pod's width (v0.4.42)

Operator decision. A pod that refuses a connection is still quarantined
and re-probed rather than terminated (v0.4.37), but its dispatch width
is no longer cut in half. The queue in front of the fleet absorbs the
surplus; narrowing the pod only starved a card that was measured at
11-13% KV cache with nothing waiting.
Co-Authored-By: 's avatarClaude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
parent d8f1e394
......@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here.
__version__ = "0.4.41"
__version__ = "0.4.42"
__tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
# Configure the CUDA caching allocator BEFORE torch is imported anywhere.
......
......@@ -4739,11 +4739,6 @@ class RunpodPodPool:
if pod not in self.pods:
return
held = int(pod.inflight)
# BEFORE the quarantine, so the width is learned from the load that
# actually broke it. This is the part that makes the fleet work at
# whatever concurrency a client offers: the pod tells us its width by
# refusing, and we stop exceeding it.
width = self.note_refusal(pod)
with self._cv:
if pod not in self.pods:
return
......@@ -4754,7 +4749,8 @@ class RunpodPodPool:
f"{reason}. NOT terminated — a refused connection is as likely to "
f"mean the pod is overloaded as gone, and it still has {held} "
f"page(s) charged to it. It is re-probed when the quarantine "
f"expires, and will then be fed at most {width} at a time.",
f"expires. Its width is NOT reduced: the queue in front of the "
f"fleet absorbs the surplus.",
flush=True)
def forget(self, pod: PodHandle, reason: str = "gone") -> bool:
......
......@@ -251,9 +251,8 @@ def test_the_snapshot_reports_the_width_in_force():
assert '"capacity": pool.pod_capacity_now(p)' in src
def test_discard_learns_the_width_before_setting_the_pod_aside():
"""Afterwards it would read an in-flight count the quarantine had already
begun to unwind, and learn the wrong number."""
def test_a_refusal_does_not_narrow_the_pod():
"""Operator decision: a refusal quarantines the pod but never halves its
width -- the queue in front of the fleet takes the surplus."""
import inspect
src = inspect.getsource(rw.RunpodPodPool.discard)
assert src.index("note_refusal") < src.index("quarantined_until = now")
assert "note_refusal" not in inspect.getsource(rw.RunpodPodPool.discard)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment