a refused connection no longer halves a pod's width (v0.4.42)

Operator decision. A pod that refuses a connection is still quarantined
and re-probed rather than terminated (v0.4.37), but its dispatch width
is no longer cut in half. The queue in front of the fleet absorbs the
surplus; narrowing the pod only starved a card that was measured at
11-13% KV cache with nothing waiting.
Co-Authored-By: 's avatarClaude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
parent d8f1e394
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.4.41" __version__ = "0.4.42"
__tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference." __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
......
...@@ -4739,11 +4739,6 @@ class RunpodPodPool: ...@@ -4739,11 +4739,6 @@ class RunpodPodPool:
if pod not in self.pods: if pod not in self.pods:
return return
held = int(pod.inflight) held = int(pod.inflight)
# BEFORE the quarantine, so the width is learned from the load that
# actually broke it. This is the part that makes the fleet work at
# whatever concurrency a client offers: the pod tells us its width by
# refusing, and we stop exceeding it.
width = self.note_refusal(pod)
with self._cv: with self._cv:
if pod not in self.pods: if pod not in self.pods:
return return
...@@ -4754,7 +4749,8 @@ class RunpodPodPool: ...@@ -4754,7 +4749,8 @@ class RunpodPodPool:
f"{reason}. NOT terminated — a refused connection is as likely to " f"{reason}. NOT terminated — a refused connection is as likely to "
f"mean the pod is overloaded as gone, and it still has {held} " f"mean the pod is overloaded as gone, and it still has {held} "
f"page(s) charged to it. It is re-probed when the quarantine " f"page(s) charged to it. It is re-probed when the quarantine "
f"expires, and will then be fed at most {width} at a time.", f"expires. Its width is NOT reduced: the queue in front of the "
f"fleet absorbs the surplus.",
flush=True) flush=True)
def forget(self, pod: PodHandle, reason: str = "gone") -> bool: def forget(self, pod: PodHandle, reason: str = "gone") -> bool:
......
...@@ -251,9 +251,8 @@ def test_the_snapshot_reports_the_width_in_force(): ...@@ -251,9 +251,8 @@ def test_the_snapshot_reports_the_width_in_force():
assert '"capacity": pool.pod_capacity_now(p)' in src assert '"capacity": pool.pod_capacity_now(p)' in src
def test_discard_learns_the_width_before_setting_the_pod_aside(): def test_a_refusal_does_not_narrow_the_pod():
"""Afterwards it would read an in-flight count the quarantine had already """Operator decision: a refusal quarantines the pod but never halves its
begun to unwind, and learn the wrong number.""" width -- the queue in front of the fleet takes the surplus."""
import inspect import inspect
src = inspect.getsource(rw.RunpodPodPool.discard) assert "note_refusal" not in inspect.getsource(rw.RunpodPodPool.discard)
assert src.index("note_refusal") < src.index("quarantined_until = now")
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment