a cap-only lease reported as already expired (v0.4.35)

Found by exercising the new ceiling against a live orchestrator:

    max_pods=128  effective_max_pods=8  client_max_pods=8
    client_pods=0  client_lease_expires_in_s=0

The lease had 1800 s left and was the reason the fleet had stopped
growing at 8. `client_lease_expires_in_s` was gated on the FLOOR, which
is 0 for a cap-only lease -- "grow no further than 8, hold nothing warm",
which is what a client with a long tail of work asks for. It now reports
whichever half of the lease lasts longer.
Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
parent 9265a7ee
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.4.34" __version__ = "0.4.35"
__tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference." __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
......
...@@ -5805,8 +5805,15 @@ def pools_schedule_status() -> list: ...@@ -5805,8 +5805,15 @@ def pools_schedule_status() -> list:
# The client lease, so a bill nobody expected has a visible cause. # The client lease, so a bill nobody expected has a visible cause.
"client_scale_allowed": bool(getattr(cfg, "allow_client_scale", False)), "client_scale_allowed": bool(getattr(cfg, "allow_client_scale", False)),
"client_pods": pool.client_floor(), "client_pods": pool.client_floor(),
# Whichever half of the lease lasts longer. Gated on the FLOOR
# alone, a cap-only lease -- "grow no further than 8, hold nothing
# warm" -- reported as already expired while it was still the
# reason the fleet had stopped growing.
"client_lease_expires_in_s": max( "client_lease_expires_in_s": max(
0, int(pool._client_floor_until - time.time())) if pool.client_floor() else 0, 0,
int(pool._client_floor_until - time.time()) if pool.client_floor() else 0,
int(getattr(pool, "_client_cap_until", 0.0) - time.time())
if pool.client_cap() else 0),
# What the lease narrowed the card choice to, if anything: a pod on # What the lease narrowed the card choice to, if anything: a pod on
# an unexpected card should have a visible cause too, not only a # an unexpected card should have a visible cause too, not only a
# bill. # bill.
......
...@@ -252,6 +252,21 @@ def test_set_client_pods_forwards_it_for_both_ways_of_naming_a_pool(): ...@@ -252,6 +252,21 @@ def test_set_client_pods_forwards_it_for_both_ways_of_naming_a_pool():
"both the live-pool branch and the by-model branch must pass it") "both the live-pool branch and the by-model branch must pass it")
def test_a_cap_only_lease_does_not_report_as_expired():
"""Observed live on 2026-10-19: a lease with 1800 s of cap left reported
`client_lease_expires_in_s: 0`, because the field was gated on the FLOOR.
A fleet that stopped growing must say why, and for how much longer."""
import time as _t
p = _pool(max_pods=128, ttl=1800)
p.set_client_floor(0, max_pods=8)
left = max(0,
int(p._client_floor_until - _t.time()) if p.client_floor() else 0,
int(p._client_cap_until - _t.time()) if p.client_cap() else 0)
assert left > 1700, left
src = open("codai/api/runpod_worker.py").read()
assert 'if pool.client_cap() else 0)' in src
def test_the_status_surface_reports_the_ceiling_in_force(): def test_the_status_surface_reports_the_ceiling_in_force():
"""A fleet that stopped growing short of max_pods must say why.""" """A fleet that stopped growing short of max_pods must say why."""
src = inspect_source = open("codai/api/runpod_worker.py").read() src = inspect_source = open("codai/api/runpod_worker.py").read()
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment