Skip to content
Commit 9d67ab8b authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

a pod's in-flight count must be able to come down on its own (v0.4.28)



`pod.inflight` is raised in `_pick` and lowered only in `release`, which makes
it a ratchet. Any dispatch path that fails to release raises it for the life of
the pod, and at max_inflight_per_pod the pod is refused all work while its card
sits idle -- reported as "at capacity", which is the most misleading thing this
file can say.

Observed on a deliberately single-pod fleet: the pool reported inflight=64 of
64 with 63 pages queued, while the pod itself reported `Running: 31,
Waiting: 0` at 10.6% KV cache. Half the slots did not exist. A single test
document then waited 900s for a pod that was half idle, and timed out.

Two such leaks have now been found and fixed -- a lease dropped during
acquisition (0.4.23) and a page whose client disconnected. The reason this
change exists is the third one, which has not been found: a counter that can
only rise is one bug away from taking a pod out of service permanently, and
every leak so far has been invisible precisely because the pool reported
confidence about a number nothing could correct.

So each slot now records when it was taken, and the LIST is the authority:

* slots held longer than INFLIGHT_TTL_S (900s) cannot belong to a live page and
  are returned. Deliberately generous -- a safety net, not a timeout. Returning
  a slot that IS in use would hand a pod more work than its cap allows, while
  holding a phantom a few minutes longer costs only those minutes.
* a counter standing above its own recorded slots is counting slots with no
  evidence of ever being taken, and is pulled down to the evidence. This is not
  conditional on anything having expired -- the first version of the sweep only
  examined pods with a stale entry, which would have skipped a pod whose
  counter had drifted upward, and skipped every pod already in the pool when
  this code arrives carrying a counter and no timestamps: exactly the
  locked-out pod the sweep exists to free.

The list is created on demand rather than required at construction, so
duck-typed pod stand-ins are not forced to carry a bookkeeping field, as with
the per-card memo.

Co-Authored-By: default avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
parent 1757a2c1
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment