-
Stefy Lanza (nextime / spora ) authored
`pod.inflight` is raised in `_pick` and lowered only in `release`, which makes it a ratchet. Any dispatch path that fails to release raises it for the life of the pod, and at max_inflight_per_pod the pod is refused all work while its card sits idle -- reported as "at capacity", which is the most misleading thing this file can say. Observed on a deliberately single-pod fleet: the pool reported inflight=64 of 64 with 63 pages queued, while the pod itself reported `Running: 31, Waiting: 0` at 10.6% KV cache. Half the slots did not exist. A single test document then waited 900s for a pod that was half idle, and timed out. Two such leaks have now been found and fixed -- a lease dropped during acquisition (0.4.23) and a page whose client disconnected. The reason this change exists is the third one, which has not been found: a counter that can only rise is one bug away from taking a pod out of service permanently, and every leak so far has been invisible precisely because the pool reported confidence about a number nothing could correct. So each slot now records when it was taken, and the LIST is the authority: * slots held longer than INFLIGHT_TTL_S (900s) cannot belong to a live page and are returned. Deliberately generous -- a safety net, not a timeout. Returning a slot that IS in use would hand a pod more work than its cap allows, while holding a phantom a few minutes longer costs only those minutes. * a counter standing above its own recorded slots is counting slots with no evidence of ever being taken, and is pulled down to the evidence. This is not conditional on anything having expired -- the first version of the sweep only examined pods with a stale entry, which would have skipped a pod whose counter had drifted upward, and skipped every pod already in the pool when this code arrives carrying a counter and no timestamps: exactly the locked-out pod the sweep exists to free. The list is created on demand rather than required at construction, so duck-typed pod stand-ins are not forced to carry a bookkeeping field, as with the per-card memo. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
9d67ab8b