• Stefy Lanza (nextime / spora )'s avatar
    a pod's in-flight count must be able to come down on its own (v0.4.28) · 9d67ab8b
    Stefy Lanza (nextime / spora ) authored
    `pod.inflight` is raised in `_pick` and lowered only in `release`, which makes
    it a ratchet. Any dispatch path that fails to release raises it for the life of
    the pod, and at max_inflight_per_pod the pod is refused all work while its card
    sits idle -- reported as "at capacity", which is the most misleading thing this
    file can say.
    
    Observed on a deliberately single-pod fleet: the pool reported inflight=64 of
    64 with 63 pages queued, while the pod itself reported `Running: 31,
    Waiting: 0` at 10.6% KV cache. Half the slots did not exist. A single test
    document then waited 900s for a pod that was half idle, and timed out.
    
    Two such leaks have now been found and fixed -- a lease dropped during
    acquisition (0.4.23) and a page whose client disconnected. The reason this
    change exists is the third one, which has not been found: a counter that can
    only rise is one bug away from taking a pod out of service permanently, and
    every leak so far has been invisible precisely because the pool reported
    confidence about a number nothing could correct.
    
    So each slot now records when it was taken, and the LIST is the authority:
    
    * slots held longer than INFLIGHT_TTL_S (900s) cannot belong to a live page and
      are returned. Deliberately generous -- a safety net, not a timeout. Returning
      a slot that IS in use would hand a pod more work than its cap allows, while
      holding a phantom a few minutes longer costs only those minutes.
    * a counter standing above its own recorded slots is counting slots with no
      evidence of ever being taken, and is pulled down to the evidence. This is not
      conditional on anything having expired -- the first version of the sweep only
      examined pods with a stale entry, which would have skipped a pod whose
      counter had drifted upward, and skipped every pod already in the pool when
      this code arrives carrying a counter and no timestamps: exactly the
      locked-out pod the sweep exists to free.
    
    The list is created on demand rather than required at construction, so
    duck-typed pod stand-ins are not forced to carry a bookkeeping field, as with
    the per-card memo.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    9d67ab8b