During a boot wave /v1/runpod/status stopped answering entirely, and a client
reasonably read that as "the front is dead" -- while /healthz answered in 2.9
ms throughout and the front recorded zero event-loop stalls. It was never the
loop. The snapshot takes the pool locks, a provisioning wave holds them, and
the request simply waited until the caller gave up.

That cost more than observability. It is the one surface an operator has for
seeing WHY a server is busy, so losing it exactly when the server is busy sent
two investigations to the wrong layer, and it silently turned a monitoring
sampler's output into zeros -- 48 pods billing at $35/hr, recorded as an empty
fleet, because a hung read and an empty fleet wrote the same row.

So every lock in `pods_status` is taken with a timeout (`_lock_for_read`,
2 s). A pool whose lock is busy contributes one `state: "busy"` row naming
itself; a locked registry answers one such row rather than an empty list,
because "no pods" and "ask again" are opposite actions and must not look
alike. Writers keep the plain lock: a reader that skips it reports less, which
is the point, while a writer that skipped it would race.

Note for anyone testing this: `_pools_lock` and `pool._cv` are RLocks, so a
test that takes one in the thread that then reads simply re-enters and passes.
The holder in production is always another thread, so the tests hold them from
one too -- three of them passed for the wrong reason until they did.

Also, in the same endpoint: `caps` has been silently empty in every answer it
has ever given. `from codai.config import config_manager` -- which does not
exist, the manager lives in `codai.admin.routes` -- raised every time and the
except swallowed it, so an operator polling for the install's spend ceilings
never received them. Identical mistake, identical shape, to the one that
pinned both thread pools to their floors on every install that ever ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM