A rented pod is the one machine in the system whose logs cannot be read. RunPod
publishes no pod-log API — its REST spec lists 23 routes and none serve logs,
which is why every attempt returns HTTP 400 — and boot.sh writes its phases to
a file and then execs uvicorn, whose output goes to the container's stdout and
nowhere else. /boot therefore covered the way up and nothing after it, and
_dump_pod_logs only ever ran on a failure, printing into the orchestrator's log.

That cost an evening. Three pods were investigated in one session and two were
already destroyed by the time there was a question to ask — one reclaimed by
RunPod mid-request, taking the only evidence with it — so "the serving hangs"
stayed a hypothesis for an hour. It was not hanging: a page took 807s, most of
it the weights download every pod repeats because global volumes are
console-only, and every client had given up at 180-300s.

So the pod keeps a bounded ring of its own stdout and stderr (codai/api/podlog)
and serves it at /logs with a cursor, and the pool drains that on every scaler
tick into <config_dir>/pod-logs/<pod_id>.log — kept AFTER the pod dies, which is
the whole point. It also drains once more before tearing a pod down, because the
interesting lines are the last ones and they are only readable while the pod
exists.

A pull, not a push: the orchestrator already reaches pods to serve requests, so
the bearer token and pinned TLS apply unchanged and a pod with no outbound route
still gets its log kept. Bounded at both ends, since neither disk is free, and a
gap in the pod's ring is written into the file instead of passing silently. The
stdout tee writes through to the real stream on every failure path — a logging
aid that could break printing would be worse than none.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW