-
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pinned pod image boots, but coderai calls its vLLM endpoint with a token the pod rejects — every page 401, every page empty". The 401 was real and the pages were really lost. It was not the image, and not a mismatch between what coderai holds and what the pod wants — the pod answers 200 to its recorded token, before and after. It was the ORDER of two reads. `RunpodBackend.load_model` snapshotted `pool.api_key` into `self._headers`, and `_acquire_endpoint()` — which is what triggers provisioning, and therefore what runs `_adopt_shared_pod` and REPLACES `pool.api_key` — runs after it. A pod for this pool was already running (ours, from before a restart), so the pool adopted its token rather than renting a second A40: 22:27:15 [runpod] adopting pod sncrmkqdnd7c57's bearer token 22:27:15 [runpod] reusing the pod already running … instead of renting another 22:27:15 Inference error: Error code: 401 - Invalid API key. Every request that crossed that moment carried the token from before it. All 64 pages in flight failed; the same lease answered 200 minutes later, which is why it looked like a property of the run rather than of one instant in it. `_auth()` now reads the pool at send time, and the three sites that used the snapshot go through it (chat completions, streaming, openai_endpoint). Serverless keeps the snapshot: it has no pool and no adoption event. The placement candidate also reads the key AFTER acquiring rather than before, since acquiring is the thing that can change it. Behind that, the OCR gateway now retries a 401/403 once against a freshly chosen upstream. A pod only answers to the token it was launched with, so a stale credential is not recoverable downstream — and throwing a page away for a credential that was already correct by the time the error arrived is the expensive half of this bug. A second refusal is returned as itself, not retried forever. Also fixes a regression in 0.3.56: `register_pod` re-stamped `created` on every write, but an adoption re-registers an existing pod — so a pod adopted hourly looked newly born hourly, and the reaper's ledger attribution (which 0.3.56 computes from `created`) would have under-counted its hours. An existing `created` is now preserved. 9 tests in tests/test_pod_token_adoption_race.py, including the ordering pinned against the source so no send site can quietly go back to the snapshot. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
c72c2134
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| __init__.py | ||
| base.py | ||
| colibri.py | ||
| colibri_families.py | ||
| cuda.py | ||
| ds4.py | ||
| ggml_rpc.py | ||
| gpu_probe.py | ||
| k3.py | ||
| ktransformers.py | ||
| overrides.py | ||
| remote_openai.py | ||
| runpod.py | ||
| vllm.py | ||
| vulkan.py |