The orchestrator already talks to a pod in parallel, and nothing in the path
serializes: the front admits 256, acquiring a model instance is a ref-count, the
pool hands the same pod to every caller and only GROWS at the in-flight
threshold, the HTTP client keeps 64-256 connections per pod without blocking,
and the blocking call runs in a 256-worker executor (measured on the production
install; Python's own default there would have been 20).

What was not parallel-safe is everything a backend cannot return in its text.
`generate_chat` returns a string, so usage, logprobs, the tool calls a remote
parsed and the thought it separated travel beside it and are read AFTER the call
through `get_last_*`. Those were attributes of the backend -- and one backend
object serves every concurrent request for its model. Each call began by
resetting them, so a second request arriving between the first one's answer and
its read blanked the first one's tool call, or handed it the second one's. That
is the same shape as the empty answers seen in production on 2026-10-21, and
under real parallelism it is not a rare interleaving but the common case.

The store is per CALL now: a context variable, because a request is an asyncio
task with its own context and `asyncio.to_thread` copies it, so the worker
thread writes into the very dict the awaiting request will read. A stream
captures the store when it opens, since a thread inherits no context variables,
and accumulates its own thought instead of appending to a shared attribute. The
backends keep their attributes as the fallback, so every caller that never opens
a store -- the OCR pipeline, a script, a test -- behaves exactly as before.

Tests drive two deliberately interleaved calls, and two interleaved streams,
through ONE backend and assert neither sees the other's tool call, thought or
usage. The suite's autouse fixtures gain one more: a store is request-scoped
state, and tests share a context, so it is cleared between them like the
template cache.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66