The front health-polls /internal/engine-state every ~2s. It called
torch.cuda.mem_get_info + get_device_name on EVERY poll, touching the CUDA context,
which can serialize behind the running forward pass and stall the handler past the
poll timeout — flipping a busy engine to "not responding". Cache the VRAM snapshot
(4s TTL) and device names (permanent), so mid-generation polls return instantly
from cache instead of blocking on CUDA. (llama-cpp-python 0.3.30 uses ctypes, which
already releases the GIL during eval, so the compute itself wasn't the blocker.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>