• Stefy Lanza (nextime / spora )'s avatar
    frontproxy: a wedged GPU can no longer hang the whole front · 09558635
    Stefy Lanza (nextime / spora ) authored
    At front startup _build_engines runs vulkaninfo to enumerate GPUs; on a
    wedged Polaris card that process blocks UNINTERRUPTIBLY (D-state), and
    subprocess.run(timeout=) can't kill a D-state child — so the front's
    uvicorn startup hung forever ("Waiting for application startup"),
    taking down the healthy NVIDIA engine too (502 everywhere).
    
    Add _call_with_timeout: runs the probe in a daemon thread and STOPS
    WAITING after a wall-clock timeout, leaving the un-killable call
    orphaned instead of blocking. Wrap vulkan_devices() (12s) so startup
    always completes, and _amd_stats() (6s) so a hung Radeon can't freeze
    the health/thermal poll threads that also monitor the healthy card.
    A wedged engine now degrades to "that engine down", not "everything down".
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
    09558635
__init__.py 4.11 KB