• Stefy Lanza (nextime / spora )'s avatar
    embeddings: per-model throttle + default 200ms pacing · dc6026ef
    Stefy Lanza (nextime / spora ) authored
    The RX 580 (Polaris/Vulkan) keeps timing out its GPU ring under relentless
    back-to-back embedding load — a global throttle can't distinguish a fragile card
    from a robust one. Make the admission gate PER-MODEL: each model id gets its own
    semaphore + backlog counter + pacing, resolved from the models.json entry
    (embed_max_concurrency / embed_max_backlog / embed_min_interval_ms), falling back
    to the CODERAI_EMBED_* env vars, then defaults. Default min interval is now
    200ms (was 0) so GPU starts are paced out of the box, giving the ring breathing
    room; a robust CUDA model can set it to 0 per-model. Bump 0.1.82.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
    dc6026ef
__init__.py 4.11 KB