• Stefy Lanza (nextime / spora )'s avatar
    Stop event-loop blocking that trips broker WebSocket keepalive timeouts · fe47ead1
    Stefy Lanza (nextime / spora ) authored
    Under load the CoderAI broker WebSocket was being torn down with
    "1011 keepalive ping timeout" and reconnecting (surfaced as 502s by the
    fronting proxy). Root cause: synchronous work run directly on the asyncio
    event loop starves the WebSocket keepalive so PONGs miss the 20s window.
    
    Remove the blockers:
    
    - analytics: dispatch the blocking token_usage / per-user INSERTs to a
      worker thread (fire-and-forget when a loop is running, inline otherwise)
      instead of calling them inline from async request handlers.
    - database: add a MySQL connection pool (ping+reconnect, direct-connect
      fallback) and return connections to the pool on with-block exit instead
      of opening a fresh TCP+auth connection per query and leaking it to GC.
    - utils: cache the tiktoken encoding and memoize count_messages_tokens by
      content, collapsing the several CPU-bound BPE passes per request into one.
    - coderai_broker: run touch()/publish_response Redis I/O off the loop via
      asyncio.to_thread (awaited to preserve chunk order), matching the existing
      consume_request pattern.
    - routes/coderai_broker: answer heartbeats immediately and process response
      frames on a dedicated FIFO worker so a slow publish can't delay the next
      heartbeat.
    Co-Authored-By: 's avatarClaude Opus 4.8 (1M context) <noreply@anthropic.com>
    fe47ead1
database.py 308 KB