-
Stefy Lanza (nextime / spora ) authored
Under load the CoderAI broker WebSocket was being torn down with "1011 keepalive ping timeout" and reconnecting (surfaced as 502s by the fronting proxy). Root cause: synchronous work run directly on the asyncio event loop starves the WebSocket keepalive so PONGs miss the 20s window. Remove the blockers: - analytics: dispatch the blocking token_usage / per-user INSERTs to a worker thread (fire-and-forget when a loop is running, inline otherwise) instead of calling them inline from async request handlers. - database: add a MySQL connection pool (ping+reconnect, direct-connect fallback) and return connections to the pool on with-block exit instead of opening a fresh TCP+auth connection per query and leaking it to GC. - utils: cache the tiktoken encoding and memoize count_messages_tokens by content, collapsing the several CPU-bound BPE passes per request into one. - coderai_broker: run touch()/publish_response Redis I/O off the loop via asyncio.to_thread (awaited to preserve chunk order), matching the existing consume_request pattern. - routes/coderai_broker: answer heartbeats immediately and process response frames on a dedicated FIFO worker so a slow publish can't delay the next heartbeat. Co-Authored-By:Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fe47ead1