• Stefy Lanza (nextime / spora )'s avatar
    Stop event-loop blocking that trips broker WebSocket keepalive timeouts · fe47ead1
    Stefy Lanza (nextime / spora ) authored
    Under load the CoderAI broker WebSocket was being torn down with
    "1011 keepalive ping timeout" and reconnecting (surfaced as 502s by the
    fronting proxy). Root cause: synchronous work run directly on the asyncio
    event loop starves the WebSocket keepalive so PONGs miss the 20s window.
    
    Remove the blockers:
    
    - analytics: dispatch the blocking token_usage / per-user INSERTs to a
      worker thread (fire-and-forget when a loop is running, inline otherwise)
      instead of calling them inline from async request handlers.
    - database: add a MySQL connection pool (ping+reconnect, direct-connect
      fallback) and return connections to the pool on with-block exit instead
      of opening a fresh TCP+auth connection per query and leaking it to GC.
    - utils: cache the tiktoken encoding and memoize count_messages_tokens by
      content, collapsing the several CPU-bound BPE passes per request into one.
    - coderai_broker: run touch()/publish_response Redis I/O off the loop via
      asyncio.to_thread (awaited to preserve chunk order), matching the existing
      consume_request pattern.
    - routes/coderai_broker: answer heartbeats immediately and process response
      frames on a dedicated FIFO worker so a slow publish can't delay the next
      heartbeat.
    Co-Authored-By: 's avatarClaude Opus 4.8 (1M context) <noreply@anthropic.com>
    fe47ead1
Name
Last commit
Last update
aisbf Loading commit data...
config Loading commit data...
docs Loading commit data...
static Loading commit data...
templates Loading commit data...
tests Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
API_EXAMPLES.md Loading commit data...
BUILD_DEPLOY.md Loading commit data...
CHANGELOG.md Loading commit data...
DEBUG_GUIDE.md Loading commit data...
DOCUMENTATION.md Loading commit data...
ENDPOINTS.md Loading commit data...
KILO_OAUTH2.md Loading commit data...
LICENSE.txt Loading commit data...
MANIFEST.in Loading commit data...
MISSING_ENDPOINTS.md Loading commit data...
PYPI.md Loading commit data...
QUICK_START_PAYMENT.md Loading commit data...
README.md Loading commit data...
RUNPOD_IMPLEMENTATION_PLAN.md Loading commit data...
SUBSCRIPTION_PRICING_FEATURE.md Loading commit data...
TODO.md Loading commit data...
TORRC_SETUP.md Loading commit data...
aisbf.sh Loading commit data...
build.sh Loading commit data...
chatgpt-api-implementation-guide.md Loading commit data...
clean.sh Loading commit data...
cli.py Loading commit data...
codex-oauth-implementation-guide.md Loading commit data...
main.py Loading commit data...
main.py.bak Loading commit data...
minio-archive-plan.md Loading commit data...
pyproject.toml Loading commit data...
requirements.txt Loading commit data...
screenshot.png Loading commit data...
setup.py Loading commit data...
start_proxy.sh Loading commit data...
verify_release.sh Loading commit data...