• Stefy Lanza (nextime / spora )'s avatar
    front: wait-keepalive on the direct streaming path (queue/model-load) · 6583c394
    Stefy Lanza (nextime / spora ) authored
    Clients hitting the public API directly (e.g. township via nginx) disconnected
    during long waits: the direct proxy() path acquired the front queue slot silently
    and awaited the engine's first byte (model load) with no output. The broker path
    already kept alive; the direct path now does too.
    
    For a STREAMING inference request, commit to a 200 text/event-stream up front and
    emit keepalive while acquiring the queue slot and during the engine's model load /
    not-ready retries, then relay the real stream (token-counting for the Tasks page),
    ending cleanly if the engine dies mid-flight.
    
    Configurable mode (models.wait_status_mode, global default 'invisible'; per-model
    override via the models.json entry):
      * invisible — empty-content SSE chunk + x_queue_info (holds the connection; no
                    content pollution)
      * visible   — short visible status text (appears in the content)
      * silent    — nothing (legacy path)
    When thinking is enabled the keepalive goes on the reasoning channel instead
    (no pollution), unless mode is silent.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    6583c394
app.py 105 KB