• Stefy Lanza (nextime / spora )'s avatar
    front: wait-keepalive on the direct streaming path (queue/model-load) · 6583c394
    Stefy Lanza (nextime / spora ) authored
    Clients hitting the public API directly (e.g. township via nginx) disconnected
    during long waits: the direct proxy() path acquired the front queue slot silently
    and awaited the engine's first byte (model load) with no output. The broker path
    already kept alive; the direct path now does too.
    
    For a STREAMING inference request, commit to a 200 text/event-stream up front and
    emit keepalive while acquiring the queue slot and during the engine's model load /
    not-ready retries, then relay the real stream (token-counting for the Tasks page),
    ending cleanly if the engine dies mid-flight.
    
    Configurable mode (models.wait_status_mode, global default 'invisible'; per-model
    override via the models.json entry):
      * invisible — empty-content SSE chunk + x_queue_info (holds the connection; no
                    content pollution)
      * visible   — short visible status text (appears in the content)
      * silent    — nothing (legacy path)
    When thinking is enabled the keepalive goes on the reasoning channel instead
    (no pollution), unless mode is silent.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    6583c394
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...