-
Stefy Lanza (nextime / spora ) authored
Clients hitting the public API directly (e.g. township via nginx) disconnected during long waits: the direct proxy() path acquired the front queue slot silently and awaited the engine's first byte (model load) with no output. The broker path already kept alive; the direct path now does too. For a STREAMING inference request, commit to a 200 text/event-stream up front and emit keepalive while acquiring the queue slot and during the engine's model load / not-ready retries, then relay the real stream (token-counting for the Tasks page), ending cleanly if the engine dies mid-flight. Configurable mode (models.wait_status_mode, global default 'invisible'; per-model override via the models.json entry): * invisible — empty-content SSE chunk + x_queue_info (holds the connection; no content pollution) * visible — short visible status text (appears in the content) * silent — nothing (legacy path) When thinking is enabled the keepalive goes on the reasoning channel instead (no pollution), unless mode is silent. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
6583c394