• Stefy Lanza (nextime / spora )'s avatar
    front: wait-keepalive on the direct streaming path (queue/model-load) · 6583c394
    Stefy Lanza (nextime / spora ) authored
    Clients hitting the public API directly (e.g. township via nginx) disconnected
    during long waits: the direct proxy() path acquired the front queue slot silently
    and awaited the engine's first byte (model load) with no output. The broker path
    already kept alive; the direct path now does too.
    
    For a STREAMING inference request, commit to a 200 text/event-stream up front and
    emit keepalive while acquiring the queue slot and during the engine's model load /
    not-ready retries, then relay the real stream (token-counting for the Tasks page),
    ending cleanly if the engine dies mid-flight.
    
    Configurable mode (models.wait_status_mode, global default 'invisible'; per-model
    override via the models.json entry):
      * invisible — empty-content SSE chunk + x_queue_info (holds the connection; no
                    content pollution)
      * visible   — short visible status text (appears in the content)
      * silent    — nothing (legacy path)
    When thinking is enabled the keepalive goes on the reasoning channel instead
    (no pollution), unless mode is silent.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    6583c394
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...