• Stefy Lanza (nextime / spora )'s avatar
    broker/text: drop the "thinking" keepalive; never pass max_tokens=None (512 cut) · 62dde8c2
    Stefy Lanza (nextime / spora ) authored
    Two fixes from live debugging:
    
    1. broker_execute_stream: remove the "the model is thinking..." keepalive chunk.
       It leaked into the assistant reply (saved turns began with the placeholder) and
       isn't wanted — we just wait for the real response. The engine waits ~5min for a
       load and the broker sends protocol-level `pending` keepalives, so liveness holds
       without injecting content. Reverts the streaming body to the simple
       retry-on-connection/transient-5xx form (no queue/keepalive).
    
    2. _clamp_max_tokens: always resolve a concrete max_tokens, never None. The main
       stream/generate paths pass request.max_tokens bare; when the client omits it the
       GGUF backend's `max_tokens or 512` default truncated replies at ~512 tokens
       mid-sentence. Now: model-level cap is authority (client honored only if smaller);
       no cap configured -> keep client value, else default 2048.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    62dde8c2
Name
Last commit
Last update
..
__init__.py Loading commit data...
admin_data.py Loading commit data...
app.py Loading commit data...
assignment.py Loading commit data...
engine_supervisor.py Loading commit data...
gpu_detect.py Loading commit data...
registry.py Loading commit data...
reqqueue.py Loading commit data...
router.py Loading commit data...
ui_pages.py Loading commit data...