• Stefy Lanza (nextime / spora )'s avatar
    broker/text: drop the "thinking" keepalive; never pass max_tokens=None (512 cut) · 62dde8c2
    Stefy Lanza (nextime / spora ) authored
    Two fixes from live debugging:
    
    1. broker_execute_stream: remove the "the model is thinking..." keepalive chunk.
       It leaked into the assistant reply (saved turns began with the placeholder) and
       isn't wanted — we just wait for the real response. The engine waits ~5min for a
       load and the broker sends protocol-level `pending` keepalives, so liveness holds
       without injecting content. Reverts the streaming body to the simple
       retry-on-connection/transient-5xx form (no queue/keepalive).
    
    2. _clamp_max_tokens: always resolve a concrete max_tokens, never None. The main
       stream/generate paths pass request.max_tokens bare; when the client omits it the
       GGUF backend's `max_tokens or 512` default truncated replies at ~512 tokens
       mid-sentence. Now: model-level cap is authority (client honored only if smaller);
       no cap configured -> keep client value, else default 2048.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    62dde8c2
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
characters.py Loading commit data...
custom_pipelines.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
images.py Loading commit data...
log.py Loading commit data...
loras.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
prompt_cache.py Loading commit data...
ratelimit.py Loading commit data...
spatial.py Loading commit data...
state.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...