-
Stefy Lanza (nextime / spora ) authored
Two fixes from live debugging: 1. broker_execute_stream: remove the "the model is thinking..." keepalive chunk. It leaked into the assistant reply (saved turns began with the placeholder) and isn't wanted — we just wait for the real response. The engine waits ~5min for a load and the broker sends protocol-level `pending` keepalives, so liveness holds without injecting content. Reverts the streaming body to the simple retry-on-connection/transient-5xx form (no queue/keepalive). 2. _clamp_max_tokens: always resolve a concrete max_tokens, never None. The main stream/generate paths pass request.max_tokens bare; when the client omits it the GGUF backend's `max_tokens or 512` default truncated replies at ~512 tokens mid-sentence. Now: model-level cap is authority (client honored only if smaller); no cap configured -> keep client value, else default 2048. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
62dde8c2
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| __init__.py | ||
| admin_data.py | ||
| app.py | ||
| assignment.py | ||
| engine_supervisor.py | ||
| gpu_detect.py | ||
| registry.py | ||
| reqqueue.py | ||
| router.py | ||
| ui_pages.py |