• Stefy Lanza (nextime / spora )'s avatar
    broker/text: drop the "thinking" keepalive; never pass max_tokens=None (512 cut) · 62dde8c2
    Stefy Lanza (nextime / spora ) authored
    Two fixes from live debugging:
    
    1. broker_execute_stream: remove the "the model is thinking..." keepalive chunk.
       It leaked into the assistant reply (saved turns began with the placeholder) and
       isn't wanted — we just wait for the real response. The engine waits ~5min for a
       load and the broker sends protocol-level `pending` keepalives, so liveness holds
       without injecting content. Reverts the streaming body to the simple
       retry-on-connection/transient-5xx form (no queue/keepalive).
    
    2. _clamp_max_tokens: always resolve a concrete max_tokens, never None. The main
       stream/generate paths pass request.max_tokens bare; when the client omits it the
       GGUF backend's `max_tokens or 512` default truncated replies at ~512 tokens
       mid-sentence. Now: model-level cap is authority (client honored only if smaller);
       no cap configured -> keep client value, else default 2048.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    62dde8c2
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...