• Stefy Lanza (nextime / spora )'s avatar
    colibri: single-client run_serve mode (kv_slots=1) to keep MTP enabled · 98963164
    Stefy Lanza (nextime / spora ) authored
    The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP
    speculative-decode head ("speculation is not ragged-safe across KV slots") and
    its batched decode loop has no MTP path at all — MTP lives only in run_serve's
    single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive
    run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes
    = fewer expert-streaming rounds, a real decode win on this IO-bound engine.
    
    MuxEngine now selects single-client mode when kv_slots==1: launches run_serve,
    drains its one-shot startup telemetry, and runs turns synchronously over the
    run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 +
    STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path.
    pause/resume are no-ops in single-client mode (run_serve would misread the frame;
    thermal falls back to the front's SIGSTOP). Bump 0.1.81.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
    98963164
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...