colibri: single-client run_serve mode (kv_slots=1) to keep MTP enabled
The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP
speculative-decode head ("speculation is not ragged-safe across KV slots") and
its batched decode loop has no MTP path at all — MTP lives only in run_serve's
single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive
run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes
= fewer expert-streaming rounds, a real decode win on this IO-bound engine.
MuxEngine now selects single-client mode when kv_slots==1: launches run_serve,
drains its one-shot startup telemetry, and runs turns synchronously over the
run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 +
STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path.
pause/resume are no-ops in single-client mode (run_serve would misread the frame;
thermal falls back to the front's SIGSTOP). Bump 0.1.81.
Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
Showing
This diff is collapsed.
Please
register
or
sign in
to comment