Skip to content
Commit 98963164 authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

colibri: single-client run_serve mode (kv_slots=1) to keep MTP enabled



The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP
speculative-decode head ("speculation is not ragged-safe across KV slots") and
its batched decode loop has no MTP path at all — MTP lives only in run_serve's
single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive
run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes
= fewer expert-streaming rounds, a real decode win on this IO-bound engine.

MuxEngine now selects single-client mode when kv_slots==1: launches run_serve,
drains its one-shot startup telemetry, and runs turns synchronously over the
run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 +
STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path.
pause/resume are no-ops in single-client mode (run_serve would misread the frame;
thermal falls back to the front's SIGSTOP). Bump 0.1.81.

Co-Authored-By: default avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
parent cedaabb3
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment