colibri: single-client run_serve mode (kv_slots=1) to keep MTP enabled

The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP
speculative-decode head ("speculation is not ragged-safe across KV slots") and
its batched decode loop has no MTP path at all — MTP lives only in run_serve's
single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive
run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes
= fewer expert-streaming rounds, a real decode win on this IO-bound engine.

MuxEngine now selects single-client mode when kv_slots==1: launches run_serve,
drains its one-shot startup telemetry, and runs turns synchronously over the
run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 +
STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path.
pause/resume are no-ops in single-client mode (run_serve would misread the frame;
thermal falls back to the front's SIGSTOP). Bump 0.1.81.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
parent cedaabb3
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.80" __version__ = "0.1.81"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment