• Stefy Lanza (nextime / spora )'s avatar
    colibri: single-client run_serve mode (kv_slots=1) to keep MTP enabled · 98963164
    Stefy Lanza (nextime / spora ) authored
    The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP
    speculative-decode head ("speculation is not ragged-safe across KV slots") and
    its batched decode loop has no MTP path at all — MTP lives only in run_serve's
    single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive
    run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes
    = fewer expert-streaming rounds, a real decode win on this IO-bound engine.
    
    MuxEngine now selects single-client mode when kv_slots==1: launches run_serve,
    drains its one-shot startup telemetry, and runs turns synchronously over the
    run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 +
    STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path.
    pause/resume are no-ops in single-client mode (run_serve would misread the frame;
    thermal falls back to the front's SIGSTOP). Bump 0.1.81.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
    98963164