compact an over-long conversation, against the window that is really there (v0.4.72)
Requests to a pod-served model were refused for exceeding the context window. Auto-compaction has shipped for a while, but three things stood between it and doing its job -- and one of them would have made enabling it worse than leaving it off. **The window was wrong.** A pod-served model's `ctx` lives in its runpod block and becomes the pod's `--max-model-len`, but the backend applied the generic `n_ctx` kwarg last -- and that kwarg carries the node-wide `vulkan.n_ctx` fallback, default 2048, whenever the entry sets no `n_ctx` of its own. So a model served with 32768 published 32768 on /v1/models (which reads the block) and reported 2048 in every answer's `usage.context_size`. Compaction compacts to a percentage of that number: switching it on would have crushed every conversation to ~1300 tokens and looked like a model with no memory. The block now wins, for pods and for vLLM, and a per-model `n_ctx` reaches the launch so the reported window and the served one cannot disagree. **The summary described the wrong messages.** It was built from "everything but the last two turns", which includes the turns kept verbatim, so the kept head and most of the kept tail were summarized AND kept -- paying twice in a window that was already too small. The split is now decided once, and exactly the dropped middle is summarized. The kept ends are configurable with it (head_messages/head_tokens/tail_tokens/target_pct, globally and per model), the head is reserved BEFORE the tail is filled (filling the tail first and adding the head after overshoots the target by the whole head), and a head that would end on an unanswered tool call is trimmed -- the mirror of the orphaned-result rule the tail already had. **The refusal was unreadable and final.** The numbers in it are the only exact ones in the pipeline -- coderai estimates prompts at chars/4, the engine has tokenized them -- and they were buried behind an apology, inside the pod's wrapper, inside the orchestrator's, each prefixing its own status line and 120-character proxy URL. A client truncating the result got three layers of prose and neither token count. Every layer now puts what it was told FIRST, the message leads with `context_length_exceeded:` and both integers, and a refusal is used rather than only reported: the real window is remembered for later requests, the estimator is corrected by the ratio the engine just measured, and the request is compacted against those numbers and sent once more (compaction.retry_on_overflow, on by default). Also fixes the layered fit, which mixed its units: the target was computed in real tokens and handed to a splitter that measures in estimate tokens, so a prompt the engine counts as too long could be judged already small enough and left exactly as it was. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
Showing
Please
register
or
sign in
to comment