- 23 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authoredMeasured against the production pod: a reasoning model asked for 64 tokens spent all 64 inside its thought, returned NO content, and the response said finish_reason "stop". max_tokens=64 finish=stop content=0 chars '' reasoning=262 max_tokens=1500 finish=stop content=13 chars 'XMU-5635-FB' reasoning=448 Two layers were each sure the other knew better. The proxy hardcoded finish_reason = "stop" and never read choices[0].finish_reason from the remote; OpenAIFormatter then hardcoded it again, so even a caller that worked the reason out had it overwritten on the way to the client. vLLM had been saying "length" all along. It matters because "length" is the one reason a client can act on. An answer cut off at the limit is half a sentence, and told it stopped normally the client has nothing to retry on and no sign anything is missing. On a reasoning model the thought is generated FIRST and spends the same budget, so a tight limit returns an EMPTY answer that claims to have finished of its own accord -- which is how this surfaced, as a retrieval test scoring zero for reasons that had nothing to do with what it was measuring. The remote's word is now carried through the per-call sink beside reasoning, tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the formatter accepts the caller's reason and keeps "stop" only as the default, and a tool call still wins -- that describes the answer's shape, and a truncation after the call was formed must not relabel it. Fixed at all five response exits, not just the one it was found at: streamed chat (both terminal-chunk paths), buffered chat, and streamed and buffered /v1/completions. A local backend is never told a reason, so the budget decides: generation that used every token it was allowed was cut off, not finished. Drawn only when a limit was actually set and reached -- never from a token count alone, or every long completion would be mislabelled. Tests verified against the reverted files: assert None == 'length' for the backends, TypeError: unexpected keyword 'finish_reason' for the formatter. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
-
Stefy Lanza (nextime / spora ) authored
The pods page drew one bar -- host RAM -- and printed VRAM underneath it as a line of small text. On a page whose job is "what is this rented A40 doing", the GPU was the one reading an operator had to take differently from everything else on the row. Both now get a bar, each labelled, each scaled to its own total. The colours differ on purpose: host RAM keeps the green/amber/red bands, because that is the ceiling that has actually killed pods (an OCR pod was OOM-killed at 128 in-flight with the GPU never the constraint). VRAM gets a neutral bar and says "reserved", because vLLM claims its gpu_memory_utilization share at boot and so reads ~90% from the first second on every healthy pod -- run that through the load bands and every pod is permanently red, an alarm that is always on, which an operator learns to ignore and then ignores the real ones with it. Either bar is omitted when its telemetry is missing: a booting or unreachable pod has none, and an empty bar reads as "idle" when the truth is "unknown". Tested by running the page's own _memCell() in node over real payload shapes. Co-Authored-By:
-
- 22 Oct, 2026 23 commits
-
-
Stefy Lanza (nextime / spora ) authored
boot.sh has printed `profile=… image=…` since the day a pod reported a dependency missing that had already been published under its tag. It prints it before the app exists, though -- to the container's stdout and the pod's own boot log -- and the orchestrator collects a pod's log FROM THE APP, so the file an operator reads begins at "lifespan start" and the build identity is not in it. A pod serving a cached `:latest` could not be told from one serving a fresh image using the log we actually have, which is the question those variables were baked in to answer. Tonight that cost a direct answer: an FP8 KV cache proved the pod was on the new image only by inference -- FP8 cannot initialise without the precompiled kernels the old image lacks -- which is sound but is not the same as reading the tag. The app says it itself now, in its first lines, and stays silent on an install where nothing was baked. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The fleet image and the OCR image were coupled in three places, not one: the profile-parity test, and -- less visibly -- the OCR image's own build check, which read the FLEET's vLLM pin and failed the build on a mismatch. That one meant the OCR image could not be built on a different vLLM at all. They may diverge now. What replaces the coupling is the thing that actually failed last time: not that the images differed, but that nothing said so, and someone had to already suspect it. So each profile must PIN vllm and the torch stack with `==` (no floating version, so two builds of one image cannot differ either), each build check holds its own venv to its OWN pin -- which still catches the resolver substituting a version nobody asked for, a failure the cross-image check could never see -- and a choice the engine would otherwise make by version is named by the image that depends on it. That needed a mechanism, so profiles gain `<profile>.env`: KEY=VALUE baked into the image and exported by boot.sh, which PRINTS each variable it applies and lets the pod's own environment win. The OCR profile uses it to pin VLLM_ATTENTION_BACKEND=FLASH_ATTN -- vLLM's own default on Ampere, named rather than inherited, because FlashInfer is the one backend that image cannot use (its kernels compile on first use and it ships no CUDA toolkit) and which backend gets selected is a property of the vLLM VERSION. A future bump can now change throughput, never whether the pod starts. Also takes one install's model names out of three shipped files. The measurement they illustrate is what matters -- a registration overwritten, a backend fallen through to "cpu" -- not whose models they were. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
They cost ~2.9 GiB and the OCR models never ask for an FP8 KV cache, which is the only thing they are there for -- and that image is already large. So the fleet image carries them and this one does not. The parity test that caught the divergence stays, but now guards the ENGINE rather than the file: every pin deciding which vLLM (and which torch under it) runs must be identical, with this one difference named explicitly and the reasoning recorded beside it. Verified by drifting the OCR vLLM pin and watching the test fail. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
FP8 KV cache halves bytes-per-token, which is the one change that materially raises concurrency on a 48 GB card: FP16 KV on an A40 gave 212,321 tokens for this 27B hybrid (64 KiB/token), so a 128k window fitted 1.62 times. Asking for it failed. vLLM accepted the dtype and resolved it to torch.float8_e4m3fn, then the engine core died during init: FP8 KV on Ampere routes attention through FlashInfer, which JIT-compiles its prefill kernel on first use and calls nvcc. No image here ships the CUDA toolkit (the runtime, yes; and PyPI's nvidia-cuda-nvcc-cu12 wheel turns out to contain only ptxas), so flashinfer/jit/cpp_ext.py get_cuda_path() RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' flashinfer-jit-cache is upstream's package of precompiled modules for exactly this flashinfer and CUDA pair -- 959 of them, 320 with an e4m3 KV cache, head dims 64/128/256 -- and jit/env.py prefers it over compiling, so the engine never looks for a compiler. No toolkit in the image and no build stage; e5m2 is NOT in it, which is why `fp8` (= e4m3fn) is the dtype to ask for. It costs ~2.9 GiB, and a pod pulls the image on every cold start. Added to the OCR profile as well, which does not need FP8: FlashInfer is in the attention path, so prebuilt in one image and JIT in the other is two different engines on the same weights -- the invisible difference that already cost three rounds of blaming that image, and what tests/test_ocr_accent_repair.py exists to prevent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The orchestrator already talks to a pod in parallel, and nothing in the path serializes: the front admits 256, acquiring a model instance is a ref-count, the pool hands the same pod to every caller and only GROWS at the in-flight threshold, the HTTP client keeps 64-256 connections per pod without blocking, and the blocking call runs in a 256-worker executor (measured on the production install; Python's own default there would have been 20). What was not parallel-safe is everything a backend cannot return in its text. `generate_chat` returns a string, so usage, logprobs, the tool calls a remote parsed and the thought it separated travel beside it and are read AFTER the call through `get_last_*`. Those were attributes of the backend -- and one backend object serves every concurrent request for its model. Each call began by resetting them, so a second request arriving between the first one's answer and its read blanked the first one's tool call, or handed it the second one's. That is the same shape as the empty answers seen in production on 2026-10-21, and under real parallelism it is not a rare interleaving but the common case. The store is per CALL now: a context variable, because a request is an asyncio task with its own context and `asyncio.to_thread` copies it, so the worker thread writes into the very dict the awaiting request will read. A stream captures the store when it opens, since a thread inherits no context variables, and accumulates its own thought instead of appending to a shared attribute. The backends keep their attributes as the fallback, so every caller that never opens a store -- the OCR pipeline, a script, a test -- behaves exactly as before. Tests drive two deliberately interleaved calls, and two interleaved streams, through ONE backend and assert neither sees the other's tool call, thought or usage. The suite's autouse fixtures gain one more: a store is request-scoped state, and tests share a context, so it is cleared between them like the template cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The two numbers that decide how much KV cache the share learner must leave an engine were constants in the module that uses them. They are now `vllm.kv_target_gb` and `vllm.kv_min_concurrency`, global or per model in its `vllm` block, on both the Settings page and the model's vLLM section — because the right answer differs per model by an order of magnitude: 2 GiB is correct for a page-at-a-time OCR model and leaves an LLM serving a 128k window unable to hold a single request. Setting the count to 0 opts out and sizes by GiB alone, which is what an OCR engine wants. The cluster page gets the same treatment the RunPod page just had: full-width wrapper and the dense table class, across all nine of its tables. The widest is ten columns, where the default 1rem of cell padding alone was 320px. Long values (a node URL, a capability list) now wrap inside their cell instead of widening the table past the window. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The share learner proposed 0.900 -> 0.460 the moment qwen38 booted on an A40: "13.27 GiB of cache on a 0.900 share leaves room to spare; 0.460 would still give ~2.0 GiB — using it from the next start". 2 GiB of this model's cache is ~32,000 tokens, measured from its own boot report (13.27 GiB bought 212,321 tokens), and the engine was serving a 131,072-token window. That is not one full request, and a vLLM whose cache cannot hold max_model_len refuses to start at all -- so the next start of that model would have failed, climbed the escalation ladder over several six-minute boots, or come up unable to serve the window it advertises. The flat 2 GiB target is right for a page-at-a-time VLM, which is what it was measured for. It is now a FLOOR, raised to whatever holds two full-length requests of the window this engine is actually serving. Bytes-per-token come from the boot report rather than the config, because they depend on the model's attention shape -- this one runs linear attention on most of its layers, at ~64 KiB/token -- and the token count is captured as it streams, like the GiB figure beside it, since both scroll out of the tail before the server is ready. With no report to read the flat target stands: a floor invented from a guess would refuse to shrink anything for ever, which is the waste this feature exists to prevent. A VLM serving 4096-token pages still hands its memory back. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Full width alone was not enough: the default cell padding is 1rem a side, which is 2rem per column and ~384px of pure padding across twelve columns -- by itself enough to push the table into a horizontal scroll on a 1280px laptop. A `.table--dense` class takes that to .4rem/.5rem. And one column was carrying nothing: "Pool" rendered "spot" or "on-demand", which the GPU cell already marks with "(spot)", while the pool's identity is in the Model column (a shared pool's rows carry the pool key there). Dropping it takes the table from twelve columns to eleven. Measured budget: ~1458px of content+padding before, ~1165px after, against a 1232px viewport once the page gutters are taken off -- so it now fits a 1280px screen instead of needing 1500px. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Twelve columns of live fleet state do not fit a 1100px reading column, so the one table an operator scans across had to be scrolled sideways to see cost, health and logs -- and the Served column added to it made that worse. It now uses the same full-width wrapper the Models page already uses, and the two meters give back 2rem each (7rem still shows "100.0/48 GB"). The header row stops wrapping, so a figure is never split across two lines. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Both of these surfaced the moment a config said 131072 while the pod was still serving the 32768 it had been launched with -- a disagreement that is normal for a few minutes after an edit, and which compaction has to survive. **The summarizer was sized from the configured window.** It chunks the history it is about to summarize to fit the summarizing model's context, and that number came straight from the backend's configured value: 131072 against a pod serving 32768, so every summarization request overflowed too. The slicer then caught the failure, left the oversized message exactly as it was, and compaction reported that it could not help. It now prefers a window an engine has already refused against, like the plan itself does. **The head's size cap was waived for its first message.** The intent was to keep at least one anchoring turn, but it meant a conversation opening with a pasted 45,000-token document kept that document verbatim as the head, dropped the rest of the history, and overflowed anyway -- the one shape where compaction matters most. An opening turn too big to keep belongs to the slicer, which summarizes its middle. Also says on the model page what the RunPod "Max context" field actually is: the window the pod is LAUNCHED with, which the general context field above it does not reach, which takes effect on the next pod rather than the running one, and which the first model to create a shared pool sets for every model on it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
With compaction on, a long conversation's streamed reply began with coderai's own progress log -- "
🗜 Compacting context (~45406+300 tok > 120% of 32768) using 'summarize'…", the slice notice, the per-chunk summarizer lines -- and only then the answer. Those went into `delta.content`, which is the text a client stores, displays and builds its next turn on. Observed on the install the moment compaction was switched on. Compaction can take a minute, so the client does need to hear something. The wait keepalive had already answered where that goes: `wait_status_mode`, silent | invisible | visible, per model or globally. Compaction now follows the same setting instead of inventing a second one -- invisible by default (the notice rides the reasoning channel when thinking is on, else an empty-content chunk with `x_compaction` metadata), silent emits an SSE comment that holds the connection without giving the client anything to render, and visible is the old behaviour for someone watching a chat UI. The same block also reported a failure as a successful answer: when compaction could not make the request fit, the error was emitted as assistant content with `finish_reason: "stop"`. The client was told the request had succeeded and handed the error as the model's reply. It is a structured error event now, like every other failure on that path. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Both of these came out of probing the live install rather than the test fixtures, and neither was visible from the code. **A streamed overflow arrived as a successful empty answer.** A remote serving an OpenAI API answers 200 the moment the stream opens, so anything that fails afterwards can only be said inside the stream -- coderai does exactly that, and deliberately, so a client treats it as an error rather than as assistant text. But that event carries no `choices`, and the proxy's SSE reader looked only at `choices[0].delta`: the event parsed to "no text", was dropped, and the stream ended. The client got HTTP 200, `finish_reason: "stop"`, the PREVIOUS request's usage and no content. This was not specific to overflow -- every mid-stream failure from a pod arrived as silence. Both proxies now raise RemoteStreamError, on "present and not None", because `{"error": {}}` is a failure while some servers put a literal `"error": null` in every ordinary chunk. **The refusal parser was written against the wrong wording.** The hand-written fixture used the older OpenAI phrasing ("you requested N tokens"); vLLM 0.27 says "you requested 300 output tokens and your prompt contains at least 32469 input tokens, for a total of at least 32769 tokens", so nothing matched and the message that was supposed to lead with the numbers led with "the request is longer than the context window" instead. The refusal a production pod actually sent is now a test fixture, wrappers and all, alongside the parameter clause ("parameter=input_tokens, value=N") that a build stating only the limit still appends. Verified on the install: `usage.context_size` now reports 32768 where it reported 2048 before, thinking still arrives as its own channel (182 chars of reasoning at 0.3s, content at 2.7s), and a 181,546-character conversation is refused with both token counts inside the first 90 characters. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Requests to a pod-served model were refused for exceeding the context window. Auto-compaction has shipped for a while, but three things stood between it and doing its job -- and one of them would have made enabling it worse than leaving it off. **The window was wrong.** A pod-served model's `ctx` lives in its runpod block and becomes the pod's `--max-model-len`, but the backend applied the generic `n_ctx` kwarg last -- and that kwarg carries the node-wide `vulkan.n_ctx` fallback, default 2048, whenever the entry sets no `n_ctx` of its own. So a model served with 32768 published 32768 on /v1/models (which reads the block) and reported 2048 in every answer's `usage.context_size`. Compaction compacts to a percentage of that number: switching it on would have crushed every conversation to ~1300 tokens and looked like a model with no memory. The block now wins, for pods and for vLLM, and a per-model `n_ctx` reaches the launch so the reported window and the served one cannot disagree. **The summary described the wrong messages.** It was built from "everything but the last two turns", which includes the turns kept verbatim, so the kept head and most of the kept tail were summarized AND kept -- paying twice in a window that was already too small. The split is now decided once, and exactly the dropped middle is summarized. The kept ends are configurable with it (head_messages/head_tokens/tail_tokens/target_pct, globally and per model), the head is reserved BEFORE the tail is filled (filling the tail first and adding the head after overshoots the target by the whole head), and a head that would end on an unanswered tool call is trimmed -- the mirror of the orphaned-result rule the tail already had. **The refusal was unreadable and final.** The numbers in it are the only exact ones in the pipeline -- coderai estimates prompts at chars/4, the engine has tokenized them -- and they were buried behind an apology, inside the pod's wrapper, inside the orchestrator's, each prefixing its own status line and 120-character proxy URL. A client truncating the result got three layers of prose and neither token count. Every layer now puts what it was told FIRST, the message leads with `context_length_exceeded:` and both integers, and a refusal is used rather than only reported: the real window is remembered for later requests, the estimator is corrected by the ratio the engine just measured, and the request is compacted against those numbers and sent once more (compaction.retry_on_overflow, on by default). Also fixes the layered fit, which mixed its units: the target was computed in real tokens and handed to a splitter that measures in estimate tokens, so a prompt the engine counts as too long could be judged already small enough and left exactly as it was. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The RunPod page showed only the instant: in-flight, KV occupancy, queue depth. Nothing said how much work a pod had done, so a healthy pod billed by the hour that had served nothing looked exactly like one carrying the fleet. Each pod now reports the requests it has finished and the prompt/generation tokens that went through it, read from the serving engine's own counters and shown as a "Served" column, with the live fleet's total beside the pod count. Two things had to be true for the number to be worth printing: * prometheus_client renames a Counter. vLLM DECLARES "vllm:num_preemptions" and serves "vllm:num_preemptions_total", alongside a "_created" sample whose value is a unix timestamp. Keying on the declared name alone silently dropped every counter -- so the preemption and prefix figures added in 0.4.69 never matched anything, and the page drew a dash that read as "not supported". Both spellings are accepted now, and "_created" is never read as a count. * vLLM's counters restart at zero with the engine process (model swap, crash, OOM kill), so a pod that had served four thousand requests would report twelve. Each drop is treated as a restart and the previous reading carried forward, which makes the column monotonic for the pod's lifetime. Also drops the prefix hit rate rather than fix its name: in vLLM 0.27.1 the prefix-cache hit counter is registered from the multimodal-cache counter, so a rate computed from it is not the prefix hit rate. A wrong number on an operator's page is worse than a dash. Verified against the shipped image's own prometheus_client, not a hand-written fixture: the exposition names and the "_created" timestamp in the tests are what ghcr.io/nextime/coderai-vllm:latest actually renders. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
digesta-alpha, reproducible on a fresh pod: [boot +0.8s] seeded models registered: qwen38-awq, diogene-alpha, digesta-alpha, qwen38-vision <- correct Resolving model: digesta-alpha [register] digesta-alpha registered in text_models <- teach, overwriting it [lora] added 'digesta-alpha' to the running engine — no reload Loading model on demand: digesta-alpha Warning: No GPU backend detected. Error loading model digesta-alpha: Unknown backend: cpu seed_model_env strips `backend` unless told the engine: a RunPod model carries `backend: "runpod"`, which is THIS install's placement and means nothing on a pod. The SEED path has passed the engine since the same fault bit diogene-alpha (the reason is written beside that call). teach_model never did. So the model was seeded correctly at boot and then RE-REGISTERED by teach -- which happens whenever an adapter has to be sent -- and the good entry was replaced by a backend-less one. It worked until something taught it, which is why it read as intermittent, and why diogene-alpha on the same pod was fine: nothing re-taught it. Orchestrator-side, so a pod already holding a broken registration can be re-taught in place rather than replaced. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The VRAM bar added in v0.4.67-68 reads ~90% on every pod from the moment it boots, because that is what it was told to do: vLLM claims its gpu_memory_utilization share up front and fills what is left after the weights with KV cache. Measured on an A40, 2026-10-22 -- 23.22 GiB weights + 3.49 activation + 2.61 CUDA graphs + 13.27 KV, out of 48, at utilisation 0.9. A bar pinned at 90% forever says nothing about load. (The model IS 4-bit; the quantization is partial by design -- the whole vision tower, lm_head, the linear_attn projections on ~48 of 63 layers and an MTP head stay FP16, which is why weights are 23 GiB and not 14.) What binds instead, and is now what the page shows: * HOST RAM, as the bar. The measured ceiling: an OCR pod went 43 GB free at 32 in-flight, 12.95 GB at 64, and was OOM-killed at 128 while the GPU was never the constraint. Read from CGROUP accounting first, because inside a container /proc/meminfo reports the HOST's memory -- a pod with a 60 GB limit on a 1 TB machine looks like it has hundreds of gigabytes free right until the kernel kills it. * KV CACHE occupancy, with running/waiting beside it: the GPU-side headroom, and what decides how many more requests can be admitted. * PREEMPTIONS, called out in red. The proof the cache ran out: vLLM evicts a running request and recomputes it later, so throughput collapses while every other counter still looks busy. Invisible in utilisation and in VRAM. Reserved VRAM stays as static text under the RAM bar -- context, not a gauge. Also removed a fiction: loaded_info reported `vram_gb: 60.58` and `device: "CPU offload 8.1 GB"` for this model. Nothing measured that -- coderai never loaded it, vLLM did, so it fell back to a config estimate for an unquantized load that never happened. A UI claiming 8 GB of CPU offload where there is none is worse than showing nothing; it now says "served by its own engine". Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.4.67 asked each pod for /admin/api/gpu-stats. Pods answer 404: a pod runs the SLIM app — /healthz, /v1/models, /internal/engine-state — and no admin API at all. Built on an assumption; the live check caught it. Probe the pod before choosing its endpoint. /internal/engine-state already carries a `vram` block: used, free and total per card with the GPU's name, built context-free and cached 4s there. It is what the front health-polls, so every pod has it and it costs the pod nothing extra. Verified against a live A40: 43.8/48 GB with qwen38 resident. The pod reports FREE, not used — that is what a context-free query returns — so used is the difference. Inverting that would show an idle pod as full. Utilisation and temperature added AT THE SOURCE: nvidia-smi now asks for utilization.gpu and temperature.gpu in the same call as the memory query rather than paying for a second one (a saturated card answers slowly enough that one call matters), with the pynvml equivalent, and both ride into engine-state's vram.devices. Absent or "[N/A]" is left out, never zeroed: a zero draws a bar that reads as "idle" when the truth is "not known". So VRAM works on every pod already running; utilisation needs the next pod image, and until then that column shows a dash — which is the degradation the page was built for. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The page listed what each pod COSTS and nothing about what it is doing, so a saturated pod and an idle one looked identical next to the same $/hr. Read from the PODS, not from RunPod. A pod runs coderai, so it already serves /admin/api/gpu-stats -- per-card util, VRAM used/total and temperature, collected with nvidia-smi off its own event loop and cached there for 1.5s. RunPod reports utilisation too, but it lags by tens of seconds and every read spends the quota that renting and boot-polling need: a 429 there is what stopped the fleet scaling for 3.3h on 2026-10-19. Telemetry must never be able to cost a fleet its ability to rent. Utilisation and temperature are the BUSIEST card, not the mean -- on a multi-card pod the saturated card decides whether it can take more work, and averaging it with an idle sibling hides exactly that condition. VRAM is summed, being the pool a model is loaded into. Bounded everywhere, because this is the page an operator opens when something is already wrong: 2.5s per pod, all pods concurrently, a 4s TTL so several open tabs collapse onto one round, and a 3s cap on the round -- a slow pod drops out and the rest still render. The spend figures beside it stay right when the telemetry does not arrive, and a pod that is booting or unreachable shows a dash, never a zero: a zeroed bar reads as "idle" when the truth is "unknown". held_pod_endpoints() is deliberately separate from pods_status(): that snapshot is serialised into a browser and a pod's bearer must not be. Covered by a test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`enable_thinking: false` came back still thinking, verified on a pod running the v0.4.65 image with --reasoning-parser qwen3 (the other four checks passed: reasoning_content populated, content clean, and streamed thinking starting 0.4s in against the answer at 2.8s). generate_chat negotiates optional kwargs by calling the backend and retrying on TypeError. Every variant that carried `passthrough` ALSO carried `enable_thinking`, which a proxy backend does not accept -- so those variants raised TypeError, and the variant that survived had dropped the passthrough with it. A caller's `chat_template_kwargs` therefore never reached a pod at all. `enable_thinking` was merely the visible case: OCR's `continue_final_message` would have been dropped on a pod the same way. `_lp` and `_pt` are already SIGNATURE-gated -- empty for a backend that cannot take them, so they never raise TypeError. Only enable_thinking and the penalties are negotiated that way, so only those are peeled off now, and the passthrough is dropped last rather than first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.4.64 aimed one layer too high. Measured against the pod directly: content = 'We need answer in Italian... 17 x 23 = ... **391**.' reasoning_content = '' has </think> = False finish = stop The thought and the answer come back as ONE string with no separator of any kind. </think> is a SPECIAL TOKEN and vLLM decodes with skip_special_tokens, so the boundary is deleted before the text exists. No parsing downstream can split on a boundary that was removed -- the earlier reading, "extraction fails on the bare </think>", was simply wrong. coderai never passed --reasoning-parser to vLLM, for any model, ever. With it vLLM splits the two at the TOKEN level, before the tag is discarded, and reports reasoning_content as its own field; the v0.4.64 side channel and delta re-tagging then carry it to the client. Chosen from the template like the tool parser, and only for a model whose template says it thinks -- a reasoning parser on a model that does not think waits for a tag that is never coming. An explicit vllm.reasoning_parser wins, "off" included, per model or globally. AND A REGRESSION FROM v0.4.64, fixed here: with the gate switched on and no close tag ever arriving, a streamed reply was held back in full as "thinking" and the client was sent an empty message -- worse than the leak it replaced (reasoning=229 chars, content=0, measured on diogene-alpha and qwen38 alike). An unclosed thought now flushes as CONTENT. A thought truly cut off mid sentence then reads as the answer, which is untidy; an empty answer is a lost reply. With the server separating the channel this path stops being reached. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
From ONE pod, ONE set of weights, behind two aliases: qwen38 -> clean answer diogene-alpha -> "The user is asking to list hearings from 2026-10-21 to..." The entire chain of thought went to the client as the answer. Four breaks: 1. THE GATE ASKED THE NAME. _reasoning_active was a regex (qwen3|qwq| deepseek-r1|...) over the request's model name: `qwen3` matches "qwen38" and not "diogene-alpha", so one alias of one model separated its thought and the other did not. It is read from the template now -- whose generation prompt ends with '<think>\n', so the model starts INSIDE the thought and emits a bare </think>, the form that leaks when nothing looks for it. The regex stays as a last resort for a template that cannot be read. 2. THE BUFFERED FORMATTER TOOK NO `reasoning`. Where the thought WAS separated it was then dropped: no leak, but no thinking either. litellm's Message has no such field, so it is attached after the dump rather than lost -- under both spellings, since Qwen's own template reads `reasoning_content`. 3. THE PROXY HOP CARRIED NEITHER FORM. Buffered, the remote sends reasoning_content beside an already-cleaned content, which the string contract cannot hold: get_last_reasoning(), like get_last_tool_calls(). Streamed, it sends reasoning DELTAS, and dropping them meant a client saw nothing at all until the answer began -- so the <think>/</think> tags go back on as they arrive and the caller's existing reasoning gate splits them live. An all-thinking stream leaves the tag open on purpose: the gate flushes an unclosed thought as reasoning, and there is no content for a close tag to precede. 4. THINKING COULD NOT BE TURNED OFF. enable_thinking defaulted to False, so "unset" and "explicitly off" were the same value, and the field sat in _CHAT_CONSUMED_FIELDS while no backend forwarded it. It defaults to None now: unset sends nothing and the model keeps its own default (a thinking model thinks), an explicit value travels as chat_template_kwargs. That needed the POD backend to gain passthrough at all -- the local vLLM backend has forwarded unknown fields for releases, a pod got none of them, so chat_template_kwargs could never reach a pod-served model. ChatMessage silently dropped `reasoning_content`, so a previous turn's thinking never reached the template that re-renders it inside <think>. Accepted and forwarded now. The chat page hardcoded stream:false, so thinking could never arrive live there, and it read only message.content. It has `stream` and `thinking` toggles (both on), renders the thought in a block that fills AS IT ARRIVES and collapses when the answer starts, and keeps it in the history it replays. Streaming is the request's choice, not forced: cleared, it takes the buffered path exactly as before. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The admin model dialog has had a "Chat template parser" dropdown all along (auto/llama/mistral/chatml/phi/gemma/qwen). It is saved on the model entry, and seed_model_env carries `parser` in its keep-list so it travels to rented pods. NOTHING EVER READ IT: choosing a parser for a model changed nothing anywhere, and selection stayed a substring match on the name -- with a second, separate guess in the browser (_autoDetectParser). A tool call is text in the format the model was trained on, and the chat template is where that format is written down, so the style belongs to the model. It now resolves, strongest first: 1. the model's own `parser` setting -- that dropdown now does something; 2. the global default, read the way resolve_gemma_tool_mode reads its own; 3. the model's chat template; 4. the name. The template reading moves out of vllm_worker into codai/models/chat_style.py, with ONE style table mapping each style to both consumers -- vLLM's --tool-call-parser and coderai's own parser class -- so the two choices cannot drift apart. Qwen XML, Hermes JSON, GLM <arg_key>, Mistral [TOOL_CALLS], Llama <|python_tag|> and DeepSeek's call tokens. Cached per model path: a model's template cannot change under us, and this is consulted per request on some paths. The template does NOT override a name that matched a family. A per-family parser carries more than the tool syntax -- gemma's token repair, Qwen's repetition guard -- so one template marker should not displace a family that matched. It decides where the name places the model in no family, which fell to the catch-all before: _family_name only resolves an alias for an ADAPTER, so a plain alias like `lisa` named no family and got the generic parser even though its entry says exactly where the weights are. A value the dispatcher does not know falls back to detection rather than silently disabling tool calls, and an unreadable template changes nothing -- not knowing is survivable, inventing a parser is not. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The turn AFTER a tool call failed, with the client replaying the assistant's tool_calls: the pod answered 500: 400 Bad Request ... 3 validation errors: {'type': 'string_type', 'loc': (..., 'ChatCompletionMessageFunctionToolCallParam', 'function', 'arguments'), 'input': {'dal': '2026-10-21', 'al': '2026-11-21'}} coderai hands its backends `function.arguments` as a MAPPING on purpose: GGUF chat templates render it with `|items`, and llama.cpp raises "Can only get item pairs from a mapping" on a string (_normalize_tool_call_arguments). The OpenAI schema says string. Both are right for their own side, so the conversion belongs at the edge where a request stops being a template's input and becomes another server's HTTP body -- in the proxy backends' _chat_payload: runpod, vllm, and ds4/ktransformers, which post the same body to a local server. The three validation errors are one cause: vLLM's union arms each rejecting that field. Absent arguments go out as "{}", not "null", which a server would try to read as an argument object. Nothing to convert returns the caller's own list unchanged -- every request passes through here, OCR's page-per-request traffic included. Only reachable once a client replays an assistant turn that HAS tool calls, so it was unreachable while those calls were being dropped (v0.4.60). Co-Authored-By:
-
- 21 Oct, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
Found while tracing the teach path: _teach read `pool.api_key`, the same mistake the v0.4.59 401 was about. Pods in one pool can carry different tokens, so the pool's key 401s against any pod that does not share it -- and the request is then lost to the very 404 that teaching was meant to cure. The bearer is handed out with the address, so it is passed through. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod serving Qwen3.8-27B logged, on every single tool-using request, hermes_tool_parser ... json.decoder.JSONDecodeError: Expecting value on text that begins <tool_call> <function=agenda_udienze> <parameter=dal>2026-10-21</parameter> vLLM reads that with `qwen3_xml`; `hermes` expects JSON inside <tool_call> and cannot. Buffered, coderai's own parser rescued the call out of the text, which is why this stayed invisible. STREAMED, vLLM had already consumed those tokens into tool-call deltas it then failed to emit, so the call was simply gone and the stream ended at </think>. A name cannot separate the two syntaxes -- Qwen2.5 and the Hermes-style finetunes genuinely are hermes, and qwen3_xml does not read JSON at all -- so the mapping cannot just be changed. The model's TEMPLATE decides instead: it is the model's own statement of what it emits, and this one says "If you choose to call a function ONLY reply in the following format: <tool_call>\n<function=example_function_name>\n<parameter=...>" with no JSON form anywhere in its 8,952 bytes. Read, not guessed, and never fatal: a repo id is fetched rather than looked up in the cache (on a rented pod nothing is cached the first time), an unreadable template leaves the name table's answer alone, and every failure path returns "" -- a pod that will not launch is worse than one that reads tool calls with the wrong parser. An explicit tool_call_parser still wins, "off" included. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every tool-using request to a pod-served model answered an EMPTY assistant message: content "", finish_reason "stop", no tool_calls. Asked directly the pod answered the call it had generated; through the orchestrator the same request answered nothing. Both the LoRA (diogene-alpha) and the base model were affected, so no adapter was involved. A proxy backend's generate_chat returns a STRING. The remote server owns the chat template, so it is the side that recognises a tool call and reports it structurally -- `{"content": null, "tool_calls": [...]}`. `message.content or ""` dropped that on the floor; the caller then had an empty string, found no call to parse in it, and answered the client with an empty message. The calls now travel beside the text, where usage and logprobs already do: get_last_tool_calls() on the backend, forwarded by the manager, preferred over re-parsing text in both the streamed and buffered chat paths. A backend that generates locally still keeps its call inside the text and answers None, so that path is untouched. Streaming needed the same: a streamed call arrives as deltas with no content at all, so merge_tool_call_deltas folds the fragments -- keyed by the call's index, never by arrival order, since concatenating the arguments of the wrong call yields valid JSON for a call nobody made. Co-Authored-By:
-
- 20 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
2026-10-20, qwen38: a restart generated a fresh random pool token (none is configured), adopted a pod still carrying the old one, and rented a second pod with the new one. A pool has one api_key, so whichever token it held answered 401 on the other pod — "Invalid API key" on every request until both pods were destroyed. The adoption guard ("only swap while we hold no pod") could not help: the rent and the adoption happened in the same minute. - PodHandle.api_key: the token THIS pod was launched with, or adopted carrying. pool.pod_key(pod) resolves it, and the pool remembers the tokens of pods it no longer holds so a quarantined one stays reachable. - Adoption keeps the adopted pod's token on that pod instead of replacing the pool's, so pods with different tokens coexist. - _acquire_endpoint returns (base, release, bearer): the key is handed out WITH the address. Reading it afterwards from the pool was wrong twice — the pool's token could change mid-acquire, and _last_pod is one attribute shared by every concurrent request, so it could name another request's pod. - The OCR worker's majority vote over the registry's keys is gone; each pod's key rides on its own handle. - The pool's own token is minted once and persisted beside the pod registry (0600), so a restart no longer orphans every pod it had rented. A configured api_key still wins, and an unwritable directory still serves. Also tried and reverted: treating an orphaned mid-boot pod as "on the way" to stop a restart renting beside it. That reintroduces the twenty-minute freeze of test_runpod_restart_orphans. Renting twice costs cents; a pool that will not rent serves nothing. The reasoning is recorded in sibling_is_provisioning. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
RunPod's live GraphQL limits for our key are 180/min, 3,600/h and 43,200/day. The day limit is the one that ran out, and the ban lasts until midnight UTC. Averaged over a day it allows 1,800/h, so the hourly cap is not the binding one. This process now counts its own GraphQL calls over a rolling hour. Past CODERAI_RUNPOD_GQL_HOURLY_BUDGET (default 1500; 0 = off), _gql_banned() reports true, so everything REST v2 can do goes to v2 before RunPod ever answers 429. The budget only reroutes. It never refuses a GraphQL call on its own, so spot rentals, which v2 cannot do, still work. The first time the budget is reached, and every 10 min while it stays reached, it says so. Co-Authored-By:
-
- 19 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
Verified against RunPod's v2 OpenAPI spec and live headers on 2026-10-20. GraphQL quota is 180/min, 3,600/h, 43,200/day; ours read x-ratelimit-remaining-day 0 with Retry-After 3939. REST v2 has a separate per-user quota (180/min, 7,200/h, 86,400/day) and read "day";r=86399 at the same moment. v2 then terminated a pod (204) that REST v1 (500) and GraphQL (429) could not. v1 is deprecated and retires 2026-11-15. - All REST traffic now goes to https://api.runpod.io/v2: get/list pods (with pagination), terminate (DELETE, 404 counts as gone), create (POST /v2/pods with gpu{id,count}, cloud, disk, ports, env, entrypoint/cmd, dataCenterIds, mounts persistent/network/global), network volumes, and the priced GPU catalogue. - v2-first with GraphQL as fallback for the high-volume reads (status, listing, boot polling) and for terminate. GraphQL stays primary for create and the catalogue because only it offers spot bidding and spot prices. They switch to v2 during a GraphQL ban. v2 refuses spot rather than renting it on-demand. - v2 phases map to the pool's words (PROVISIONING/STARTING -> RUNNING, ERROR -> FAILED). Runtime memory and CPU utilisation are kept per pod. - v2's own 429 and Retry-After are honoured, and its live RateLimit header is kept. - The global volume is the documented mounts.global on v2. A refused one is retried without it, the same guard as the GraphQL path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Observed through the 3.3 h ban: REST reads kept working, but every REST write (create, and a body-less DELETE) failed with 500 "unexpected end of JSON input". REST writes appear to go through RunPod's GraphQL backend and share its rate limit, so REST cannot rent around a ban. The fix is not to earn one. - wait_ready polled GraphQL get_pod every 5 s per booting pod, which is hundreds of calls a minute in a 64-pod scale-up. Direct-TCP pods (the default) now poll REST and open a TCP connect to the mapped port. RunPod refuses that connect until the server inside listens, which is a stronger signal than GraphQL listing the port at container start. GraphQL stays the fallback outside a ban. - list_pods (the reaper and the survivor sweep, every minute) is REST-first, with includeMachine=true. Card name and VRAM come from the catalogue. - GraphQL calls are counted by operation and the rate is logged every 10 min, so the volume is visible before RunPod bans the key. - A REST write that fails this way during a ban says why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod rate-limited GraphQL for ~3.3 h on 2026-10-19 while REST kept answering. Every pool operation stopped: renting, the GPU catalogue that prices cards, watching a pod boot, the reaper, and teardown. While the ban (v0.4.54's backoff) is in force: - create_pod goes to REST. That fixes _create_pod_rest, which used direct_tcp and global_volume_id without accepting them, so it had never worked. - get_pod, list_pods and terminate_pod use REST and return the same shapes. REST portMappings and publicIp become the GraphQL-style ports list. A 404 on DELETE means the pod is already gone, which counts as success. - list_gpu_types returns the last catalogue saved to disk. It is now saved beside the pod registry on every successful fetch. An unpriced card is still never chosen, so ceilings hold. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Measured on a rented A40 under 128 pages in flight. The engine stopped answering. _ensure_service_once treated any service that failed a 3 s health probe as gone: it dropped it from the table, without stopping the process, and launched another. A second caller during a start did the same, so two EngineCores came up one second apart. For twenty minutes vLLM was relaunched every few seconds and killed with -9 mid-start. This is also the earlier "10.22 of 47 GiB free" crash. - A service records "starting" until it is ready. Another caller waits for it (_StillStarting -> _wait_for_start) instead of launching a duplicate. - A live process that misses the probe gets a second 15 s probe. If that fails too it is BUSY and is handed out, not replaced. - A dead one is stopped properly before it is replaced. - The OCR gateway's /health no longer runs placement.any_candidate (which can start vLLM) on the event loop. Every surya engine probes it, and it froze a pod for up to 408 s. The no-blocking-calls guard now covers placement.any_candidate and placement.candidates, and fails on the old code. - RunPod 429s are honoured: retryAfter is in seconds (observed 11,871, about 3.3 h), and no GraphQL call is made until it passes (capped at 6 h). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A page holds its thread for the whole read, a minute or more. _Pool.recognize used asyncio.to_thread, which puts that wait on the default executor: capped at 256 and shared with model lookups and the admin pages. On a rented A40 the engine had 512 lanes and vLLM sat at 9% KV cache, so the thread pool, not the GPU, set the pod's ceiling. Past 256 the model lookup each page's chat request needs (off the loop since v0.4.52) would also have queued behind the page waits. Page reads now use the pod pool. It is sized from ocr.max_concurrency (up to 16384, within the task budget), and on a pod nothing else uses it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The v0.4.51 stack dump named the OCR pod freeze. chat_completions called multi_model_manager.get_model_for_request() on the loop. For a model not loaded yet that goes _load_default_model -> load_model -> vllm_worker.ensure_service(), which waits for vLLM in a time.sleep(2) loop. On a pod it froze the whole process for 324 s at boot, /healthz included, and for minutes whenever it decided to (re)start vLLM, so the orchestrator took busy pods for dead ones. chat_completions and completions now resolve the model in a thread. The manager's load lock already prevents duplicate loads. The no-blocking-calls AST guard now also forbids get_model_for_request, _load_default_model and ensure_service in a coroutine; it fails on the old code and passes on the new. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Rented OCR pods froze for 108-784 s under load ("[loop] blocked for 784.6s"), and there is no shell on a pod to attach a profiler. A daemon thread now watches the sampler's heartbeat. After CODERAI_LOOP_DUMP_AFTER seconds (default 3) of silence it prints the loop thread's current stack, again every 30 s while the stall lasts, at most 20 per hour. It exits when its loop closes. If the top frame is the selector, the loop was starved of the GIL rather than blocked in a call. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
require only bound the lease. A page that timed out on a pod, or a pool that refused, still sent the whole document to the local pipeline. Live, that was 4 surya processes at ~9 GB each and swap full on a GPU-less orchestrator, with those pages slower than a retry on a pod. Under require, that case is now a 503, so the client retries and the page goes back to the cards. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A single pod whose vLLM failed to start (another vLLM still held 37 of its 47 GB) answered 503 to every page. _read_on_pods set the whole pool's pod pipeline aside for 300 s on that one answer, so every page in that process was read on the orchestrator. Live, that meant ~10 pages/s from 34 pods, with the orchestrator's CPU doing the reading. Now a refusal takes that pod out of rotation (lease.dead -> quarantine) and the page tries another pod. The pool is set aside only when a page's attempts were refused by at least two different pods, i.e. when it looks like the image and not one card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
One Python process forwards ~16-23 pages/s however many pods it has. Measured on a 35-pod fleet: front 99% CPU, engine 101%, 14 of 16 cores idle, pod GPUs at 0-7% KV cache. 128 pods at ~0.65 pages/s each need ~85. ocr.workers (or CODERAI_OCR_WORKERS) = N starts N --engine-only processes (codai/ocr_fleet_workers.py, a supervisord program) and writes nginx includes that send exact /v1/ocr and /v1/ocr/batch to them, least_conn on a shared zone, front as backup. Everything else still goes through the front. 0 (the default) leaves routing exactly as before. The main engine stays the only process that rents, scales, quarantines, adopts or terminates. Workers (CODERAI_OCR_WORKER=i/N) follow the pod registry, take a balanced sticky slice (runpod_fleet_share.assign), and report per-pod in-flight/used/suspect back. The controller merges those reports into every scale/idle/teardown decision through _load(). Drains go to workers as flags. Each worker takes 1/N of the per-process ceilings (max_concurrency, max_queued_pages, *_instances, rounded up) and never writes config.json or models.json. nginx retries another worker only on a connect error, never on a timeout (that would read a page twice, billed twice), with no try cap so the backup is reachable. worker_connections 1024 -> 16384. Co-Authored-By:
-