- 11 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe gateway built an httpx.AsyncClient for every request, so each page paid a fresh TCP and TLS handshake to the upstream. With surya_instances: 24 that is 24 streams reconnecting per page, against a round trip that already dominates the cost of reading one. One pooled client with keep-alive instead, so a document's pages reuse the connection. Correctness-neutral; it only ever cost time. Keyed by the event loop, because an AsyncClient binds to the loop that created it and coderai runs more than one (the OCR pool keeps its own). The key is the loop OBJECT held weakly, not id(): keying by id would leak an entry per dead loop and, since CPython reuses addresses, could hand a new loop a client bound to the closed one that used to live at that address — a fault that would only appear under the load this change exists to handle. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
-
Stefy Lanza (nextime / spora ) authored
The fan-out gateway lived at /v1/ocr/engine/<lease>/… and the new catalogue at /v1/ocr/engines. They do not collide, but both auth gates exempt the gateway by PREFIX — so any later route under /v1/ocr/engine* would have inherited that exemption silently, and the two names are indistinguishable at a glance in a middleware condition. The gateway is now /v1/ocr/attach/<lease>/…: internal by name, and nothing a public path family will grow into. Tests assert /v1/ocr/engines still requires ordinary authentication at both gates, and that a plausible future sibling (/v1/ocr/engine-stats) does too. Free to rename because nothing consumes the lease URL yet — it is minted per attachment at run time and never written down. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Naming an engine an install had not enabled was a 400, and there was no way to find out beforehand — the client had to know, or guess and handle the refusal. GET /v1/ocr/engines answers it from this install's own config: every engine with `enabled` and, when it is not, a `reason` that names the setting to change (an unaccepted licence is reported as such rather than as merely disabled, because that is a different checkbox). Each row also carries `kind` — an engine owns its weights, a pipeline drives a model — the `aliases` that also resolve to it, and `returns`, so a caller that needs coordinates can see that olmOCR has none rather than discovering it in an empty field. The top level reports both what the operator wrote as the default and what it resolves to. Where the model runs is deliberately not in the answer. It is configuration, it can differ between two requests, and a client that could see it would start depending on it. A test asserts the payload mentions no pod, node or address. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docTR called .cuda() when torch happened to see a card and otherwise simply stayed on the CPU, so a GPU-less install read every page many times slower with nothing in the response to say why — the result looked normal, it just arrived minutes late. Nothing else in coderai quietly relocates GPU work, and paddle refuses outright in exactly this situation; docTR was the odd one out. `doctr_use_gpu` is now a requirement: no CUDA device means a 503 that names the setting. Running docTR on the CPU is still fully supported and unchanged — it just has to be asked for, with doctr_use_gpu = false. The test drives the real load() with docTR and torch stubbed, rather than re-implementing the placement rules beside it; reintroducing the fallback makes it fail, which is the only reason to trust it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three corrections to the fan-out, all the same mistake in different places: the code was deciding things the configuration had already decided. Nodes are no longer discovered. Asking every cluster node what it holds and using whoever answered is configuration by probe: it puts work on a machine nobody chose and moves it the moment a different node replies first. A node serves a model because the model is PINNED to it — `engine` on the entry, the same hard constraint the front router already honours, comma-separated when several nodes should share the work. A pinned node that does not hold the model is reported, not silently replaced. The decision is not an OCR decision, so it does not live in codai/ocr: any model, any caller that needs an endpoint rather than a proxy hop, gets the same answer from codai/models/placement.py. For requests that pass THROUGH coderai this was already true — the router honours the pin, treats a node as an engine by name, reaches RunPod through the backend. This is that decision for a caller that must hold an address itself, which is why OCR needed it. And "how busy is too busy" is the model's own scale_up_inflight_per_pod, on the model page, instead of the parallel ocr.serve_spill_inflight knob I had added — one number per model, already there, already meaning exactly this. A model with no RunPod block has nothing to rent, so load alone never moves its pages. Finally, the caller: resolve_engine accepted the four pipeline names only, so a client naming the model it can see in /v1/models got "unknown OCR engine". Both now work — engine=surya and engine=datalab-to/surya-ocr-2 reach the same place — because which pipeline reads a page, and where its model runs, are ours to know and not the client's. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An OCR pipeline is handed one server address and drives it itself, which is what keeps its page images, its strict json_schema and its logprobs intact. One address cannot spread load, and three things followed from that: * a pool grows on in-flight requests counted as they pass through it. An attached engine never passed through, so the pool saw no load however hard OCR ran, never grew, and the engine had one pod's URL anyway; * several cluster nodes serving the model resolved to whichever answered first, for ever, however busy it became; * a local model configured to burst to RunPod never did: that trigger lives on the proxy path an attached engine bypasses. So the engine gets a lease on an address of ours, and codai/ocr/dispatch.py chooses per request: a pool that is the model's configured home takes every page (picking the least-loaded pod, renting when they are all at the threshold); otherwise this machine and the nodes holding the model compete on load — including the load a node reports for its whole install, so a node busy with someone else's work is left alone — and the overflow past ocr.serve_spill_inflight rents instead of queueing. An explicit `backend` pin still wins over discovery: that was a decision, and load-aware picking must not quietly repeal it. The hop is allowed to exist only because it decides nothing: body bytes in, body bytes out. The losses it is there to avoid came from PARSING a request and re-emitting it through the backend abstraction. Two gates stood in the way and neither could be satisfied the usual way: the engine has no API token, and surya's /health and /v1/models probes send no headers at all (bare httpx GETs), while refusing to attach if /health is not 200. The lease is therefore read from the URL, and the engine-only internal gate exempts that path alone — tested to be no wider than that. olmOCR's model mode asks the same dispatcher, and still takes the in-process route when the answer is local, where the model manager's loading, eviction and queueing belong. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Surya asks every request for logprobs and averages exp(logprob) over the generated tokens to score each line. coderai accepted the parameter, never sent it, and hardcoded `logprobs: None` in every response builder. Surya's fallback is what makes that expensive: with no logprobs it does not report "unknown", it reports a literal 1.0 (recognition/__init__.py:248, :302, layout/__init__.py:97). Every line of every page comes back perfectly confident, the guesses included, and nothing downstream can tell which judicial document needs a human to look at it. So the round trip is wired where it can exist: the proxy backends (vLLM, RunPod) forward `logprobs`/`top_logprobs` and hand back what the server measured, buffered and streamed, accumulating the streamed pieces into one response-shaped object. The manager asks the backend's signature rather than negotiating by TypeError, which would have called it twice for a kwarg it never had. A request that did not ask is sent exactly as before, and a backend that cannot answer still returns null. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Everywhere else in coderai an engine is what RUNS a model. `ocr.default_engine` borrows the word for two different kinds of thing: paddle and docTR are runners with their own weights, while surya-2 and olmOCR are pipelines over a model that vLLM runs. Only the second kind has to be told where a server is — and that is the same question the model layer already answers for everything else. So answer it in all the places a model can actually be served. A cluster node was the one that was missing: a node is a whole coderai registered with the head as a remote ENGINE, so nothing in the backend layer knows it exists, and a VLM hosted there came back as "not served over HTTP" from an install that was serving it perfectly well. The node's own catalogue is the authority, so it is asked. An explicit backend pin still wins — discovery is the fallback, not an override — and the nodes are not probed at all when clustering is off, which would have added a timeout to the first page of every document. Also: one test here was passing for the wrong reason. `from codai.models import manager` reads the package attribute and never consults sys.modules, so a fake installed there was simply not seen. Patched as attributes of the real module, which is what made the precedence test fail honestly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three silent drops on the way to a rented pod, all found while wiring Surya to the model layer, all of the same shape: coderai accepted a field, forwarded a reduced version of it, and the pod answered something plausible. * Images. The front flattens multipart content to the literal text "[image_url content]" unless the backend claims vision, and only the llama.cpp backend ever did. So every page sent to a VLM served on RunPod — olmOCR in "model" mode, or Surya through a coderai-vllm pod — arrived as that placeholder, and the model answered fluently about nothing. A proxy is the wrong place to judge: forwarded, a text-only server returns a plain 400 and a VLM server reads the page. * Structured output. Only {"type":"json_object"} survived, so a json_schema — the strict form current clients send, Surya's guided decoding among them — was dropped and the reply came back as free text the caller could not parse. The streamed path ignored the field entirely, json_object included. * extra_args. Whitelisted per engine on a model's runpod block since pods shipped, parsed, stored, and never put on a command line. Setting it looked like tuning and did nothing, which is worse than refusing it: the pod boots, serves, and ignores the decision. A page-reading model wants prefix caching and the Qwen-VL pixel bounds, and until now the only way in was docker_args, which replaces the whole generated line. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
OCR on the Digesta install answered 503: "vLLM isolated venv not found at /cache/vllm_venv". That host has no GPU at all — renting A40 pods is its whole job — so the venv it was told to build could never have run, while the Surya-2 checkpoint it needed was already a configured model (backend: runpod, engine: coderai-vllm) nothing in the OCR path could ask for. The model layer was never the problem; `ocr.surya_serve` was asking the wrong question. It enumerated server PROCESSES — torch in a local venv, our own vLLM, a URL typed in by hand — when the answer the operator had already written down was a MODEL. olmOCR has had that bridge since it shipped ("model" mode, through our own chat path); Surya never got one, because Surya2 drives its server directly with guided-JSON decoding and logprobs that a proxy hop would drop. So it gets the same bridge, in the only shape it can take: resolve the model through whichever backend owns it and hand the worker the address. Three values, because Surya refuses a server whose /v1/models disagrees with the checkpoint it expects, and a pod URL is useless without its token — which only Surya's vLLM client reads, so a bearer-locked endpoint is always attached to as "vllm". The address is a lease, not a location: a min_pods=0 pod is reaped when OCR goes quiet, and the page that finds it gone re-resolves once — renting a fresh one — instead of every later page failing against a dead URL. A failure against a server that still answers /health is a real error and is reported as one. Co-Authored-By:
-
- 10 Oct, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
When zeiss ran out of memory this morning the OOM killer picked coderai-nvidia. Not because it was at fault — a runaway test process was — but because oom_score is essentially "how much memory does this task hold", and an engine with a model loaded is always near the top of that list. The leak carried on; the thing that was serving died; the box needed a hard reset. Linux offers one lever, /proc/<pid>/oom_score_adj, and two rules that decide where it can be pulled from. Both were verified on the two production hosts rather than assumed: raising is free, lowering needs CAP_SYS_RESOURCE — and a docker container does not have it (tested: in-container `echo -500` is denied, `echo 200` succeeds). So the protected base has to come from the engine daemon, via --oom-score-adj on the run, and everything inside can only step back UP from it. rootless podman cannot protect at all: a negative value is clamped to 0. 0 is still worth having on the Digesta host, where coderai currently sits at +200 — inherited from systemd's user manager, which makes it a PREFERRED victim with a kernel score of 800 against zeiss's 670. So: the container gets -500 (CODERAI_OOM_SCORE_ADJ, 0 disables), and inside it codai/util/oom.py states the order as three tiers relative to that base — engines +100, workers +300, training +500. Relative, because the base differs by host and a tier should keep its distance whatever it turns out to be. Engines are marked in the existing _engine_preexec hook, which covers the isolated model workers too: they are spawned by the engine, not the front, so they inherit it. Training is marked from the parent after Popen — a preexec_fn would run Python between fork and exec in a threaded server, which is the documented way to deadlock. The ordering is the point. Protecting everything equally would just move the bullet to an innocent bystander; this way the kernel sheds a training run (hours, restartable) before an engine (loses its in-flight work, and the front notices and respawns it) before the front (loses everything, including the ability to respawn anything). Every helper can only ever make a process MORE killable: nudge() refuses a negative delta rather than attempting a privileged write, because a process that believes it is protected and is not is worse than one that knows it is exposed. All of it is best-effort — a process that cannot adjust its score still serves. 16 tests, including that the number reaches the kernel and moves the score, that a child can be marked from the parent, and that the engine mark sits inside a try (it runs between fork and exec, so anything that raises there fails the spawn). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
zeiss came back from this morning's hard reset with no container at all, and the hourly upgrade then skipped every tick — so the host sat on old code and, worse, stayed DOWN, with nothing in the loop that would ever have recovered it. Both idle probes are HTTP, so an absent server is indistinguishable from a wrong token: inflight() said "unknown" and the fail-closed guard refused to act. The Digesta host, whose container was up, upgraded itself fourteen seconds after the push; zeiss needed hands. container_state() answers the one question the probes cannot — is there a server at all — by name, then by any running container of the image being upgraded. Only a POSITIVE "not there" counts as idle: an engine that cannot be asked (daemon down, wrong socket) stays unknown and still fails closed, because the point is to recognise an absent server, not to assume one. Checked before the token, since a host with no server is in the same position as one with no token. Both installs run their engine's ps as the service user and name the container `coderai`, so the default matches; CONTAINER_NAME covers an install that does not. The second half: every knob reads `VAR="${VAR:-default}"`, which looks like the environment may override it, but the conf is sourced first and assigns unconditionally — so `MAX_DEFERRALS=1 coderai-autoupgrade` did nothing at all, with no line in the log to say why. I hit that landing this morning's restart and had to source the conf from a wrapper to get round it. The environment is now snapshotted before the conf and restored after, for that run only, and a test checks the snapshot list against every knob the script resolves so the next one added cannot quietly fall out. require-idle and max-deferrals are also logged now: "it skipped again" and "it was told to skip" used to look identical. The conf example claimed that without a token "the restart happens whenever the timer says so". It is the opposite — the check fails closed, so a running server is then never upgraded. Corrected, and REQUIRE_IDLE_CHECK documented next to it. tests/test_autoupgrade_recovers_a_down_server.py RUNS the script against a stubbed engine, runner, restart command and health file: the env restore is eval-based and the container check branches on an exit code, neither of which reading the source can confirm. 12 new tests, 56 across the three autoupgrade files. No version bump: nothing in the image changed, and bumping would send both production installs through an image upgrade and a restart for a host-side script. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A full `pytest tests/` run reached 46 GB RSS plus 20 GB of swap on a 54 GB machine. Swap filled, the OOM killer took coderai-nvidia — which was serving — and the box needed a hard reset, which lost the last forty minutes of kern.log with it. atop's history is what identified the process: a pytest worker, carrying the `coderai-front` process title because importing the front sets it, growing 7.8 GB → 34.8 GB → 46.6 GB across three ten-minute samples. The defect is `except asyncio.CancelledError: pass` wrapped around an await whose job is to collect a CHILD task. asyncio delivers a cancellation as a CancelledError at the next await point, so that handler catches two unrelated things: the child reporting it stopped (fine to swallow) and the caller asking us to stop (never fine). run_forever's reconnect handler sat on exactly such an await, so `task.cancel()` was absorbed and the loop carried on. With the reconnect delay mocked to zero by the test driving it, that became a busy loop allocating ~78 MB/s — AsyncMock records every call it receives — and 400,000 iterations in the five seconds I measured. being_cancelled() (Task.cancelling(), 3.11+) tells the two apart, and all four sites now re-raise ours: _stop_heartbeat_task, _cancel_inflight_tasks, the two keepalive collects in handle_message, and BrokerService.stop. The inflight ones matter in production independently of the leak — absorbing a cancel there means a shutdown waits out a whole video generation instead of stopping it, and run_forever simply reconnected forever instead of exiting. run_forever's own cancellation path clears the cancel for the duration of its cleanup and re-raises at the end, so honouring the cancel does not cost the websocket close. The test that drove it now brakes itself: the zero-delay sleep stand-in counts, and raises a BaseException (anything else is caught by the reconnect handler) well past the two reconnects it needs. With the product bug reintroduced the file fails in 1.5 s at 441 MB instead of eating the machine. tests/conftest.py adds what was missing in general: an RSS ceiling (6 GB, CODERAI_TEST_RSS_LIMIT_MB) and a per-test stall ceiling (180 s, CODERAI_TEST_STALL_SECONDS), both dumping every thread's stack to a file before exiting — pytest captures stderr per test and drops the buffer when the process does not unwind, so a watchdog that only printed would have died silently. No new dependency. "A test can consume the host" is a property of the run, not of one test. tests/test_broker_protocol.py could not finish before this — it hung at the 14th test. The whole suite now completes for the first time: 78 s, 1.57 GB peak, 1635 passed. The 25 failures are the 17 that already failed plus the 8 stale register-ack expectations in that file, which were unreachable behind the hang and are a separate question (the fixtures send `accepted: true`; the client has required `status: "ok"` for some time). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Township's clip lengths were tuned when the only video model could hold ~81 frames in one call, so a 70-second match became a dozen short cuts. LongCat was pretrained on continuation and the server renders a long shot in segments, so the same match wants a handful of long takes — and the frame numbers that suit one are wrong for the other by a factor of four. Only the two starter templates knew that, and only because someone typed it in. Worse, nothing snapped a picked count onto what the model can emit: random.randint handed the server lengths like 250, which is not 4n+1 for any VAE and not on LongCat's 93+80k segment grid, so the server rounded and the plan's own numbers became fiction. On LongCat that also silently costs block-sparse attention, because every point of 93+80k is a 16n+13 count BSA can tile and nothing between them is. Server — codai/models/video_geometry.py is the geometry once: native rate, the legal frame grid, how much ONE render holds, whether continuation is native, and the side multiple below which the fast attention path is lost. Published per model on /v1/models (ModelInfo.video) and used by the render path itself, so what the server advertises and what it rounds to cannot drift. The numbers were spread across three modules that could not tell a client; the literals here are pinned to each source by tests, since the LongCat and H3 workers run in 3.10 venvs this cannot import. models.json overrides every field. Township — the five frame fields now default to 0 = AUTO: derive the budget from the model's geometry and this tool's seconds band, then snap both ends and every pick onto the model's grid. Seconds, not frames, because clip count falls out of long_target / (frames/fps). An explicit number still wins, so a saved config keeps exactly the budget it had. Entrances and the face-off get their own shorter band instead of borrowing the fight budget (on LongCat that made every entrance an 11-22s take). Outcome shares are snapped per shot and the total rewritten, the per-render chunk comes from the model (Wan keeps its conservative 50), and the chaining decision reads the published continuation rather than matching the model's name. At the defaults a 70s match goes from 9-11 Wan cuts to 4-6 LongCat takes, each one request with native continuation, with BSA still on. 1607 tests pass; the 17 failures and the test_broker_protocol hang are pre-existing and unrelated (verified with the diff stashed). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An i2v generation died in the first denoising step with a bare AssertionError out of flash_attn_bsa_3d, on a geometry the guard had just declared valid: 93 frames, sides divisible by 64. bsa_problems only ever checked the noise branch. A conditioning pass does not tile the latent once — Attention.forward splits the sequence into a conditioning block and a noise block and calls flash_attn_bsa_3d on each (modules/attention.py:123-134), so the cond depth and the remaining noise depth must divide by the chunk as well. generate_i2v hardcodes one conditioning latent (pipeline:796) and 1 % 4 != 0, which means block-sparse attention has never once been able to run an i2v pass, at any frame count or resolution. The avatar pipeline hardcodes the same 1 in four places, and generate_vc computes a count it never rounds. Only generate_refine gets it right, by padding both counts to the granularity itself (pipeline:1245-1250). (a) The guard now knows about the split. _cond_latent_passes enumerates every conditioning count the request can produce — segment 0 uses the task's own entry point and every later segment uses generate_vc, so even t2v runs a conditioning pass once num_segments > 1, and BSA is one flag for the whole request — and a count that cannot tile puts the pass on dense attention with a line saying which condition failed. (b) bsa_pad_cond, OFF by default, rounds the count up to the granularity on the way into the DiT, which is what generate_refine does for itself, so an i2v or continuation pass can keep BSA. Patched onto the DiT instance rather than the vendored source, the same way the denoising bars are rebound: the count is born inside generate_i2v's loop, so there is no outer argument to correct. Off by default because the pipeline marks only the real conditioning frames clean (timestep[:, :1] = 0), so rounding 1 up to 4 hands the conditioning block three latent frames that are still noise — well-defined attention, but not what the model was trained on, and on a video model that kind of error shows up as temporal drift. Co-Authored-By:
-
- 09 Oct, 2026 16 commits
-
-
Stefy Lanza (nextime / spora ) authored
Probed every API surface coderai can use for attaching a global volume, each with a body valid enough that the extra-key check actually runs: POST /v1/pods key … not in input schema: 'globalNetworkVolumeId' podFindAndDeployOnDemand field not defined by the input type POST /v2/pods additional properties 'globalVolumeId' not allowed templates (mounts) every candidate key refused There is no public API for it: global volumes are console-only, as the docs say. Which makes `global_volume_id` a loaded gun — sending the field fails the create, so configuring a global volume would have stopped the install renting pods at all. Both create paths now recognise that specific rejection, drop the field and retry. The pod boots without the volume and downloads its weights: slower and it costs bandwidth, and far better than serving nothing. Logged once per process, naming the consequence and the override to set when RunPod publishes the field name. An unrelated failure is NOT treated this way — a capacity error retried with the volume stripped would bring a pod up silently wrong. Earlier probes that reported the field as "accepted" were wrong: with an otherwise-invalid body the validators stop before the extra-key check. Worth remembering next time I probe a schema. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`gpu_type` was a filter: "NVIDIA A40" meant an A40 or nothing, so when RunPod had no A40 free the request failed rather than taking a card that would have served it. Now `gpu_types` is an ordered PREFERENCE and `allow_other_gpus` lets the search fall through to anything else that still satisfies min_vram_gb and max_hourly_usd. A preferred card ranks ahead of every fallback whatever selection_criteria says — otherwise "cheaper" would put the fallback first and naming a card would mean nothing. Each ranked option carries `preferred`, and provisioning a card nobody asked for prints "[FALLBACK — no preferred card available]": correct behaviour, but never silent. Off by default. A pinned card is sometimes pinned for a reason, and an install that quietly started renting something else would be a surprise on the invoice. Name matching is exact on the id or display name, case-insensitive, tolerating a missing "NVIDIA " prefix, and deliberately NOT a substring: "A40" would otherwise match "NVIDIA RTX A4000" — 16 GB for a job that asked for 48. GPU names split on commas only, because "NVIDIA RTX A6000" has spaces in it. Also: the admin handler rebuilds the runpod block from a whitelist, so all six keys added today (global_volume_id, data_centers, gpu_types, allow_other_gpus, allow_client_scale, client_scale_ttl_s) were being dropped the next time anyone saved a model from the GUI — taking the warm schedule and the volume with them. They are in the whitelist now, with the two ordered lists kept in order. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Weights in one place, pods wherever a card is free. `global_volume_id` attaches a RunPod GLOBAL volume (beta since Sept 2026): region-independent storage, so unlike `network_volume_id` it does NOT pin the pod's data centre — which is the only reason to use one. Resolved by its own function precisely so the two cannot be confused: the region lookup stays under the network-volume branch, and a test holds them apart. `data_centers` is the other half. A single `data_center` pins one region and a blank one lets RunPod pick any region on earth, so there was no way to say "anywhere in the EU" — which is what data that may not leave the EU requires. The list is tried in order, and each (card, region) pair becomes its own candidate so the capacity fallback that already walks GPU options walks regions too, with no new control flow. The pod handle records the region it actually got, not the pool's first choice. Two honest limits, both documented in the code and the guide. A global volume must be created in the RunPod CONSOLE: POST /v2/network-volumes allows only STANDARD|HIGH_PERFORMANCE and demands a dataCenter. And attaching is console-documented only — no field exists in the Pod API reference, and the v1 REST validator accepts unknown body keys silently, so a wrong name cannot be caught by probing; it yields a pod with no volume that re-downloads its weights. We send globalNetworkVolumeId, overridable by CODERAI_RUNPOD_GLOBAL_VOLUME_FIELD, and it must be verified on a real pod. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The root cause under the last three fixes. The pod registry recorded os.getpid() and both readers skipped an entry whose pid matched ours as "our own, the local pool handles it". Inside a container the engine comes up as the SAME pid on every boot — 74, on the production host — so after a restart the previous container's pod read as ours and was skipped by both find_shared_pod (nothing to adopt) and sibling_is_provisioning (nothing booting). The warm path then rented another A40 beside the one already running. Three restarts this afternoon, three extra A40s, each billing until it was terminated by hand. Entries now carry a per-process nonce and identity is that. The pid is still recorded, and an entry written before the nonce falls back to comparing it, which is the old behaviour and no worse. test_a_surviving_pods_token_is_remembered_across_a_restart simulated the new process by monkeypatching os.getpid — the very thing that does not change. It patches the nonce now, which is why it failed on this commit and passes on it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The last hole in the warm path. sibling_is_provisioning already recognises a pod registered at creation with no URL yet — and after a restart the recorded pid no longer matches, so our own booting pod qualifies — but ensure_ready never asked. Seen live: a pod created at 18:57 was six minutes into its image pull when the 19:03 upgrade restarted the service, and the warm path rented a second A40 beside it. The upgrade runs hourly and an image pull is ~6 minutes, so that is not a rare alignment. Order in ensure_ready is now: a healthy pod of our own, else adopt a ready one, else wait for one that is booting, else rent. Also prune the pod registry in reap_orphans: an id RunPod no longer has stayed "known", which both shielded it from reaping and let the file grow for the life of the install — including pods terminated out of band, of which this afternoon produced five. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A warm floor is re-established at every boot, and ensure_ready() went straight to _provision_one(). The adoption logic — "a pod for this pool is already running, take it over" — existed only inside acquire(), so the warm path never saw it. Three restarts on the production orchestrator this afternoon rented three A40s and left each previous one billing until the reaper's 20-minute grace expired. An hourly unattended upgrade would have done that every hour. find_shared_pod already covers the case: after a restart the recorded pid no longer matches, so our own previous pod is adoptable. It is now one method, _adopt_shared_pod, called from acquire(), ensure_ready() and maintain()'s warm top-up. It also stops lying about the cost. An adopted pod was recorded at $0/hr as "billed by its owner", which is true of a sibling engine's pod and false of one we are inheriting from our own restart — a real A40 hidden from the $/hr cap and from the spend page. register_pod now stores the rate and adoption restores it; an older registry entry without one still adopts, at 0. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
NameError in _provision_one: it read `dc`, a local computed inside _create_with_fallback, from a function that has no such name. The sequence that costs money is this — create_pod succeeds, the pod boots on RunPod, the PodHandle raises, the pool never records the pod, the caller provisions again. Four A40s existed for a warm floor of one before anyone looked. Both paths now call pool._data_center(), which keeps the same precedence (the volume's region, then the model's, then the account's) in one place. The two tests that covered this asserted the broken expression as a SOURCE STRING — 'data_center=(getattr(self, "_volume_dc", "") or dc or "")' — so they passed on a NameError and would have gone on passing. They now call the resolver, including with nothing configured anywhere, which is the shape that raised. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found while configuring the Digesta catalogue: the launcher reads /root/.coderai/config.json (engine_specs, published port) while the front and the engine read /root/.coderai/coderai/config.json. The RunPod api_key and the spend caps were in the first one, so the engine refused to provision anything and the admin page reported no caps at all. Records what was moved where, and that CODERAI_CONFIG_DIR=/config remains the fix that makes it unnecessary. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Watched it happen on the production orchestrator: configuring qwen38-awq warm during office hours provisioned TWO A40s a second apart. A pod does not enter self.pods until it answers its health probe, which is minutes away. ensure_ready() never set the _provisioning flag, and the warm top-up in maintain() never read it — so the 15 s scaler saw "0 healthy, floor 1" while a pod was already booting and rented another. It stopped at two only because _provision_one blocks the scaler thread for the whole boot; a non-blocking provision would have added one every 15 seconds. The demand path (_should_grow) has always checked the flag. Now all three agree: one on the way is one pod, not none, including one a sibling engine sharing the pool is starting. Also: /v1/runpod/scale answered with the internal pool key ("pool:digesta-qwen38") rather than the model the client asked about. Several models share a pool, and that string is not something the caller sent or can look up — so the answer names the model and reports the pool beside it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Configuring the Digesta catalogue on the production orchestrator turned up three faults in warm_configured_pools, all of which spend money: * `enabled: false` was not consulted. A catalogue entry parked for later with keep_warm still set rented a GPU at every boot. * the entry was named `alias or path`. A RunPod-only entry has neither — no weights live on that machine, and an alias only exists when a second name is wanted — so it warmed under the key "" and the pool was never the one that went on to serve requests. `id` is now the fallback, and an entry with no name at all says so instead of warming nothing quietly. * the schedule was ignored. ensure_ready() is the "a request is waiting" path and provisions unconditionally; at boot nothing is waiting, so a restart at 02:00 on a Sunday rented exactly the pod the window exists to avoid. It now checks effective_min_pods first, which also means a client lease that outlives a restart is honoured rather than dropped. Found on the host, not by reading: the warm pod the configuration asked for never appeared. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The autoscaler reacts to load — a pod is added once the least-loaded one already has scale_up_inflight_per_pod requests in flight. That is right for traffic nobody predicted and wrong for a client that KNOWS what is coming: a batch of a thousand documents otherwise discovers the pool one cold start at a time. So a client may now say so: POST /v1/runpod/scale {model, pods}. It is a lease, bounded three ways, because a client that can raise the floor can raise the bill: * allow_client_scale, off by default, so no model is client-scalable unless someone said so; * clamped to max_pods — asking for more returns the ceiling with "clamped": true rather than an error, since the client asked for "as many as you can" and refusing would leave it with none; * client_scale_ttl_s (default 30 min), because the failure that matters is not a wrong number, it is a client that asks for four A40s and then dies. A lease that never expired would be a bill that never stopped, so a 0 in the config falls back to the default instead of meaning "forever". The lease and a warm-pod schedule are MAXED, never summed: both mean "hold this many ready", so a client cannot lower a configured floor and a closed window cannot cancel a lease. /v1/runpod/status reports the live lease per model, so an unexpected bill has a visible cause. Not admin-scoped, unlike /v1/runpod/spend: the gate is the per-model flag, not the key, and the account's $/hr and spend caps still refuse to provision. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Proven on zeiss with a real LoRA training run: the skip fired correctly but read "SKIP: serving 1 request(s) [engines=0 jobs=1]". There was no request -- that is the entire case this check exists for, since /v1/loras/train with wait=false returns at once and the work continues for hours. The log now says "work in progress", which is what it means. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The unattended upgrade measured idleness as requests in flight, and a LoRA training run has none: the request that starts it returns immediately and the work continues for hours, so coderai_engine_inflight is 0 throughout. An hourly upgrade would have restarted straight through one -- the standing "never restart during training" rule, reintroduced by automation. Nothing was missing in the training path. loras.py already registers a `training` task, the engine already ships active tasks in /internal/engine-state, and the front already stores them on EngineEntry.tasks. The gap was that /metrics never exposed them and the probe never looked, so this adds coderai_engine_jobs_active per engine and coderai_jobs_active across them, counting running/queued/paused -- paused on purpose, because a thermally paused generation has not finished and restarting loses it. The probe now reads both numbers from one scrape and takes the MAXIMUM, never the sum: a generation is both an in-flight request and a running task, so adding them would double-count and the numbers in the log would match nothing an operator can see. A front that predates the gauge reports "-" and the detail line says "front too old", rather than claiming zero and silently restoring the blind spot. tests/test_autoupgrade_sees_training.py renders the exposition from a fake engine carrying a training task, runs the probe's awk program as the script actually has it, and pins both halves of the premise -- the registration in loras.py and the engine's reporting of active states -- so the gauge cannot go quiet without a test failing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
run_oci.sh takes its engine from CONTAINER_ENGINE and defaults to docker, so on the rootless-podman host this script was written for it would have looked for a docker socket that is not there -- failing on every hourly tick with nothing to show but a non-zero exit. Found while installing it on the Digesta production server, before it ran. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The Digesta host added both 10.89.0.178 and ::ffff:10.89.0.178 to trusted_proxies and host->8777 still refused. The ::ffff: hunch was right: the internal nginx listens on 8776 AND [::]:8776, so a connection landing on the v6 socket makes $remote_addr -- and therefore X-Forwarded-For, and therefore the peer -- the IPv4-mapped form, which a string membership test rejects even when the plain address is listed. Nothing in the configuration explains that refusal. Peers are now compared as addresses: a mapped v6 peer matches a plain v4 entry and the reverse, and CIDR entries work, so a container network can be trusted as 10.89.0.0/24 instead of pinning one address in the quadlet. A non-address entry (a socket path) is still compared literally, and a v6 peer never matches a v4 network. Their broadened list could not have taken effect anyway -- it went into the config.json the app is not reading, which is why `enabled` there did nothing and the env var was needed. So that test did not rule the peer check out; the list in force was still the default pair. The rest of the report is answered in the doc rather than in code, because there is no bug behind it: `Server: nginx` appears on PROXIED responses too unless proxy_pass_header Server is set, and the missing `GET /admin` access-log line is _PollNoiseFilter, which drops reads of /admin, /static, /login and /logout from the front's access log while leaving /healthz alone -- exactly the asymmetry that looked like nginx short-circuiting by source. A 302 to /coderai/login with no Set-Cookie is what the front returns when forward-auth declines. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two production installs now follow this repo: zeiss (nexlab) and the Digesta host in Aruba. The Aruba one should track `production` without anyone logging in, so this adds coderai-autoupgrade plus a systemd user service and an hourly timer. It skips the upgrade outright while CoderAI is serving -- checked BEFORE the image is touched, not merely before the restart, so a busy host never reaches the state where new code waits in the image. Two probes and the higher count wins: the front's coderai_engine_inflight (every request in flight, pod-served ones included) and the pods' own summary.inflight from /v1/runpod/status. The RunPod one carries the weight on an orchestrator where every model is remote: the local engines are idle by definition, so the pods' count is the only sign work is happening, and killing a pod-served request loses the work AND keeps paying for the pod that was doing it. The check fails CLOSED -- no token, wrong URL or an unreachable endpoint skips the run, because an unreadable probe is nearly always a misconfiguration rather than an idle server. It decides by reading the image, not by an exit code: run_oci.sh exits 0 both when it upgraded and when the image was already current (the rc==10 branch prints its message and falls through to the same exit), so an exit-code-driven loop would have restarted the service every hour forever. The script compares __version__ baked in the image before and after, and a change is the only thing that triggers a restart. By default MAX_DEFERRALS=0, so a request in flight is never interrupted: the new code waits and every run says so. And it rolls back -- the pre-upgrade image is tagged first, and a service that is not healthy within HEALTH_TIMEOUT gets the old tag back, another restart and a non-zero exit that says whether it recovered. Writing the example config surfaced a trap worth the test that now guards it: the file is sourced, so `RESTART_CMD=systemctl --user restart coderai.service` unquoted assigns "systemctl" and RUNS the rest -- restarting nothing, failing the health check and rolling back a good upgrade. Everything in the example is quoted and each run echoes the restart command it resolved. Verified against the live server on zeiss: already-current does nothing; an overlapping run is a no-op; a missing token skips; three in-flight engine requests skip; and engines idle with a pod serving two still skips. Co-Authored-By:
-
- 08 Oct, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
The Digesta host reported SSO minting a session from inside the container (:8776) but refusing from the host through the published port (:8777), with the container's address in trusted_proxies and the front's own access log showing that very address as the client. Both that and "enabled: true in config.json did not switch the feature on" are one cause. The peer _fa_trusted checks is not the socket peer: uvicorn installs ProxyHeadersMiddleware by default with trusted_hosts="127.0.0.1", so a hop arriving over loopback from the internal nginx has scope["client"] rewritten from X-Forwarded-For before any of our middleware runs. On a container network that is the address the proxy was SNATed to. So the log and the check agreed and the peer really was 10.89.0.178 -- but the list it was compared against was the DEFAULT ["127.0.0.1", "::1"], because their config.json sat at a path the app does not read (the $CODERAI_CONFIG_DIR/coderai trap), which is equally why the file's `enabled` did nothing and the env var was needed. Loopback is in that default; the container address is not. The loader is fine and `enabled` from the file does suffice -- verified, not assumed. What was missing was a way to supply the peers the same way the switch and the secret can already be supplied, so CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_ PROXIES now overrides the list; blank or unset falls through to config.json. Enabling the feature from a unit file and then having it reject every request from your own proxy was a gap in the contract. And a refusal now says why, once per peer and reason per 60s, naming the list it was checked against and never the secret. A refusal is a 302 to the login form, indistinguishable from "no session yet"; the report had to ask for exactly this instrumentation, so it is permanent now. tests/test_forward_auth_peer_trust.py reproduces the peer resolution for all three hops, so the table in the doc is executable rather than reasoned. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Block-sparse attention tiles the latent into (4, 4, 4) chunks and asserts every axis divides evenly, so a side must be a multiple of 64 -- the VAE's 16px cell times 4. At height 480 the preset turned the flag on and the pass silently fell back to DENSE attention, which at ~37k tokens is both much slower and what ran a 24GB card out of memory on the first continuation segment of a 304-frame clip. 832 already divides by 64; 512 is the nearest height that does. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 304-frame request sat at "generating 0/16 (0%)" for its whole nine-minute first segment while the service's log showed Denoising: 15/16. Two defects. The service set _progress["step"] = 0 at the start of every segment and never again, so the step count existed only inside the vendored pipeline's tqdm(total=len(timesteps), desc="Denoising") -- which takes no callback. Nothing reached GET /progress; the poller in video.py faithfully reported the 0 it was handed. Each pipeline module did `from tqdm import tqdm`, so the bar is instrumented by rebinding the name on the MODULE (patching tqdm.tqdm would be too late), and only bars labelled "Denoising" are tracked so a checkpoint-shard bar cannot move a request's progress. The totals also counted ONE segment: a four-segment render would have run 0->16 four times against a denominator of 16. `step` now carries the completed segments' offset so it counts the stage end to end, and the bar spans steps x segments -- with refinement excluded, being a single pass over the finished video. A request that asked for a duration rather than a segment count only learns the real count from the service, so the denominator is corrected on the first report, through a setter that does not restart the bar and lose the elapsed time the rate is derived from. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Digesta needs to see what the fleet is doing, not only what it costs: how many pods are running, where, and which model. The spend endpoint already carried the pod rows, but a pod did not report its placement and there was no roll-up to read at a glance. A pod now carries and reports its data_center, cloud_type and gpu_count, plus the pool and engine it belongs to. GET /v1/runpod/status (an alias of /v1/runpod/spend, one implementation so the two can never disagree) adds a `summary`: pods_total/ready/booting/unhealthy, gpus_in_use, inflight, spot_pods, hourly_usd, and tallies by model, region, GPU and cloud type. Same gate as the spend figures — an admin-scoped API key, 403 otherwise. Three deliberate distinctions, documented: summary.hourly_usd counts only pods actually serving (a booting pod bills but answers nothing, so counting it would overstate capacity); pods_unhealthy is the alert-worthy number, a pod past boot that is costing money and serving nothing; and gpus_in_use counts cards, so a gpu_count:2 pod counts twice. An unknown region or GPU is labelled "unknown" rather than dropped, so the tallies always sum to pods_total. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Implements docs/admin-forward-auth.md. Off by default, switchable from Settings -> Single sign-on, from config.json, or from the container's env (CODERAI_ADMIN_FORWARD_AUTH=1|0, CODERAI_ADMIN_FORWARD_AUTH_SECRET) — the env switch can disable it as well as enable it, so a unit file can override a mounted config either way. A request is forward-authed only when BOTH checks pass: the immediate peer is in trusted_proxies AND the shared secret header matches (constant time). Either alone is forgeable, so the identity header is not even read until trust is established — a client sending X-Forwarded-User directly is ignored. Enabled with no secret stays OFF with a log line, because without one there is nothing separating the proxy from any other client. It mints a real signed session and injects it into the current request, not just a rendered page: the dashboard's own /admin/api/* calls authenticate by cookie, so a page with 401ing AJAX would not be a working GUI. An auto-created account gets a random password nobody is told and is never asked to change it. keep_local_login=false makes /login refuse. Logout honours post_logout_url, since returning to our own login would be met by the proxy's headers and log the user straight back in. GUI only: /v1/* keeps its bearer tokens whatever the proxy asserts, and the readiness probes stay credential-free. The settings API reports shared_secret_set as a boolean and never the secret. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reproduced on the published 0.2.83 image, GPU-less, no Authorization header: GET /v1/models -> 200 <- open POST /v1/chat/completions -> 503 (no model in the test, so inconclusive) GET /v1/runpod/spend -> 502 Nothing returned 401, because the front was never an enforcement point: it answers some /v1 paths from its own registry and proxies the rest, so whether a request needed a token depended on which component served it. On a host published through a reverse proxy that is an open relay for the model list, and potentially for generation. The front now gates /v1 itself with the same credentials the engine accepts: the env token a capability image is locked with, the shared cluster token a head presents, an API token from auth.json, a logged-in browser session, the in-process broker bridge, and the aisbf relay marker. Constant-time comparison throughout. One deliberate difference from the engine's middleware: it does NOT fall open when no credentials are configured. That fallback is defensible on a loopback workstation and is precisely what exposed this one, so the front refuses instead. Readiness and progress endpoints stay exempt — a probe that 401s is read as "backend down", and surya-ocr's vLLM client probes {service_url}/health. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reproduced the Digesta "engine#0 crash-loops despite backend.type=cpu and engine_specs" report on the published 0.2.83 image, GPU-less, with their config — and it is not the front failing to propagate the engine spec. coderai-entrypoint creates $CODERAI_CONFIG_DIR/coderai and symlinks ~/.coderai to it (lines 49, 70); the app resolves config from the HOME-style path. A config.json written at $CODERAI_CONFIG_DIR is therefore never read. On first start the app writes a DEFAULT coderai/config.json — backend null, engine_specs null, engines null — reads that, and the engine auto-detects, finds no GPU and aborts. Which is exactly the "backend: null" they saw and read as the front regenerating the config. Documents the live path, the one-liner to check which file is in use, and that mounting straight at /config/coderai sidesteps it. Also ships coderai-upgrade in Dockerfile.oci-venv. It was only in Dockerfile.oci, so every image we actually publish (venv-built) failed `coderai-docker --upgrade` with "stat /usr/local/bin/coderai-upgrade: no such file or directory". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide said the venv ships in an image. It does not, in either image, and it does not need to: the engine builds it itself from requirements-surya.txt onto the persistent cache mount — but only when ocr.surya_auto_build is true, and it defaults to FALSE (config.py:908). That default is why the Digesta bring-up built the venv by hand and called it a packaging gap. It is a one-line config setting on the orchestrator. Documents the pillow<11 vs >=12 conflict that forces the venv to be isolated, where it lands, and how to pre-build it instead. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide told the operator to install ghcr.io/nextime/coderai-ocr as the orchestrator. That is backwards and it is the mistake the Digesta bring-up actually made: a capability image is a POD, what CoderAI rents and launches, and it has no GUI; its admin pages are not behind the API's bearer check, so it is not something to expose either. Rewritten around the full image on port 8776, with the two settings a GPU-less host needs and the reason each one is needed: - backend.type must be "cpu", not "auto": with auto, main.py searches nvidia/vulkan/opencl and exits "No supported backend detected" when it finds none. Set explicitly, that search is skipped. - engine_specs must name the engine: left to auto-detect, the front creates a no-GPU engine NAMED cpu but launches it with backend="auto" (engine_supervisor.py:311), which hits the same abort whatever backend.type says. That is why the log reads "[cpu] Available backends: {'cpu': True}" and then "No supported backend detected" — [cpu] is the name, auto was the backend. Also states that server.engines=0 means AUTO (one per GPU, min 1), that no "zero engines" setting exists, and that you do not want one: the RunPod pools, scaler and reaper live inside the primary engine, so one idle CPU engine is the component that manages the pods, not a workaround. Service examples, the upgrade section and the reference table all follow. Co-Authored-By:
-