- 09 Oct, 2026 15 commits
-
-
Stefy Lanza (nextime / spora ) authored`gpu_type` was a filter: "NVIDIA A40" meant an A40 or nothing, so when RunPod had no A40 free the request failed rather than taking a card that would have served it. Now `gpu_types` is an ordered PREFERENCE and `allow_other_gpus` lets the search fall through to anything else that still satisfies min_vram_gb and max_hourly_usd. A preferred card ranks ahead of every fallback whatever selection_criteria says — otherwise "cheaper" would put the fallback first and naming a card would mean nothing. Each ranked option carries `preferred`, and provisioning a card nobody asked for prints "[FALLBACK — no preferred card available]": correct behaviour, but never silent. Off by default. A pinned card is sometimes pinned for a reason, and an install that quietly started renting something else would be a surprise on the invoice. Name matching is exact on the id or display name, case-insensitive, tolerating a missing "NVIDIA " prefix, and deliberately NOT a substring: "A40" would otherwise match "NVIDIA RTX A4000" — 16 GB for a job that asked for 48. GPU names split on commas only, because "NVIDIA RTX A6000" has spaces in it. Also: the admin handler rebuilds the runpod block from a whitelist, so all six keys added today (global_volume_id, data_centers, gpu_types, allow_other_gpus, allow_client_scale, client_scale_ttl_s) were being dropped the next time anyone saved a model from the GUI — taking the warm schedule and the volume with them. They are in the whitelist now, with the two ordered lists kept in order. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com>
-
Stefy Lanza (nextime / spora ) authored
Weights in one place, pods wherever a card is free. `global_volume_id` attaches a RunPod GLOBAL volume (beta since Sept 2026): region-independent storage, so unlike `network_volume_id` it does NOT pin the pod's data centre — which is the only reason to use one. Resolved by its own function precisely so the two cannot be confused: the region lookup stays under the network-volume branch, and a test holds them apart. `data_centers` is the other half. A single `data_center` pins one region and a blank one lets RunPod pick any region on earth, so there was no way to say "anywhere in the EU" — which is what data that may not leave the EU requires. The list is tried in order, and each (card, region) pair becomes its own candidate so the capacity fallback that already walks GPU options walks regions too, with no new control flow. The pod handle records the region it actually got, not the pool's first choice. Two honest limits, both documented in the code and the guide. A global volume must be created in the RunPod CONSOLE: POST /v2/network-volumes allows only STANDARD|HIGH_PERFORMANCE and demands a dataCenter. And attaching is console-documented only — no field exists in the Pod API reference, and the v1 REST validator accepts unknown body keys silently, so a wrong name cannot be caught by probing; it yields a pod with no volume that re-downloads its weights. We send globalNetworkVolumeId, overridable by CODERAI_RUNPOD_GLOBAL_VOLUME_FIELD, and it must be verified on a real pod. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The root cause under the last three fixes. The pod registry recorded os.getpid() and both readers skipped an entry whose pid matched ours as "our own, the local pool handles it". Inside a container the engine comes up as the SAME pid on every boot — 74, on the production host — so after a restart the previous container's pod read as ours and was skipped by both find_shared_pod (nothing to adopt) and sibling_is_provisioning (nothing booting). The warm path then rented another A40 beside the one already running. Three restarts this afternoon, three extra A40s, each billing until it was terminated by hand. Entries now carry a per-process nonce and identity is that. The pid is still recorded, and an entry written before the nonce falls back to comparing it, which is the old behaviour and no worse. test_a_surviving_pods_token_is_remembered_across_a_restart simulated the new process by monkeypatching os.getpid — the very thing that does not change. It patches the nonce now, which is why it failed on this commit and passes on it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The last hole in the warm path. sibling_is_provisioning already recognises a pod registered at creation with no URL yet — and after a restart the recorded pid no longer matches, so our own booting pod qualifies — but ensure_ready never asked. Seen live: a pod created at 18:57 was six minutes into its image pull when the 19:03 upgrade restarted the service, and the warm path rented a second A40 beside it. The upgrade runs hourly and an image pull is ~6 minutes, so that is not a rare alignment. Order in ensure_ready is now: a healthy pod of our own, else adopt a ready one, else wait for one that is booting, else rent. Also prune the pod registry in reap_orphans: an id RunPod no longer has stayed "known", which both shielded it from reaping and let the file grow for the life of the install — including pods terminated out of band, of which this afternoon produced five. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A warm floor is re-established at every boot, and ensure_ready() went straight to _provision_one(). The adoption logic — "a pod for this pool is already running, take it over" — existed only inside acquire(), so the warm path never saw it. Three restarts on the production orchestrator this afternoon rented three A40s and left each previous one billing until the reaper's 20-minute grace expired. An hourly unattended upgrade would have done that every hour. find_shared_pod already covers the case: after a restart the recorded pid no longer matches, so our own previous pod is adoptable. It is now one method, _adopt_shared_pod, called from acquire(), ensure_ready() and maintain()'s warm top-up. It also stops lying about the cost. An adopted pod was recorded at $0/hr as "billed by its owner", which is true of a sibling engine's pod and false of one we are inheriting from our own restart — a real A40 hidden from the $/hr cap and from the spend page. register_pod now stores the rate and adoption restores it; an older registry entry without one still adopts, at 0. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
NameError in _provision_one: it read `dc`, a local computed inside _create_with_fallback, from a function that has no such name. The sequence that costs money is this — create_pod succeeds, the pod boots on RunPod, the PodHandle raises, the pool never records the pod, the caller provisions again. Four A40s existed for a warm floor of one before anyone looked. Both paths now call pool._data_center(), which keeps the same precedence (the volume's region, then the model's, then the account's) in one place. The two tests that covered this asserted the broken expression as a SOURCE STRING — 'data_center=(getattr(self, "_volume_dc", "") or dc or "")' — so they passed on a NameError and would have gone on passing. They now call the resolver, including with nothing configured anywhere, which is the shape that raised. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found while configuring the Digesta catalogue: the launcher reads /root/.coderai/config.json (engine_specs, published port) while the front and the engine read /root/.coderai/coderai/config.json. The RunPod api_key and the spend caps were in the first one, so the engine refused to provision anything and the admin page reported no caps at all. Records what was moved where, and that CODERAI_CONFIG_DIR=/config remains the fix that makes it unnecessary. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Watched it happen on the production orchestrator: configuring qwen38-awq warm during office hours provisioned TWO A40s a second apart. A pod does not enter self.pods until it answers its health probe, which is minutes away. ensure_ready() never set the _provisioning flag, and the warm top-up in maintain() never read it — so the 15 s scaler saw "0 healthy, floor 1" while a pod was already booting and rented another. It stopped at two only because _provision_one blocks the scaler thread for the whole boot; a non-blocking provision would have added one every 15 seconds. The demand path (_should_grow) has always checked the flag. Now all three agree: one on the way is one pod, not none, including one a sibling engine sharing the pool is starting. Also: /v1/runpod/scale answered with the internal pool key ("pool:digesta-qwen38") rather than the model the client asked about. Several models share a pool, and that string is not something the caller sent or can look up — so the answer names the model and reports the pool beside it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Configuring the Digesta catalogue on the production orchestrator turned up three faults in warm_configured_pools, all of which spend money: * `enabled: false` was not consulted. A catalogue entry parked for later with keep_warm still set rented a GPU at every boot. * the entry was named `alias or path`. A RunPod-only entry has neither — no weights live on that machine, and an alias only exists when a second name is wanted — so it warmed under the key "" and the pool was never the one that went on to serve requests. `id` is now the fallback, and an entry with no name at all says so instead of warming nothing quietly. * the schedule was ignored. ensure_ready() is the "a request is waiting" path and provisions unconditionally; at boot nothing is waiting, so a restart at 02:00 on a Sunday rented exactly the pod the window exists to avoid. It now checks effective_min_pods first, which also means a client lease that outlives a restart is honoured rather than dropped. Found on the host, not by reading: the warm pod the configuration asked for never appeared. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The autoscaler reacts to load — a pod is added once the least-loaded one already has scale_up_inflight_per_pod requests in flight. That is right for traffic nobody predicted and wrong for a client that KNOWS what is coming: a batch of a thousand documents otherwise discovers the pool one cold start at a time. So a client may now say so: POST /v1/runpod/scale {model, pods}. It is a lease, bounded three ways, because a client that can raise the floor can raise the bill: * allow_client_scale, off by default, so no model is client-scalable unless someone said so; * clamped to max_pods — asking for more returns the ceiling with "clamped": true rather than an error, since the client asked for "as many as you can" and refusing would leave it with none; * client_scale_ttl_s (default 30 min), because the failure that matters is not a wrong number, it is a client that asks for four A40s and then dies. A lease that never expired would be a bill that never stopped, so a 0 in the config falls back to the default instead of meaning "forever". The lease and a warm-pod schedule are MAXED, never summed: both mean "hold this many ready", so a client cannot lower a configured floor and a closed window cannot cancel a lease. /v1/runpod/status reports the live lease per model, so an unexpected bill has a visible cause. Not admin-scoped, unlike /v1/runpod/spend: the gate is the per-model flag, not the key, and the account's $/hr and spend caps still refuse to provision. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Proven on zeiss with a real LoRA training run: the skip fired correctly but read "SKIP: serving 1 request(s) [engines=0 jobs=1]". There was no request -- that is the entire case this check exists for, since /v1/loras/train with wait=false returns at once and the work continues for hours. The log now says "work in progress", which is what it means. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The unattended upgrade measured idleness as requests in flight, and a LoRA training run has none: the request that starts it returns immediately and the work continues for hours, so coderai_engine_inflight is 0 throughout. An hourly upgrade would have restarted straight through one -- the standing "never restart during training" rule, reintroduced by automation. Nothing was missing in the training path. loras.py already registers a `training` task, the engine already ships active tasks in /internal/engine-state, and the front already stores them on EngineEntry.tasks. The gap was that /metrics never exposed them and the probe never looked, so this adds coderai_engine_jobs_active per engine and coderai_jobs_active across them, counting running/queued/paused -- paused on purpose, because a thermally paused generation has not finished and restarting loses it. The probe now reads both numbers from one scrape and takes the MAXIMUM, never the sum: a generation is both an in-flight request and a running task, so adding them would double-count and the numbers in the log would match nothing an operator can see. A front that predates the gauge reports "-" and the detail line says "front too old", rather than claiming zero and silently restoring the blind spot. tests/test_autoupgrade_sees_training.py renders the exposition from a fake engine carrying a training task, runs the probe's awk program as the script actually has it, and pins both halves of the premise -- the registration in loras.py and the engine's reporting of active states -- so the gauge cannot go quiet without a test failing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
run_oci.sh takes its engine from CONTAINER_ENGINE and defaults to docker, so on the rootless-podman host this script was written for it would have looked for a docker socket that is not there -- failing on every hourly tick with nothing to show but a non-zero exit. Found while installing it on the Digesta production server, before it ran. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The Digesta host added both 10.89.0.178 and ::ffff:10.89.0.178 to trusted_proxies and host->8777 still refused. The ::ffff: hunch was right: the internal nginx listens on 8776 AND [::]:8776, so a connection landing on the v6 socket makes $remote_addr -- and therefore X-Forwarded-For, and therefore the peer -- the IPv4-mapped form, which a string membership test rejects even when the plain address is listed. Nothing in the configuration explains that refusal. Peers are now compared as addresses: a mapped v6 peer matches a plain v4 entry and the reverse, and CIDR entries work, so a container network can be trusted as 10.89.0.0/24 instead of pinning one address in the quadlet. A non-address entry (a socket path) is still compared literally, and a v6 peer never matches a v4 network. Their broadened list could not have taken effect anyway -- it went into the config.json the app is not reading, which is why `enabled` there did nothing and the env var was needed. So that test did not rule the peer check out; the list in force was still the default pair. The rest of the report is answered in the doc rather than in code, because there is no bug behind it: `Server: nginx` appears on PROXIED responses too unless proxy_pass_header Server is set, and the missing `GET /admin` access-log line is _PollNoiseFilter, which drops reads of /admin, /static, /login and /logout from the front's access log while leaving /healthz alone -- exactly the asymmetry that looked like nginx short-circuiting by source. A 302 to /coderai/login with no Set-Cookie is what the front returns when forward-auth declines. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two production installs now follow this repo: zeiss (nexlab) and the Digesta host in Aruba. The Aruba one should track `production` without anyone logging in, so this adds coderai-autoupgrade plus a systemd user service and an hourly timer. It skips the upgrade outright while CoderAI is serving -- checked BEFORE the image is touched, not merely before the restart, so a busy host never reaches the state where new code waits in the image. Two probes and the higher count wins: the front's coderai_engine_inflight (every request in flight, pod-served ones included) and the pods' own summary.inflight from /v1/runpod/status. The RunPod one carries the weight on an orchestrator where every model is remote: the local engines are idle by definition, so the pods' count is the only sign work is happening, and killing a pod-served request loses the work AND keeps paying for the pod that was doing it. The check fails CLOSED -- no token, wrong URL or an unreachable endpoint skips the run, because an unreadable probe is nearly always a misconfiguration rather than an idle server. It decides by reading the image, not by an exit code: run_oci.sh exits 0 both when it upgraded and when the image was already current (the rc==10 branch prints its message and falls through to the same exit), so an exit-code-driven loop would have restarted the service every hour forever. The script compares __version__ baked in the image before and after, and a change is the only thing that triggers a restart. By default MAX_DEFERRALS=0, so a request in flight is never interrupted: the new code waits and every run says so. And it rolls back -- the pre-upgrade image is tagged first, and a service that is not healthy within HEALTH_TIMEOUT gets the old tag back, another restart and a non-zero exit that says whether it recovered. Writing the example config surfaced a trap worth the test that now guards it: the file is sourced, so `RESTART_CMD=systemctl --user restart coderai.service` unquoted assigns "systemctl" and RUNS the rest -- restarting nothing, failing the health check and rolling back a good upgrade. Everything in the example is quoted and each run echoes the restart command it resolved. Verified against the live server on zeiss: already-current does nothing; an overlapping run is a no-op; a missing token skips; three in-flight engine requests skip; and engines idle with a pod serving two still skips. Co-Authored-By:
-
- 08 Oct, 2026 18 commits
-
-
Stefy Lanza (nextime / spora ) authored
The Digesta host reported SSO minting a session from inside the container (:8776) but refusing from the host through the published port (:8777), with the container's address in trusted_proxies and the front's own access log showing that very address as the client. Both that and "enabled: true in config.json did not switch the feature on" are one cause. The peer _fa_trusted checks is not the socket peer: uvicorn installs ProxyHeadersMiddleware by default with trusted_hosts="127.0.0.1", so a hop arriving over loopback from the internal nginx has scope["client"] rewritten from X-Forwarded-For before any of our middleware runs. On a container network that is the address the proxy was SNATed to. So the log and the check agreed and the peer really was 10.89.0.178 -- but the list it was compared against was the DEFAULT ["127.0.0.1", "::1"], because their config.json sat at a path the app does not read (the $CODERAI_CONFIG_DIR/coderai trap), which is equally why the file's `enabled` did nothing and the env var was needed. Loopback is in that default; the container address is not. The loader is fine and `enabled` from the file does suffice -- verified, not assumed. What was missing was a way to supply the peers the same way the switch and the secret can already be supplied, so CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_ PROXIES now overrides the list; blank or unset falls through to config.json. Enabling the feature from a unit file and then having it reject every request from your own proxy was a gap in the contract. And a refusal now says why, once per peer and reason per 60s, naming the list it was checked against and never the secret. A refusal is a 302 to the login form, indistinguishable from "no session yet"; the report had to ask for exactly this instrumentation, so it is permanent now. tests/test_forward_auth_peer_trust.py reproduces the peer resolution for all three hops, so the table in the doc is executable rather than reasoned. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Block-sparse attention tiles the latent into (4, 4, 4) chunks and asserts every axis divides evenly, so a side must be a multiple of 64 -- the VAE's 16px cell times 4. At height 480 the preset turned the flag on and the pass silently fell back to DENSE attention, which at ~37k tokens is both much slower and what ran a 24GB card out of memory on the first continuation segment of a 304-frame clip. 832 already divides by 64; 512 is the nearest height that does. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 304-frame request sat at "generating 0/16 (0%)" for its whole nine-minute first segment while the service's log showed Denoising: 15/16. Two defects. The service set _progress["step"] = 0 at the start of every segment and never again, so the step count existed only inside the vendored pipeline's tqdm(total=len(timesteps), desc="Denoising") -- which takes no callback. Nothing reached GET /progress; the poller in video.py faithfully reported the 0 it was handed. Each pipeline module did `from tqdm import tqdm`, so the bar is instrumented by rebinding the name on the MODULE (patching tqdm.tqdm would be too late), and only bars labelled "Denoising" are tracked so a checkpoint-shard bar cannot move a request's progress. The totals also counted ONE segment: a four-segment render would have run 0->16 four times against a denominator of 16. `step` now carries the completed segments' offset so it counts the stage end to end, and the bar spans steps x segments -- with refinement excluded, being a single pass over the finished video. A request that asked for a duration rather than a segment count only learns the real count from the service, so the denominator is corrected on the first report, through a setter that does not restart the bar and lose the elapsed time the rate is derived from. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Digesta needs to see what the fleet is doing, not only what it costs: how many pods are running, where, and which model. The spend endpoint already carried the pod rows, but a pod did not report its placement and there was no roll-up to read at a glance. A pod now carries and reports its data_center, cloud_type and gpu_count, plus the pool and engine it belongs to. GET /v1/runpod/status (an alias of /v1/runpod/spend, one implementation so the two can never disagree) adds a `summary`: pods_total/ready/booting/unhealthy, gpus_in_use, inflight, spot_pods, hourly_usd, and tallies by model, region, GPU and cloud type. Same gate as the spend figures — an admin-scoped API key, 403 otherwise. Three deliberate distinctions, documented: summary.hourly_usd counts only pods actually serving (a booting pod bills but answers nothing, so counting it would overstate capacity); pods_unhealthy is the alert-worthy number, a pod past boot that is costing money and serving nothing; and gpus_in_use counts cards, so a gpu_count:2 pod counts twice. An unknown region or GPU is labelled "unknown" rather than dropped, so the tallies always sum to pods_total. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Implements docs/admin-forward-auth.md. Off by default, switchable from Settings -> Single sign-on, from config.json, or from the container's env (CODERAI_ADMIN_FORWARD_AUTH=1|0, CODERAI_ADMIN_FORWARD_AUTH_SECRET) — the env switch can disable it as well as enable it, so a unit file can override a mounted config either way. A request is forward-authed only when BOTH checks pass: the immediate peer is in trusted_proxies AND the shared secret header matches (constant time). Either alone is forgeable, so the identity header is not even read until trust is established — a client sending X-Forwarded-User directly is ignored. Enabled with no secret stays OFF with a log line, because without one there is nothing separating the proxy from any other client. It mints a real signed session and injects it into the current request, not just a rendered page: the dashboard's own /admin/api/* calls authenticate by cookie, so a page with 401ing AJAX would not be a working GUI. An auto-created account gets a random password nobody is told and is never asked to change it. keep_local_login=false makes /login refuse. Logout honours post_logout_url, since returning to our own login would be met by the proxy's headers and log the user straight back in. GUI only: /v1/* keeps its bearer tokens whatever the proxy asserts, and the readiness probes stay credential-free. The settings API reports shared_secret_set as a boolean and never the secret. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reproduced on the published 0.2.83 image, GPU-less, no Authorization header: GET /v1/models -> 200 <- open POST /v1/chat/completions -> 503 (no model in the test, so inconclusive) GET /v1/runpod/spend -> 502 Nothing returned 401, because the front was never an enforcement point: it answers some /v1 paths from its own registry and proxies the rest, so whether a request needed a token depended on which component served it. On a host published through a reverse proxy that is an open relay for the model list, and potentially for generation. The front now gates /v1 itself with the same credentials the engine accepts: the env token a capability image is locked with, the shared cluster token a head presents, an API token from auth.json, a logged-in browser session, the in-process broker bridge, and the aisbf relay marker. Constant-time comparison throughout. One deliberate difference from the engine's middleware: it does NOT fall open when no credentials are configured. That fallback is defensible on a loopback workstation and is precisely what exposed this one, so the front refuses instead. Readiness and progress endpoints stay exempt — a probe that 401s is read as "backend down", and surya-ocr's vLLM client probes {service_url}/health. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reproduced the Digesta "engine#0 crash-loops despite backend.type=cpu and engine_specs" report on the published 0.2.83 image, GPU-less, with their config — and it is not the front failing to propagate the engine spec. coderai-entrypoint creates $CODERAI_CONFIG_DIR/coderai and symlinks ~/.coderai to it (lines 49, 70); the app resolves config from the HOME-style path. A config.json written at $CODERAI_CONFIG_DIR is therefore never read. On first start the app writes a DEFAULT coderai/config.json — backend null, engine_specs null, engines null — reads that, and the engine auto-detects, finds no GPU and aborts. Which is exactly the "backend: null" they saw and read as the front regenerating the config. Documents the live path, the one-liner to check which file is in use, and that mounting straight at /config/coderai sidesteps it. Also ships coderai-upgrade in Dockerfile.oci-venv. It was only in Dockerfile.oci, so every image we actually publish (venv-built) failed `coderai-docker --upgrade` with "stat /usr/local/bin/coderai-upgrade: no such file or directory". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide said the venv ships in an image. It does not, in either image, and it does not need to: the engine builds it itself from requirements-surya.txt onto the persistent cache mount — but only when ocr.surya_auto_build is true, and it defaults to FALSE (config.py:908). That default is why the Digesta bring-up built the venv by hand and called it a packaging gap. It is a one-line config setting on the orchestrator. Documents the pillow<11 vs >=12 conflict that forces the venv to be isolated, where it lands, and how to pre-build it instead. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide told the operator to install ghcr.io/nextime/coderai-ocr as the orchestrator. That is backwards and it is the mistake the Digesta bring-up actually made: a capability image is a POD, what CoderAI rents and launches, and it has no GUI; its admin pages are not behind the API's bearer check, so it is not something to expose either. Rewritten around the full image on port 8776, with the two settings a GPU-less host needs and the reason each one is needed: - backend.type must be "cpu", not "auto": with auto, main.py searches nvidia/vulkan/opencl and exits "No supported backend detected" when it finds none. Set explicitly, that search is skipped. - engine_specs must name the engine: left to auto-detect, the front creates a no-GPU engine NAMED cpu but launches it with backend="auto" (engine_supervisor.py:311), which hits the same abort whatever backend.type says. That is why the log reads "[cpu] Available backends: {'cpu': True}" and then "No supported backend detected" — [cpu] is the name, auto was the backend. Also states that server.engines=0 means AUTO (one per GPU, min 1), that no "zero engines" setting exists, and that you do not want one: the RunPod pools, scaler and reaper live inside the primary engine, so one idle CPU engine is the component that manages the pods, not a workaround. Service examples, the upgrade section and the reference table all follow. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pod image is never an orchestrator; the full image always is. Acting on the Digesta report's second option put orchestrator behaviour into codai.api.app, which is what a RENTED POD runs, and that is wrong in three separate ways: - the GUI pages are not under BearerAuthMiddleware (it guards /v1/* only), so every rented pod would have served /admin and /login on its public address; - the first-admin bootstrap would have created an admin account, with a password in the pod's log, in each pod's throwaway auth.json; - initialising the session manager would have broken allow_open_pod, where the operator deliberately launches a pod with no token: the pod would then refuse every request instead of accepting all of them. Reverted all three. Bug 1 stays — /health and /v1/health are a genuine pod-side fix, since surya probes {service_url}/health. The real Bug 3 is elsewhere and the report's premise was also wrong: server.engines=0 does not mean "no engines", it means AUTO — "one per detected GPU, min 1" (config.py:70) — which is why their host kept spawning engine#0. A true GPU-less orchestrator in the full image needs its own change, not a flag that already exists. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
My own check broke the build: `from surya.common.surya.schema import TaskNames` is not a path in surya-ocr 0.22.1, so the ocr image failed with ModuleNotFoundError at the venv deep check — I wrote an import from memory instead of reading what the worker uses. It now imports what codai/ocr/workers/ocr_worker.py actually constructs, DetectionPredictor and RecognitionPredictor, and keeps the pillow assertion. The test asserts those paths are present and that the invented one is gone. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Enforcing API tokens (f4482fd5) had a sharp edge: on a brand-new config dir there is no token, so every /v1 call is refused, AND no user, so nobody can log in to create one. Nothing in the codebase auto-creates that first user — grepped for it rather than assumed. The app now creates a single `admin` account at startup when auth.json has no users, with a random password printed once and must_change_password set, and logs plainly when there are users but no tokens. Idempotent: verified it never touches an existing user or rewrites its hash. The guide gains "You never configure a pod — CoderAI does": a table of what the orchestrator sets on a rented pod by itself (the per-pool cra- token so a pod is never open, TLS from this install's CA, seed models, engine args including tensor-parallel and LoRA modules, admission limits, the volume mount, idle teardown and the reaper) and the rule that follows — no SSH into a pod, no client pointed at a pod URL; throughput comes from max_pods and scale_up_inflight_per_pod, not from hammering one pod. Also corrects the guide's first-run text, which promised credentials the code did not actually create until now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found while checking the production proxy path. BearerAuthMiddleware falls OPEN when admin.routes.session_manager is None and no CODERAI_API_TOKEN is set: with no user database to check a token against it lets the request through rather than locking out an install with no auth configured. That is defensible on a loopback workstation. main.py and build_system_app() both initialise that database. A bare `uvicorn codai.api.app:app` start — exactly how the GPU-less orchestrator runs — did not, so once published through a reverse proxy every /v1/* call was served WITHOUT credentials. Verified before and after: session_manager None on import, /v1/models 200 with no header; now initialised at import and 401 for no header, a wrong Bearer, and nginx's Basic. The readiness probes stay open on purpose (/healthz, /health, /v1/health): a probe behind auth is read as "backend down" by every health checker, surya-ocr's vLLM client included. The guide gains the three traps of that deployment: auth_basic and Bearer share one header so the API path must not be basic-authed, proxy_buffering must be off for streaming, and how to check in one curl whether tokens are actually enforced. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide told an operator which keys exist but not how to lay out a real fleet. Adds a section covering: - why a network volume is the single biggest lever: without one a pod re-downloads its weights on EVERY cold start, which is ten minutes for a 15 GB repo and one to two hours for a 150 GB MoE, paid as rental before it answers anything; - that the volume's region overrides data_center and the account setting, because a pod elsewhere cannot attach it — so two regions means two volumes; - a suggested volume layout, and venv_on_volume; - pooling: two models sharing `pool` share pods, but only CoderAI-engine models can, since a vLLM pod is launched --model X; - worked blocks for vLLM, llama.cpp/GGUF, embeddings (warm + scheduled), native-engine OCR, and the big MoE engine pods, which REQUIRE a volume because colibri and k3 never download; - LoRA/QLoRA: portable adapters travel as a reference and vLLM gets --enable-lora wired automatically, while a local-only adapter forces a CoderAI pod because vLLM and llama.cpp have no endpoint to receive one; the volume is the durable answer. Plus the two QLoRA traps — a quantization that must match what the adapter was trained against (it loads fine and silently degrades otherwise), and llama.cpp taking exactly one GGUF-converted adapter; - a checklist for before you add the tenth model. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adds to the operator guide: a systemd unit and a rootless podman quadlet (the production host uses systemd), a sysvinit script for hosts without it, a wrapper script for the run line, and the in-image upgrade path. The service examples carry the two traps worth inheriting from the build machine's own scripts: the CONTAINER is the source of truth for "is it running", not the launcher, because dockerd outlives the shell and `run --rm --name` dies on a name conflict; and a boot-time shell has neither your env nor your PATH, which is why the live-code switch is a flag FILE and the runner path is spelled out. The upgrade section documents what `coderai-docker --upgrade` actually does (shallow clone, version gate, swap /opt/coderai/app, delta pip, exit 0/10, host commits back onto the same tag) and says plainly when NOT to use it: it mutates a tag in place, so afterwards the local image is no longer byte-identical to the signed one in the registry. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An operator guide for the deployment shape the Digesta host uses: no local GPU, models rented per request, scaled by the pool, billed into a capped ledger. Covers install and signature check, the sub-path nginx config, the global caps to set BEFORE adding a model, the per-model block, the four scaling numbers plus the two pod-side admission ones that otherwise cap a dedicated card at ~6 pages/s, the warm-pod schedule, all three layers of cost control (caps, GUI, /v1/runpod/spend with an admin-scoped key), the call shapes, and a symptom table. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Bug 3 from the Digesta report: no published image was both a GPU-less RunPod orchestrator and an admin GUI. The cause was a split nobody had noticed: admin_router is the admin API (POST /login, /admin/api/*), while the GUI PAGES — GET /admin, /login, /admin/models … — live in frontproxy/ui_pages.py and were registered only on the front app. An install running `uvicorn codai.api.app:app` as a pure orchestrator therefore shipped the entire admin API with no way to reach it: GET /admin answered 404 and GET /login 405, because only POST /login existed. Reproduced both exactly before changing anything. codai/api/app.py now registers those pages itself, against the same config dir the admin API writes, so one image is orchestrator + GUI. Guarded: a missing templates or config dir logs and leaves the API untouched, the same contract the front has. Verified: /admin 404->302 /login, GET /login 405->200, /admin/models and /admin/runpod redirect to login, and under X-Forwarded-Prefix=/coderai the redirect goes to /coderai/login. /healthz and /v1/models unaffected. Bug 4 (the light app 500ing under X-Forwarded-Prefix) did NOT reproduce: _ForwardedPrefixMiddleware is installed in this app and /coderai/healthz, /coderai/v1/models and the GUI all answer correctly with the header set, stripped or un-stripped. The reported 500 was most likely /admin itself (404 before this fix) surfacing through their proxy. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta host, where coderai runs GPU-less and proxies to A40 pods. 1. surya-ocr's vLLM client probes {service_url}/health and reads anything else as "backend down", so /v1/ocr never became ready against a coderai front that only served /healthz. /health and /v1/health now alias it, and /v1/health is exempt from the Bearer check — a readiness probe that needs a token is not a readiness probe. 2. A rented pod ran the shared-workstation admission defaults (2 in flight, 6 queued), so its own gate capped an A40 at ~6 pages/s and everything above 16 concurrent got 429 while the GPU idled. The pod has no config file, so the knobs now arrive as env (CODERAI_SERVER_MAX_PARALLEL_REQUESTS / _QUEUE_MAX_SIZE) and are set per model from the model page (pod_max_parallel_requests / pod_queue_max_size), seeded when the pod is provisioned. 3. The image baked no surya venv, and could not have: ocr.txt put surya-ocr in the MAIN venv, where its pillow<11 cap fights coderai's pillow>=12. It now has its own ocr.venv-surya profile with a deep check that proves the pin held. The venv belongs on the orchestrator, not the card: with surya_serve=vllm the model runs in vLLM on the pod and this side only renders and post-processes. Also a bug that fix exposed: _resolve_venv_dir looked only at /opt/coderai/<name>_venv while the image builder bakes /opt/coderai/venvs/<alias>, so a baked venv was invisible and got rebuilt on the cache mount at first use. ocr-paddle has been shipping a 2 GB paddleocr venv nothing ever found. Co-Authored-By:
-
- 07 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
The API already took the flag; without this, granting it meant editing auth.json by hand. Opt-in at creation, and a toggle per row. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three gaps, all configurable from the model page: - data_center pins a model's pods to one RunPod region. Precedence is volume > model > account: a pod in another region cannot attach the volume, so the volume still decides. - schedule_enabled/start/end/days/tz gate the WARM FLOOR only. Outside the window the floor is 0 and the warm pod is released without waiting out idle_timeout_s; a request arriving out of hours still cold-starts one, exactly as min_pods=0 always has. An end before the start runs overnight and belongs to the day it started on. An incomplete or unparseable window is treated as no schedule rather than a floor of zero, and an unknown time zone falls back to local, not UTC. - GET /v1/runpod/spend serves the pods, ledger, caps and schedule status to an API key marked admin, so an external monitor can poll spend without an admin session. Opt-in per token; an ordinary key gets 403. pools_schedule_status() reports in_window/next_change/effective_min_pods on /admin/api/runpod/stats and on the new endpoint. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The full image is 0.2.81 and its real size is ~25 GB on disk / ~14 GB pulled, not ~28 GB. Also say that capability images version independently and name the nine still sitting at 0.2.18. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
logs/, the empty .dev-code marker and a customer's integration requirements were being copied into the image by COPY . /opt/coderai/app. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
encode_prompt's signature defaults device to None. Upstream's own callers always pass it, so this never fires today, but a cached tuple returned on the CPU would meet a DiT on the card — and the fallback is one expression, where discovering it later is a device-mismatch traceback from inside a generation. Falls back the way _get_t5_prompt_embeds does. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A deferred encoder is not there when _place_pipeline runs, so pipe.to("cuda") has nothing to move. Under offload that was already right — the encoder belongs on the host and the wrapper brings it over for the encode — but with offload_strategy "" the encoder was built on the CPU and left there, while _get_t5_prompt_embeds sends it inputs on the card. The first cache miss would have died on a device mismatch. Placement cannot be re-derived on a miss: by then the offload argument is long gone. So _place_pipeline records where a realised encoder belongs, in each of its branches, and realise() replays it. A quantised encoder is exempt — device_map put it on the card as it loaded and bitsandbytes weights are not meant to be moved afterwards. That check now lives in _is_quantised_module, which _encoder_is_quantised delegates to, because a lazily realised encoder has to be asked the same question BEFORE it is attached to a pipeline and the two answers must not drift. test_quantisation_is_detected_from_the_model_not_the_config followed the names into the helper and now asserts the delegation too, so neither function can start consulting the config. Found by re-reading the committed diff, not by a test: nothing here runs the real pipeline, which is exactly the gap that let it through. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The UMT5-XXL text encoder is ~11 GB and turns a prompt into a few MB of embeddings that are a pure function of the prompt and the encoder. v0.2.76 made that run once per request instead of once per segment; this makes it once per prompt, ever. The saved GPU pass is not the point — it is seconds. The point is that a cached prompt needs no encoder AT ALL, so with the cache on the encoder is built lazily: ~11 GB of host RAM and about a minute of load that a repeated prompt never spends. That is the answer to "can we keep everything in VRAM": not by quantising the encoder (v0.2.79 measured int8 resident at 24.0 GB on a 24 GB card, and NF4 at 22.1 GB with less headroom than a 29-frame clip already uses), but by not having an encoder resident at all. Off by default — embeddings are derived data, but they are also several MB each under the user's cache directory, so growing that is opted into: "prompt_cache": "off", # off | disk "prompt_cache_dir": "", # blank = <coderai home>/prompt_cache/longcat "prompt_cache_max_gb": 0, # 0 = 2 GB, least-recently-used evicted per model or server-wide, validated at startup rather than after a seven-minute load, settable from the models page, and in api_model_configure's whitelist so a save from that page does not silently drop it. An entry served to the wrong pipeline would be a silent quality change, which is worse than a miss, so the key covers the checkpoint, the dtype, the encoder's quantisation and the call's own arguments — an NF4 encoder cannot be served a bf16 entry. `device` is deliberately excluded: it moves the embeddings without changing them, and including it would split one entry into one per device. Every failure is a miss, never an error. Entries are written to a .tmp and renamed, read with weights_only=True (a cache directory is not a trusted source for a pickle), deleted when unreadable, and a cache that cannot be written warns once and generation carries on without it. _LazyEncoder is deliberately not a transparent proxy: upstream guards every use with `if self.text_encoder is not None`, so a proxy that looked like a module would be moved to the card by pipe.to() and offloaded by _place_pipeline, building the very weights it exists to avoid. pipe.text_encoder stays None and the holder sits beside it in _state, which is why both paths already do the right thing. Found while wiring it: a lazily realised encoder can be quantised, and the offload swap would have shuttled bitsandbytes weights to the host. The swap guard now checks the model, not just the offload mode. /health reports the cache and whether the encoder was ever built — laziness is invisible otherwise — and GET /prompt-cache returns its stats. No new dependencies. Unverified on the card: the round trip, eviction, corrupt entries and the key are covered by 54 tests against real tensors, but no generation has run through it yet. Co-Authored-By:
-