- 09 Oct, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authoredrun_oci.sh takes its engine from CONTAINER_ENGINE and defaults to docker, so on the rootless-podman host this script was written for it would have looked for a docker socket that is not there -- failing on every hourly tick with nothing to show but a non-zero exit. Found while installing it on the Digesta production server, before it ran. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com>
-
Stefy Lanza (nextime / spora ) authored
The Digesta host added both 10.89.0.178 and ::ffff:10.89.0.178 to trusted_proxies and host->8777 still refused. The ::ffff: hunch was right: the internal nginx listens on 8776 AND [::]:8776, so a connection landing on the v6 socket makes $remote_addr -- and therefore X-Forwarded-For, and therefore the peer -- the IPv4-mapped form, which a string membership test rejects even when the plain address is listed. Nothing in the configuration explains that refusal. Peers are now compared as addresses: a mapped v6 peer matches a plain v4 entry and the reverse, and CIDR entries work, so a container network can be trusted as 10.89.0.0/24 instead of pinning one address in the quadlet. A non-address entry (a socket path) is still compared literally, and a v6 peer never matches a v4 network. Their broadened list could not have taken effect anyway -- it went into the config.json the app is not reading, which is why `enabled` there did nothing and the env var was needed. So that test did not rule the peer check out; the list in force was still the default pair. The rest of the report is answered in the doc rather than in code, because there is no bug behind it: `Server: nginx` appears on PROXIED responses too unless proxy_pass_header Server is set, and the missing `GET /admin` access-log line is _PollNoiseFilter, which drops reads of /admin, /static, /login and /logout from the front's access log while leaving /healthz alone -- exactly the asymmetry that looked like nginx short-circuiting by source. A 302 to /coderai/login with no Set-Cookie is what the front returns when forward-auth declines. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two production installs now follow this repo: zeiss (nexlab) and the Digesta host in Aruba. The Aruba one should track `production` without anyone logging in, so this adds coderai-autoupgrade plus a systemd user service and an hourly timer. It skips the upgrade outright while CoderAI is serving -- checked BEFORE the image is touched, not merely before the restart, so a busy host never reaches the state where new code waits in the image. Two probes and the higher count wins: the front's coderai_engine_inflight (every request in flight, pod-served ones included) and the pods' own summary.inflight from /v1/runpod/status. The RunPod one carries the weight on an orchestrator where every model is remote: the local engines are idle by definition, so the pods' count is the only sign work is happening, and killing a pod-served request loses the work AND keeps paying for the pod that was doing it. The check fails CLOSED -- no token, wrong URL or an unreachable endpoint skips the run, because an unreadable probe is nearly always a misconfiguration rather than an idle server. It decides by reading the image, not by an exit code: run_oci.sh exits 0 both when it upgraded and when the image was already current (the rc==10 branch prints its message and falls through to the same exit), so an exit-code-driven loop would have restarted the service every hour forever. The script compares __version__ baked in the image before and after, and a change is the only thing that triggers a restart. By default MAX_DEFERRALS=0, so a request in flight is never interrupted: the new code waits and every run says so. And it rolls back -- the pre-upgrade image is tagged first, and a service that is not healthy within HEALTH_TIMEOUT gets the old tag back, another restart and a non-zero exit that says whether it recovered. Writing the example config surfaced a trap worth the test that now guards it: the file is sourced, so `RESTART_CMD=systemctl --user restart coderai.service` unquoted assigns "systemctl" and RUNS the rest -- restarting nothing, failing the health check and rolling back a good upgrade. Everything in the example is quoted and each run echoes the restart command it resolved. Verified against the live server on zeiss: already-current does nothing; an overlapping run is a no-op; a missing token skips; three in-flight engine requests skip; and engines idle with a pod serving two still skips. Co-Authored-By:
-
- 08 Oct, 2026 18 commits
-
-
Stefy Lanza (nextime / spora ) authored
The Digesta host reported SSO minting a session from inside the container (:8776) but refusing from the host through the published port (:8777), with the container's address in trusted_proxies and the front's own access log showing that very address as the client. Both that and "enabled: true in config.json did not switch the feature on" are one cause. The peer _fa_trusted checks is not the socket peer: uvicorn installs ProxyHeadersMiddleware by default with trusted_hosts="127.0.0.1", so a hop arriving over loopback from the internal nginx has scope["client"] rewritten from X-Forwarded-For before any of our middleware runs. On a container network that is the address the proxy was SNATed to. So the log and the check agreed and the peer really was 10.89.0.178 -- but the list it was compared against was the DEFAULT ["127.0.0.1", "::1"], because their config.json sat at a path the app does not read (the $CODERAI_CONFIG_DIR/coderai trap), which is equally why the file's `enabled` did nothing and the env var was needed. Loopback is in that default; the container address is not. The loader is fine and `enabled` from the file does suffice -- verified, not assumed. What was missing was a way to supply the peers the same way the switch and the secret can already be supplied, so CODERAI_ADMIN_FORWARD_AUTH_TRUSTED_ PROXIES now overrides the list; blank or unset falls through to config.json. Enabling the feature from a unit file and then having it reject every request from your own proxy was a gap in the contract. And a refusal now says why, once per peer and reason per 60s, naming the list it was checked against and never the secret. A refusal is a 302 to the login form, indistinguishable from "no session yet"; the report had to ask for exactly this instrumentation, so it is permanent now. tests/test_forward_auth_peer_trust.py reproduces the peer resolution for all three hops, so the table in the doc is executable rather than reasoned. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Block-sparse attention tiles the latent into (4, 4, 4) chunks and asserts every axis divides evenly, so a side must be a multiple of 64 -- the VAE's 16px cell times 4. At height 480 the preset turned the flag on and the pass silently fell back to DENSE attention, which at ~37k tokens is both much slower and what ran a 24GB card out of memory on the first continuation segment of a 304-frame clip. 832 already divides by 64; 512 is the nearest height that does. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 304-frame request sat at "generating 0/16 (0%)" for its whole nine-minute first segment while the service's log showed Denoising: 15/16. Two defects. The service set _progress["step"] = 0 at the start of every segment and never again, so the step count existed only inside the vendored pipeline's tqdm(total=len(timesteps), desc="Denoising") -- which takes no callback. Nothing reached GET /progress; the poller in video.py faithfully reported the 0 it was handed. Each pipeline module did `from tqdm import tqdm`, so the bar is instrumented by rebinding the name on the MODULE (patching tqdm.tqdm would be too late), and only bars labelled "Denoising" are tracked so a checkpoint-shard bar cannot move a request's progress. The totals also counted ONE segment: a four-segment render would have run 0->16 four times against a denominator of 16. `step` now carries the completed segments' offset so it counts the stage end to end, and the bar spans steps x segments -- with refinement excluded, being a single pass over the finished video. A request that asked for a duration rather than a segment count only learns the real count from the service, so the denominator is corrected on the first report, through a setter that does not restart the bar and lose the elapsed time the rate is derived from. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Digesta needs to see what the fleet is doing, not only what it costs: how many pods are running, where, and which model. The spend endpoint already carried the pod rows, but a pod did not report its placement and there was no roll-up to read at a glance. A pod now carries and reports its data_center, cloud_type and gpu_count, plus the pool and engine it belongs to. GET /v1/runpod/status (an alias of /v1/runpod/spend, one implementation so the two can never disagree) adds a `summary`: pods_total/ready/booting/unhealthy, gpus_in_use, inflight, spot_pods, hourly_usd, and tallies by model, region, GPU and cloud type. Same gate as the spend figures — an admin-scoped API key, 403 otherwise. Three deliberate distinctions, documented: summary.hourly_usd counts only pods actually serving (a booting pod bills but answers nothing, so counting it would overstate capacity); pods_unhealthy is the alert-worthy number, a pod past boot that is costing money and serving nothing; and gpus_in_use counts cards, so a gpu_count:2 pod counts twice. An unknown region or GPU is labelled "unknown" rather than dropped, so the tallies always sum to pods_total. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Implements docs/admin-forward-auth.md. Off by default, switchable from Settings -> Single sign-on, from config.json, or from the container's env (CODERAI_ADMIN_FORWARD_AUTH=1|0, CODERAI_ADMIN_FORWARD_AUTH_SECRET) — the env switch can disable it as well as enable it, so a unit file can override a mounted config either way. A request is forward-authed only when BOTH checks pass: the immediate peer is in trusted_proxies AND the shared secret header matches (constant time). Either alone is forgeable, so the identity header is not even read until trust is established — a client sending X-Forwarded-User directly is ignored. Enabled with no secret stays OFF with a log line, because without one there is nothing separating the proxy from any other client. It mints a real signed session and injects it into the current request, not just a rendered page: the dashboard's own /admin/api/* calls authenticate by cookie, so a page with 401ing AJAX would not be a working GUI. An auto-created account gets a random password nobody is told and is never asked to change it. keep_local_login=false makes /login refuse. Logout honours post_logout_url, since returning to our own login would be met by the proxy's headers and log the user straight back in. GUI only: /v1/* keeps its bearer tokens whatever the proxy asserts, and the readiness probes stay credential-free. The settings API reports shared_secret_set as a boolean and never the secret. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reproduced on the published 0.2.83 image, GPU-less, no Authorization header: GET /v1/models -> 200 <- open POST /v1/chat/completions -> 503 (no model in the test, so inconclusive) GET /v1/runpod/spend -> 502 Nothing returned 401, because the front was never an enforcement point: it answers some /v1 paths from its own registry and proxies the rest, so whether a request needed a token depended on which component served it. On a host published through a reverse proxy that is an open relay for the model list, and potentially for generation. The front now gates /v1 itself with the same credentials the engine accepts: the env token a capability image is locked with, the shared cluster token a head presents, an API token from auth.json, a logged-in browser session, the in-process broker bridge, and the aisbf relay marker. Constant-time comparison throughout. One deliberate difference from the engine's middleware: it does NOT fall open when no credentials are configured. That fallback is defensible on a loopback workstation and is precisely what exposed this one, so the front refuses instead. Readiness and progress endpoints stay exempt — a probe that 401s is read as "backend down", and surya-ocr's vLLM client probes {service_url}/health. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reproduced the Digesta "engine#0 crash-loops despite backend.type=cpu and engine_specs" report on the published 0.2.83 image, GPU-less, with their config — and it is not the front failing to propagate the engine spec. coderai-entrypoint creates $CODERAI_CONFIG_DIR/coderai and symlinks ~/.coderai to it (lines 49, 70); the app resolves config from the HOME-style path. A config.json written at $CODERAI_CONFIG_DIR is therefore never read. On first start the app writes a DEFAULT coderai/config.json — backend null, engine_specs null, engines null — reads that, and the engine auto-detects, finds no GPU and aborts. Which is exactly the "backend: null" they saw and read as the front regenerating the config. Documents the live path, the one-liner to check which file is in use, and that mounting straight at /config/coderai sidesteps it. Also ships coderai-upgrade in Dockerfile.oci-venv. It was only in Dockerfile.oci, so every image we actually publish (venv-built) failed `coderai-docker --upgrade` with "stat /usr/local/bin/coderai-upgrade: no such file or directory". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide said the venv ships in an image. It does not, in either image, and it does not need to: the engine builds it itself from requirements-surya.txt onto the persistent cache mount — but only when ocr.surya_auto_build is true, and it defaults to FALSE (config.py:908). That default is why the Digesta bring-up built the venv by hand and called it a packaging gap. It is a one-line config setting on the orchestrator. Documents the pillow<11 vs >=12 conflict that forces the venv to be isolated, where it lands, and how to pre-build it instead. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide told the operator to install ghcr.io/nextime/coderai-ocr as the orchestrator. That is backwards and it is the mistake the Digesta bring-up actually made: a capability image is a POD, what CoderAI rents and launches, and it has no GUI; its admin pages are not behind the API's bearer check, so it is not something to expose either. Rewritten around the full image on port 8776, with the two settings a GPU-less host needs and the reason each one is needed: - backend.type must be "cpu", not "auto": with auto, main.py searches nvidia/vulkan/opencl and exits "No supported backend detected" when it finds none. Set explicitly, that search is skipped. - engine_specs must name the engine: left to auto-detect, the front creates a no-GPU engine NAMED cpu but launches it with backend="auto" (engine_supervisor.py:311), which hits the same abort whatever backend.type says. That is why the log reads "[cpu] Available backends: {'cpu': True}" and then "No supported backend detected" — [cpu] is the name, auto was the backend. Also states that server.engines=0 means AUTO (one per GPU, min 1), that no "zero engines" setting exists, and that you do not want one: the RunPod pools, scaler and reaper live inside the primary engine, so one idle CPU engine is the component that manages the pods, not a workaround. Service examples, the upgrade section and the reference table all follow. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pod image is never an orchestrator; the full image always is. Acting on the Digesta report's second option put orchestrator behaviour into codai.api.app, which is what a RENTED POD runs, and that is wrong in three separate ways: - the GUI pages are not under BearerAuthMiddleware (it guards /v1/* only), so every rented pod would have served /admin and /login on its public address; - the first-admin bootstrap would have created an admin account, with a password in the pod's log, in each pod's throwaway auth.json; - initialising the session manager would have broken allow_open_pod, where the operator deliberately launches a pod with no token: the pod would then refuse every request instead of accepting all of them. Reverted all three. Bug 1 stays — /health and /v1/health are a genuine pod-side fix, since surya probes {service_url}/health. The real Bug 3 is elsewhere and the report's premise was also wrong: server.engines=0 does not mean "no engines", it means AUTO — "one per detected GPU, min 1" (config.py:70) — which is why their host kept spawning engine#0. A true GPU-less orchestrator in the full image needs its own change, not a flag that already exists. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
My own check broke the build: `from surya.common.surya.schema import TaskNames` is not a path in surya-ocr 0.22.1, so the ocr image failed with ModuleNotFoundError at the venv deep check — I wrote an import from memory instead of reading what the worker uses. It now imports what codai/ocr/workers/ocr_worker.py actually constructs, DetectionPredictor and RecognitionPredictor, and keeps the pillow assertion. The test asserts those paths are present and that the invented one is gone. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Enforcing API tokens (f4482fd5) had a sharp edge: on a brand-new config dir there is no token, so every /v1 call is refused, AND no user, so nobody can log in to create one. Nothing in the codebase auto-creates that first user — grepped for it rather than assumed. The app now creates a single `admin` account at startup when auth.json has no users, with a random password printed once and must_change_password set, and logs plainly when there are users but no tokens. Idempotent: verified it never touches an existing user or rewrites its hash. The guide gains "You never configure a pod — CoderAI does": a table of what the orchestrator sets on a rented pod by itself (the per-pool cra- token so a pod is never open, TLS from this install's CA, seed models, engine args including tensor-parallel and LoRA modules, admission limits, the volume mount, idle teardown and the reaper) and the rule that follows — no SSH into a pod, no client pointed at a pod URL; throughput comes from max_pods and scale_up_inflight_per_pod, not from hammering one pod. Also corrects the guide's first-run text, which promised credentials the code did not actually create until now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found while checking the production proxy path. BearerAuthMiddleware falls OPEN when admin.routes.session_manager is None and no CODERAI_API_TOKEN is set: with no user database to check a token against it lets the request through rather than locking out an install with no auth configured. That is defensible on a loopback workstation. main.py and build_system_app() both initialise that database. A bare `uvicorn codai.api.app:app` start — exactly how the GPU-less orchestrator runs — did not, so once published through a reverse proxy every /v1/* call was served WITHOUT credentials. Verified before and after: session_manager None on import, /v1/models 200 with no header; now initialised at import and 401 for no header, a wrong Bearer, and nginx's Basic. The readiness probes stay open on purpose (/healthz, /health, /v1/health): a probe behind auth is read as "backend down" by every health checker, surya-ocr's vLLM client included. The guide gains the three traps of that deployment: auth_basic and Bearer share one header so the API path must not be basic-authed, proxy_buffering must be off for streaming, and how to check in one curl whether tokens are actually enforced. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide told an operator which keys exist but not how to lay out a real fleet. Adds a section covering: - why a network volume is the single biggest lever: without one a pod re-downloads its weights on EVERY cold start, which is ten minutes for a 15 GB repo and one to two hours for a 150 GB MoE, paid as rental before it answers anything; - that the volume's region overrides data_center and the account setting, because a pod elsewhere cannot attach it — so two regions means two volumes; - a suggested volume layout, and venv_on_volume; - pooling: two models sharing `pool` share pods, but only CoderAI-engine models can, since a vLLM pod is launched --model X; - worked blocks for vLLM, llama.cpp/GGUF, embeddings (warm + scheduled), native-engine OCR, and the big MoE engine pods, which REQUIRE a volume because colibri and k3 never download; - LoRA/QLoRA: portable adapters travel as a reference and vLLM gets --enable-lora wired automatically, while a local-only adapter forces a CoderAI pod because vLLM and llama.cpp have no endpoint to receive one; the volume is the durable answer. Plus the two QLoRA traps — a quantization that must match what the adapter was trained against (it loads fine and silently degrades otherwise), and llama.cpp taking exactly one GGUF-converted adapter; - a checklist for before you add the tenth model. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adds to the operator guide: a systemd unit and a rootless podman quadlet (the production host uses systemd), a sysvinit script for hosts without it, a wrapper script for the run line, and the in-image upgrade path. The service examples carry the two traps worth inheriting from the build machine's own scripts: the CONTAINER is the source of truth for "is it running", not the launcher, because dockerd outlives the shell and `run --rm --name` dies on a name conflict; and a boot-time shell has neither your env nor your PATH, which is why the live-code switch is a flag FILE and the runner path is spelled out. The upgrade section documents what `coderai-docker --upgrade` actually does (shallow clone, version gate, swap /opt/coderai/app, delta pip, exit 0/10, host commits back onto the same tag) and says plainly when NOT to use it: it mutates a tag in place, so afterwards the local image is no longer byte-identical to the signed one in the registry. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An operator guide for the deployment shape the Digesta host uses: no local GPU, models rented per request, scaled by the pool, billed into a capped ledger. Covers install and signature check, the sub-path nginx config, the global caps to set BEFORE adding a model, the per-model block, the four scaling numbers plus the two pod-side admission ones that otherwise cap a dedicated card at ~6 pages/s, the warm-pod schedule, all three layers of cost control (caps, GUI, /v1/runpod/spend with an admin-scoped key), the call shapes, and a symptom table. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Bug 3 from the Digesta report: no published image was both a GPU-less RunPod orchestrator and an admin GUI. The cause was a split nobody had noticed: admin_router is the admin API (POST /login, /admin/api/*), while the GUI PAGES — GET /admin, /login, /admin/models … — live in frontproxy/ui_pages.py and were registered only on the front app. An install running `uvicorn codai.api.app:app` as a pure orchestrator therefore shipped the entire admin API with no way to reach it: GET /admin answered 404 and GET /login 405, because only POST /login existed. Reproduced both exactly before changing anything. codai/api/app.py now registers those pages itself, against the same config dir the admin API writes, so one image is orchestrator + GUI. Guarded: a missing templates or config dir logs and leaves the API untouched, the same contract the front has. Verified: /admin 404->302 /login, GET /login 405->200, /admin/models and /admin/runpod redirect to login, and under X-Forwarded-Prefix=/coderai the redirect goes to /coderai/login. /healthz and /v1/models unaffected. Bug 4 (the light app 500ing under X-Forwarded-Prefix) did NOT reproduce: _ForwardedPrefixMiddleware is installed in this app and /coderai/healthz, /coderai/v1/models and the GUI all answer correctly with the header set, stripped or un-stripped. The reported 500 was most likely /admin itself (404 before this fix) surfacing through their proxy. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta host, where coderai runs GPU-less and proxies to A40 pods. 1. surya-ocr's vLLM client probes {service_url}/health and reads anything else as "backend down", so /v1/ocr never became ready against a coderai front that only served /healthz. /health and /v1/health now alias it, and /v1/health is exempt from the Bearer check — a readiness probe that needs a token is not a readiness probe. 2. A rented pod ran the shared-workstation admission defaults (2 in flight, 6 queued), so its own gate capped an A40 at ~6 pages/s and everything above 16 concurrent got 429 while the GPU idled. The pod has no config file, so the knobs now arrive as env (CODERAI_SERVER_MAX_PARALLEL_REQUESTS / _QUEUE_MAX_SIZE) and are set per model from the model page (pod_max_parallel_requests / pod_queue_max_size), seeded when the pod is provisioned. 3. The image baked no surya venv, and could not have: ocr.txt put surya-ocr in the MAIN venv, where its pillow<11 cap fights coderai's pillow>=12. It now has its own ocr.venv-surya profile with a deep check that proves the pin held. The venv belongs on the orchestrator, not the card: with surya_serve=vllm the model runs in vLLM on the pod and this side only renders and post-processes. Also a bug that fix exposed: _resolve_venv_dir looked only at /opt/coderai/<name>_venv while the image builder bakes /opt/coderai/venvs/<alias>, so a baked venv was invisible and got rebuilt on the cache mount at first use. ocr-paddle has been shipping a 2 GB paddleocr venv nothing ever found. Co-Authored-By:
-
- 07 Oct, 2026 15 commits
-
-
Stefy Lanza (nextime / spora ) authored
The API already took the flag; without this, granting it meant editing auth.json by hand. Opt-in at creation, and a toggle per row. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three gaps, all configurable from the model page: - data_center pins a model's pods to one RunPod region. Precedence is volume > model > account: a pod in another region cannot attach the volume, so the volume still decides. - schedule_enabled/start/end/days/tz gate the WARM FLOOR only. Outside the window the floor is 0 and the warm pod is released without waiting out idle_timeout_s; a request arriving out of hours still cold-starts one, exactly as min_pods=0 always has. An end before the start runs overnight and belongs to the day it started on. An incomplete or unparseable window is treated as no schedule rather than a floor of zero, and an unknown time zone falls back to local, not UTC. - GET /v1/runpod/spend serves the pods, ledger, caps and schedule status to an API key marked admin, so an external monitor can poll spend without an admin session. Opt-in per token; an ordinary key gets 403. pools_schedule_status() reports in_window/next_change/effective_min_pods on /admin/api/runpod/stats and on the new endpoint. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The full image is 0.2.81 and its real size is ~25 GB on disk / ~14 GB pulled, not ~28 GB. Also say that capability images version independently and name the nine still sitting at 0.2.18. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
logs/, the empty .dev-code marker and a customer's integration requirements were being copied into the image by COPY . /opt/coderai/app. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
encode_prompt's signature defaults device to None. Upstream's own callers always pass it, so this never fires today, but a cached tuple returned on the CPU would meet a DiT on the card — and the fallback is one expression, where discovering it later is a device-mismatch traceback from inside a generation. Falls back the way _get_t5_prompt_embeds does. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A deferred encoder is not there when _place_pipeline runs, so pipe.to("cuda") has nothing to move. Under offload that was already right — the encoder belongs on the host and the wrapper brings it over for the encode — but with offload_strategy "" the encoder was built on the CPU and left there, while _get_t5_prompt_embeds sends it inputs on the card. The first cache miss would have died on a device mismatch. Placement cannot be re-derived on a miss: by then the offload argument is long gone. So _place_pipeline records where a realised encoder belongs, in each of its branches, and realise() replays it. A quantised encoder is exempt — device_map put it on the card as it loaded and bitsandbytes weights are not meant to be moved afterwards. That check now lives in _is_quantised_module, which _encoder_is_quantised delegates to, because a lazily realised encoder has to be asked the same question BEFORE it is attached to a pipeline and the two answers must not drift. test_quantisation_is_detected_from_the_model_not_the_config followed the names into the helper and now asserts the delegation too, so neither function can start consulting the config. Found by re-reading the committed diff, not by a test: nothing here runs the real pipeline, which is exactly the gap that let it through. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The UMT5-XXL text encoder is ~11 GB and turns a prompt into a few MB of embeddings that are a pure function of the prompt and the encoder. v0.2.76 made that run once per request instead of once per segment; this makes it once per prompt, ever. The saved GPU pass is not the point — it is seconds. The point is that a cached prompt needs no encoder AT ALL, so with the cache on the encoder is built lazily: ~11 GB of host RAM and about a minute of load that a repeated prompt never spends. That is the answer to "can we keep everything in VRAM": not by quantising the encoder (v0.2.79 measured int8 resident at 24.0 GB on a 24 GB card, and NF4 at 22.1 GB with less headroom than a 29-frame clip already uses), but by not having an encoder resident at all. Off by default — embeddings are derived data, but they are also several MB each under the user's cache directory, so growing that is opted into: "prompt_cache": "off", # off | disk "prompt_cache_dir": "", # blank = <coderai home>/prompt_cache/longcat "prompt_cache_max_gb": 0, # 0 = 2 GB, least-recently-used evicted per model or server-wide, validated at startup rather than after a seven-minute load, settable from the models page, and in api_model_configure's whitelist so a save from that page does not silently drop it. An entry served to the wrong pipeline would be a silent quality change, which is worse than a miss, so the key covers the checkpoint, the dtype, the encoder's quantisation and the call's own arguments — an NF4 encoder cannot be served a bf16 entry. `device` is deliberately excluded: it moves the embeddings without changing them, and including it would split one entry into one per device. Every failure is a miss, never an error. Entries are written to a .tmp and renamed, read with weights_only=True (a cache directory is not a trusted source for a pickle), deleted when unreadable, and a cache that cannot be written warns once and generation carries on without it. _LazyEncoder is deliberately not a transparent proxy: upstream guards every use with `if self.text_encoder is not None`, so a proxy that looked like a module would be moved to the card by pipe.to() and offloaded by _place_pipeline, building the very weights it exists to avoid. pipe.text_encoder stays None and the holder sits beside it in _state, which is why both paths already do the right thing. Found while wiring it: a lazily realised encoder can be quantised, and the offload swap would have shuttled bitsandbytes weights to the host. The swap guard now checks the model, not just the offload mode. /health reports the cache and whether the encoder was ever built — laziness is invisible otherwise — and GET /prompt-cache returns its stats. No new dependencies. Unverified on the card: the round trip, eviction, corrupt entries and the key are covered by 54 tests against real tensors, but no generation has run through it yet. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Sweeping the pod images for the gap the LongCat one had: Triton compiles its CUDA driver module at RUNTIME, and only vllm and engines-kt carried a compiler. Thirteen other torch-bearing profiles did not. The risk is not theoretical and not confined to block-sparse attention. core.txt — which every capability image installs — carries bitsandbytes, and bitsandbytes reaches Triton's autotuner on import; that is how the NF4 verification failed first time round, with "libcuda.so cannot found!" and then a failed gcc invocation. On a pod the first quantised forward would have died with "Failed to find C compiler", exactly as Surya did locally before v0.2.62. So gcc and libc6-dev go into Dockerfile.capability-base, where every profile image inherits them, rather than into thirteen profiles one at a time. capability-base-light is deliberately left without one: it ships no torch, so it has no Triton to build for. That is precisely why the LongCat image installs its own — it adds a 3.10 venv with torch and triton to a parent that has neither — and there is now a test saying so, so a later cleanup does not "simplify" it away. Also: the quantised text-encoder load now reads torch.cuda.current_device() instead of hardcoding 0. The worker masks the card with CUDA_VISIBLE_DEVICES so 0 was right here, but assuming it would not survive an install that does its device selection differently. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
bitsandbytes NF4 works — verified on the 3090, in the LongCat venv, loading the real UMT5-XXL: 144 Linear4bit modules, is_loaded_in_4bit=True, finite embeddings of the right shape. But it does not cost what v0.2.77 claimed. bitsandbytes replaces Linear layers only. This encoder's 256k-token embedding table and one linear per layer stay at the load dtype, leaving 6.13 GB that cannot shrink. Measured on the card, against ~11.0 GB at bf16: nf4 + double quant 8.00 GB (the bits-per-weight estimate said 2.8) int8 9.76 GB (it said 5.6) So the estimate now comes from those measurements rather than the bit width, because under-reserving by 3x is an OOM at load time. What the card must hold: bf16 encoder, offloaded 13.9 GB 9.7 GB headroom nf4 encoder, resident 22.1 GB 1.4 GB headroom int8 encoder, resident 24.0 GB does not fit bf16 encoder, resident 25.2 GB does not fit Which inverts the recommendation this was built for: an NF4 encoder does fit a 24 GB card, but leaves less room for activations than a 29-frame clip already used, so it would OOM on anything longer. Offload plus the per-request embedding cache from v0.2.76 keeps 9.7 GB free and touches the encoder once, so it stays the better configuration here. The setting remains, correctly sized, for cards where it is the right trade. The model page now states the measured figures too, since "nf4 — ~2.8 GB" would have invited exactly the configuration that OOMs. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Configurable means configurable from the UI, not only from models.json. The model page now carries the four settings added over the last commits — text encoder quantisation, block-sparse attention, the base checkpoint an avatar family borrows tokenizer/text_encoder/vae from, and the context-parallel tile — each with the trade-off in the option text, because "nf4" means nothing without "~2.8 GB, stays resident" beside it. They are in api_model_configure's whitelist too, or the page would offer settings it then dropped on save. And the RunPod image: block-sparse attention is pure Triton, Triton compiles its CUDA driver module the first time it runs, and neither capability base image carries a compiler. A pod would have failed with "Failed to find C compiler" on the first denoising step — exactly how Surya failed in the local image before v0.2.62. gcc and libc6-dev added, and the compiler test now covers this Dockerfile alongside the two local ones, since checking only some of them is how the gap got there in the first place. Only this one pod image installs requirements-longcat.txt, so it is the whole blast radius of the accelerate and bitsandbytes pins; the main venv already carried bitsandbytes for the diffusers 4-bit path, in a separate venv, so there is no conflict. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The DiT has upstream's INT8 path. The UMT5-XXL text encoder does not, and at bf16 it is ~11 GB — 45% of why a 13.6 GB INT8 DiT still overflows a 24 GB card (13.6 + 11.2 + VAE = 25.1 GB before a single activation). Offload worked around it by making the DiT and the encoder take turns; quantising the encoder removes the reason to. Measured against the real checkpoint, what the card must hold: bf16 encoder, offloaded 13.9 GB bf16 encoder, resident 25.2 GB <- the configuration that OOMs int8 encoder, resident 19.5 GB nf4 encoder, resident 16.7 GB So `text_encoder_quant` is a setting: none | int8 | nf4 | fp4, per model or server-wide, validated at startup rather than after a seven-minute load. bitsandbytes is pinned in requirements-longcat.txt and asserted by both the pod image build and the local image's smoke test. A quantised encoder is RESIDENT and never offloaded: bitsandbytes quantises as the weights land on CUDA and is not built to shuttle them back — and at 2.8 GB it does not need to. That is detected from the MODEL, not the config, so a pipeline loaded quantised cannot be offloaded by a config that changed since, and defensively, so a component that cannot be walked is simply not quantised. Inserting these helpers by anchor dropped one of them between @contextlib.contextmanager and the function it was meant to decorate, so _encoder_is_quantised returned a context manager — truthy, making every pipeline look quantised — and _bsa_for quietly stopped being one. Both now have a test asserting their shape, because the symptom was five unrelated failures. Unverified: bitsandbytes has not executed on this GPU. A throwaway container cannot reach the card here (torch reports no CUDA devices under both --gpus all and --runtime=nvidia, though the production container's own CUDA is fine), so the NF4 path is proven only by its unit tests until a service run exercises it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
_run_segments calls generate_t2v once PER SEGMENT, and every one of those calls self.encode_prompt. A one-minute video at 15 fps is 12 segments, so that was 12 identical encodes of the same prompt — and under offload each one also moved the 13.6 GB DiT off the card and the ~11 GB text encoder on and back again. About 35 GB over PCIe per segment, twelve times, for an answer that cannot change. The embeddings are a few MB, so they are now cached on the arguments that produce them, bounded to the last four. Segment 2 onward never touches the encoder: for the rest of a long generation the card holds nothing but the DiT and the VAE. Caching applies without offload too, where it saves the redundant forwards if not the transfers. The wrapper passes its arguments through verbatim instead of naming the first two — naming them baked in an assumption about upstream's signature that the avatar pipeline need not share, and it collided on the first test that used a differently-shaped one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
File "codai/api/video.py", line 4176, in _generate_longcat payload["audio"] = _as_b64(_audio) NameError: name '_as_b64' is not defined _as_b64 was defined INSIDE _generate_h3, MiniMax H3's handler, and _generate_longcat referenced it across that scope. So every LongCat path that carries binary input raised NameError before reaching the worker: the avatar audio track (at2v), the reference image (ai2v and i2v), and continuation frames. Text-to-video is the only path that never touches it, which is why t2v was the only one that ever worked. It cost a 20-minute wait for the GPU to find out, because the NameError fires after the model is loaded and the swap gate has handed over. Now module level, beside the _decode_b64_or_url it wraps. The test walks the AST for a call site in a function that neither defines it nor is nested in one that does, so the same mistake in another backend fails here instead of on a rented GPU. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.2.69 turned the dense backend OFF when enabling BSA, reasoning that _process_attn tests enable_bsa first so anything else would be a silent second opinion. That is true of SELF-attention and wrong about the model: attention.py:248, _process_cross_attn elif self.enable_xformers: ... else: raise RuntimeError("Unsupported attention operations.") Cross-attention has no enable_bsa branch at all. With the dense backend off, every cross-attention call in all 48 blocks raised — the first real BSA run died on step 0 of the first generation. So enable_bsa is now set alongside the chosen dense backend: BSA wins for self-attention on video because it is tested first, and cross-attention keeps a path to run on. Measured on the 3090, 29 frames, warm steady-state per step: dense 480x832 (30 latent rows) 8.13 s/it BSA 448x832 (28 latent rows) 6.34 s/it Adjusting for the 6.7% token difference the two geometries force (BSA needs sides divisible by 64), that is roughly 10-15% per step — well short of the 16x the 0.9375 sparsity suggests, because at this size attention is not the dominant cost; the weight-only INT8 linears are. BSA should matter more at 93 frames and 720p, where attention grows quadratically. It also pays a one-time Triton compile of about four minutes per service. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Turning bsa on produced a bare AssertionError out of the first denoising step: File ".../block_sparse_attention/bsa_interface.py", line 639 assert Tq % tq == 0 and Hq % hq == 0 and Wq % wq == 0 flash_attn_bsa_3d tiles the latent into 3D chunks and needs it to divide evenly on every axis. With the checkpoint's shipped chunk of (4, 4, 4) that means sides divisible by 64 — the VAE's 16-pixel latent cell times 4 — and a latent depth divisible by 4, which puts num_frames at 13, 29, 45, 61, 77, 93 … and NOT at the step*k+1 the 4n+1 generation rule would suggest. Upstream's own default 480x832 fails it (480/16 = 30), which is presumably why the flag ships off: nothing in that AssertionError mentions geometry. bsa_problems() now says which axis is wrong and what to try instead, with the chunk read from the checkpoint's bsa_params rather than assumed. The service checks it per pass and, when the shape cannot be tiled, runs dense attention for that pass and logs why: turning the flag on in a config must not break requests that already work, and a silent fallback would leave someone wondering why nothing got faster. The flag is restored afterwards, so the next request with a tileable shape still gets it. The suggestion arithmetic was wrong first time round — it offered 33, which is exactly the frame count that fails — and the test that checks every suggestion is itself tileable is what caught it. Co-Authored-By:
-
- 06 Oct, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
LongCat-Video-Avatar-1.5 is a transformer and an audio encoder: base_model/, base_model_int8/, lora/dmd_lora, scheduler/, whisper-large-v3/. It has NO tokenizer, text_encoder or vae. Its README says so in its own frontmatter (`base_model: meituan-longcat/LongCat-Video`) and upstream's instructions download both repos side by side. Validated against the base checkpoint's layout it reported four missing directories that were never part of the repo, and the loader would then have asked from_pretrained for a text_encoder/ that does not exist — and for subfolder="dit", where the avatar transformer is base_model/. So the layout is family-aware now. An avatar checkpoint contributes its scheduler, its transformer (base_model/ at bf16, base_model_int8/ at int8) and its LoRA; tokenizer/text_encoder/vae are borrowed from a base checkpoint, named by `base_model` per model or falling back to the upstream repo id, which resolve_checkpoint() turns into a local snapshot. A base checkpoint that is absent or incomplete is reported AS the base checkpoint, rather than as a tokenizer missing from the avatar repo, which sends you looking in the wrong place. Either transformer alone is enough to validate: base_model/ is 31.7 GB of bf16 that cannot fit a 24 GB card, so it is deliberately not downloaded here. Verified against the real checkpoint: family avatar-1.5, shared components resolving to the base snapshot, 4 INT8 shards with 2241 keys matching the index and 633 quantised Linear layers (the base DiT has 437 — the difference is the audio cross-attention), dmd_lora present. All three validators return []. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The estimator asked a 24 GB card for 74.3 GB and evicted every other model on the box before each generation. It sizes a model by summing its whole HF cache entry, and LongCat's holds three things that are never all resident: the bf16 DiT (51 GB, stored fp32), the INT8 DiT built beside it (13.6 GB), and an fp32 UMT5-XXL text encoder (22 GB). _longcat_resident_gb now counts only the DiT that `variant` selects, scales each component from its OWN stored dtype — one ratio cannot serve both an INT8 DiT that loads as it is stored and an fp32 encoder cast to bf16 — and, when offload is on, takes max(DiT, encoder) + VAE, because they take turns on the card. The dtype comes from the safetensors header, which is one small read per component and never touches the weights. Returning 0.0 leaves an unsizable model to the generic path rather than claiming it needs nothing. Measured against the real checkpoint: 13.9 GB for int8+offload, where the run actually used 13.6 GB. It was 74.3 GB. And accelerate, which upstream does not pin: without it transformers falls back to low_cpu_mem_usage=False and materialises the whole fp32 encoder in host RAM before casting it to bf16. Pinned in requirements-longcat.txt and asserted by BOTH the pod image build and the local image's smoke test, so a rebuild cannot quietly drop it. What accelerate does NOT fix, measured after installing it: the load time. The 5 shards still take ~63 s each, because /AI is ext4 on a spinning disk that reads at 28 MB/s cold — 22 GB of fp32 is ~5 minutes of pure I/O. accelerate removes the host-RAM spike, not the wait. Halving the bytes read (a bf16 copy of the encoder) or moving the checkpoint to NVMe is what would shorten it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two defects between a loading model and a finished clip, both found by running it. A repo id is not a directory. models.json holds what the model page registered — "meituan-longcat/LongCat-Video" — and resolve_model_path handed it straight to the service, which opens config.json, the VAE and the INT8 shards by path. Every load died with "checkpoint directory does not exist" before reading a byte. resolve_checkpoint() maps a repo id onto its newest snapshot in the hub cache, by hand rather than through huggingface_hub so the server's 3.13 venv and the service's 3.10 venv cannot disagree about where the weights are, and so a checkpoint that was never fetched stays an error instead of becoming a silent 74 GB download. And the offload added in v0.2.67 was only half of one. It held the text encoder off the card until encode_prompt and then brought it over WITH the DiT still resident: 13.6 GB of INT8 DiT plus ~11 GB of UMT5-XXL is 24.8 GB on a 24 GB card. It OOM'd on the first prompt of the first generation. Model offload means only the active component is on the card, so they take turns now — the DiT steps off for the encode and comes back for the loop, in a finally, because a failed prompt that stranded the DiT on the host would leave every later generation running at PCIe speed with nothing to say why. Peak is max(), not sum. The placement test passed on the broken version: it asserted the encoder arrived and left, never that the DiT was not there too. It asserts co-residency now, and fails on the old code. Measured end to end on the 3090: 33 frames at 480x832, distilled stage, 16 steps at 9.4 s/it, 13.6 GB resident, HTTP 200 with the mp4. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Attention is the expensive part at video token counts — ~37k tokens for 93 frames at 480x832, and it is O(N^2) in that. The checkpoint already carries bsa_params with sparsity 0.9375, roughly a 16x cut, and upstream's implementation is pure Triton, so none of it needs the flash-attn source build. So it is now a setting: longcat.bsa server-wide, "bsa" per model, --bsa on the service. Off by default because it is an approximation and video shows approximation as temporal flicker, not one soft frame. "auto" stays off when triton is missing; "on" says so instead, because someone who asked for it would otherwise be left wondering why nothing got faster. BSA replaces the dense backend rather than joining it, since Attention tests enable_bsa first. And cp_split_hw, found while wiring this: the DiT's forward opens with `if self.cp_split_hw[0] * self.cp_split_hw[1] > 1:` and has no None guard, but dit/config.json ships "cp_split_hw": null — so a DiT built from the config as published raises TypeError on its first forward. Our only assignment made it worse by storing cp_size, a process count, where a [h, w] pair belongs: an int is not subscriptable either. It defaults to [1, 1] now, derives [1, cp_size] under context parallelism, and takes an explicit tile that is checked against the group size. Co-Authored-By:
-