- 08 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authoredEnforcing API tokens (f4482fd5) had a sharp edge: on a brand-new config dir there is no token, so every /v1 call is refused, AND no user, so nobody can log in to create one. Nothing in the codebase auto-creates that first user — grepped for it rather than assumed. The app now creates a single `admin` account at startup when auth.json has no users, with a random password printed once and must_change_password set, and logs plainly when there are users but no tokens. Idempotent: verified it never touches an existing user or rewrites its hash. The guide gains "You never configure a pod — CoderAI does": a table of what the orchestrator sets on a rented pod by itself (the per-pool cra- token so a pod is never open, TLS from this install's CA, seed models, engine args including tensor-parallel and LoRA modules, admission limits, the volume mount, idle teardown and the reaper) and the rule that follows — no SSH into a pod, no client pointed at a pod URL; throughput comes from max_pods and scale_up_inflight_per_pod, not from hammering one pod. Also corrects the guide's first-run text, which promised credentials the code did not actually create until now. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com>
-
Stefy Lanza (nextime / spora ) authored
Found while checking the production proxy path. BearerAuthMiddleware falls OPEN when admin.routes.session_manager is None and no CODERAI_API_TOKEN is set: with no user database to check a token against it lets the request through rather than locking out an install with no auth configured. That is defensible on a loopback workstation. main.py and build_system_app() both initialise that database. A bare `uvicorn codai.api.app:app` start — exactly how the GPU-less orchestrator runs — did not, so once published through a reverse proxy every /v1/* call was served WITHOUT credentials. Verified before and after: session_manager None on import, /v1/models 200 with no header; now initialised at import and 401 for no header, a wrong Bearer, and nginx's Basic. The readiness probes stay open on purpose (/healthz, /health, /v1/health): a probe behind auth is read as "backend down" by every health checker, surya-ocr's vLLM client included. The guide gains the three traps of that deployment: auth_basic and Bearer share one header so the API path must not be basic-authed, proxy_buffering must be off for streaming, and how to check in one curl whether tokens are actually enforced. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The guide told an operator which keys exist but not how to lay out a real fleet. Adds a section covering: - why a network volume is the single biggest lever: without one a pod re-downloads its weights on EVERY cold start, which is ten minutes for a 15 GB repo and one to two hours for a 150 GB MoE, paid as rental before it answers anything; - that the volume's region overrides data_center and the account setting, because a pod elsewhere cannot attach it — so two regions means two volumes; - a suggested volume layout, and venv_on_volume; - pooling: two models sharing `pool` share pods, but only CoderAI-engine models can, since a vLLM pod is launched --model X; - worked blocks for vLLM, llama.cpp/GGUF, embeddings (warm + scheduled), native-engine OCR, and the big MoE engine pods, which REQUIRE a volume because colibri and k3 never download; - LoRA/QLoRA: portable adapters travel as a reference and vLLM gets --enable-lora wired automatically, while a local-only adapter forces a CoderAI pod because vLLM and llama.cpp have no endpoint to receive one; the volume is the durable answer. Plus the two QLoRA traps — a quantization that must match what the adapter was trained against (it loads fine and silently degrades otherwise), and llama.cpp taking exactly one GGUF-converted adapter; - a checklist for before you add the tenth model. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adds to the operator guide: a systemd unit and a rootless podman quadlet (the production host uses systemd), a sysvinit script for hosts without it, a wrapper script for the run line, and the in-image upgrade path. The service examples carry the two traps worth inheriting from the build machine's own scripts: the CONTAINER is the source of truth for "is it running", not the launcher, because dockerd outlives the shell and `run --rm --name` dies on a name conflict; and a boot-time shell has neither your env nor your PATH, which is why the live-code switch is a flag FILE and the runner path is spelled out. The upgrade section documents what `coderai-docker --upgrade` actually does (shallow clone, version gate, swap /opt/coderai/app, delta pip, exit 0/10, host commits back onto the same tag) and says plainly when NOT to use it: it mutates a tag in place, so afterwards the local image is no longer byte-identical to the signed one in the registry. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An operator guide for the deployment shape the Digesta host uses: no local GPU, models rented per request, scaled by the pool, billed into a capped ledger. Covers install and signature check, the sub-path nginx config, the global caps to set BEFORE adding a model, the per-model block, the four scaling numbers plus the two pod-side admission ones that otherwise cap a dedicated card at ~6 pages/s, the warm-pod schedule, all three layers of cost control (caps, GUI, /v1/runpod/spend with an admin-scoped key), the call shapes, and a symptom table. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Bug 3 from the Digesta report: no published image was both a GPU-less RunPod orchestrator and an admin GUI. The cause was a split nobody had noticed: admin_router is the admin API (POST /login, /admin/api/*), while the GUI PAGES — GET /admin, /login, /admin/models … — live in frontproxy/ui_pages.py and were registered only on the front app. An install running `uvicorn codai.api.app:app` as a pure orchestrator therefore shipped the entire admin API with no way to reach it: GET /admin answered 404 and GET /login 405, because only POST /login existed. Reproduced both exactly before changing anything. codai/api/app.py now registers those pages itself, against the same config dir the admin API writes, so one image is orchestrator + GUI. Guarded: a missing templates or config dir logs and leaves the API untouched, the same contract the front has. Verified: /admin 404->302 /login, GET /login 405->200, /admin/models and /admin/runpod redirect to login, and under X-Forwarded-Prefix=/coderai the redirect goes to /coderai/login. /healthz and /v1/models unaffected. Bug 4 (the light app 500ing under X-Forwarded-Prefix) did NOT reproduce: _ForwardedPrefixMiddleware is installed in this app and /coderai/healthz, /coderai/v1/models and the GUI all answer correctly with the header set, stripped or un-stripped. The reported 500 was most likely /admin itself (404 before this fix) surfacing through their proxy. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta host, where coderai runs GPU-less and proxies to A40 pods. 1. surya-ocr's vLLM client probes {service_url}/health and reads anything else as "backend down", so /v1/ocr never became ready against a coderai front that only served /healthz. /health and /v1/health now alias it, and /v1/health is exempt from the Bearer check — a readiness probe that needs a token is not a readiness probe. 2. A rented pod ran the shared-workstation admission defaults (2 in flight, 6 queued), so its own gate capped an A40 at ~6 pages/s and everything above 16 concurrent got 429 while the GPU idled. The pod has no config file, so the knobs now arrive as env (CODERAI_SERVER_MAX_PARALLEL_REQUESTS / _QUEUE_MAX_SIZE) and are set per model from the model page (pod_max_parallel_requests / pod_queue_max_size), seeded when the pod is provisioned. 3. The image baked no surya venv, and could not have: ocr.txt put surya-ocr in the MAIN venv, where its pillow<11 cap fights coderai's pillow>=12. It now has its own ocr.venv-surya profile with a deep check that proves the pin held. The venv belongs on the orchestrator, not the card: with surya_serve=vllm the model runs in vLLM on the pod and this side only renders and post-processes. Also a bug that fix exposed: _resolve_venv_dir looked only at /opt/coderai/<name>_venv while the image builder bakes /opt/coderai/venvs/<alias>, so a baked venv was invisible and got rebuilt on the cache mount at first use. ocr-paddle has been shipping a 2 GB paddleocr venv nothing ever found. Co-Authored-By:
-
- 07 Oct, 2026 15 commits
-
-
Stefy Lanza (nextime / spora ) authored
The API already took the flag; without this, granting it meant editing auth.json by hand. Opt-in at creation, and a toggle per row. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three gaps, all configurable from the model page: - data_center pins a model's pods to one RunPod region. Precedence is volume > model > account: a pod in another region cannot attach the volume, so the volume still decides. - schedule_enabled/start/end/days/tz gate the WARM FLOOR only. Outside the window the floor is 0 and the warm pod is released without waiting out idle_timeout_s; a request arriving out of hours still cold-starts one, exactly as min_pods=0 always has. An end before the start runs overnight and belongs to the day it started on. An incomplete or unparseable window is treated as no schedule rather than a floor of zero, and an unknown time zone falls back to local, not UTC. - GET /v1/runpod/spend serves the pods, ledger, caps and schedule status to an API key marked admin, so an external monitor can poll spend without an admin session. Opt-in per token; an ordinary key gets 403. pools_schedule_status() reports in_window/next_change/effective_min_pods on /admin/api/runpod/stats and on the new endpoint. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The full image is 0.2.81 and its real size is ~25 GB on disk / ~14 GB pulled, not ~28 GB. Also say that capability images version independently and name the nine still sitting at 0.2.18. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
logs/, the empty .dev-code marker and a customer's integration requirements were being copied into the image by COPY . /opt/coderai/app. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
encode_prompt's signature defaults device to None. Upstream's own callers always pass it, so this never fires today, but a cached tuple returned on the CPU would meet a DiT on the card — and the fallback is one expression, where discovering it later is a device-mismatch traceback from inside a generation. Falls back the way _get_t5_prompt_embeds does. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A deferred encoder is not there when _place_pipeline runs, so pipe.to("cuda") has nothing to move. Under offload that was already right — the encoder belongs on the host and the wrapper brings it over for the encode — but with offload_strategy "" the encoder was built on the CPU and left there, while _get_t5_prompt_embeds sends it inputs on the card. The first cache miss would have died on a device mismatch. Placement cannot be re-derived on a miss: by then the offload argument is long gone. So _place_pipeline records where a realised encoder belongs, in each of its branches, and realise() replays it. A quantised encoder is exempt — device_map put it on the card as it loaded and bitsandbytes weights are not meant to be moved afterwards. That check now lives in _is_quantised_module, which _encoder_is_quantised delegates to, because a lazily realised encoder has to be asked the same question BEFORE it is attached to a pipeline and the two answers must not drift. test_quantisation_is_detected_from_the_model_not_the_config followed the names into the helper and now asserts the delegation too, so neither function can start consulting the config. Found by re-reading the committed diff, not by a test: nothing here runs the real pipeline, which is exactly the gap that let it through. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The UMT5-XXL text encoder is ~11 GB and turns a prompt into a few MB of embeddings that are a pure function of the prompt and the encoder. v0.2.76 made that run once per request instead of once per segment; this makes it once per prompt, ever. The saved GPU pass is not the point — it is seconds. The point is that a cached prompt needs no encoder AT ALL, so with the cache on the encoder is built lazily: ~11 GB of host RAM and about a minute of load that a repeated prompt never spends. That is the answer to "can we keep everything in VRAM": not by quantising the encoder (v0.2.79 measured int8 resident at 24.0 GB on a 24 GB card, and NF4 at 22.1 GB with less headroom than a 29-frame clip already uses), but by not having an encoder resident at all. Off by default — embeddings are derived data, but they are also several MB each under the user's cache directory, so growing that is opted into: "prompt_cache": "off", # off | disk "prompt_cache_dir": "", # blank = <coderai home>/prompt_cache/longcat "prompt_cache_max_gb": 0, # 0 = 2 GB, least-recently-used evicted per model or server-wide, validated at startup rather than after a seven-minute load, settable from the models page, and in api_model_configure's whitelist so a save from that page does not silently drop it. An entry served to the wrong pipeline would be a silent quality change, which is worse than a miss, so the key covers the checkpoint, the dtype, the encoder's quantisation and the call's own arguments — an NF4 encoder cannot be served a bf16 entry. `device` is deliberately excluded: it moves the embeddings without changing them, and including it would split one entry into one per device. Every failure is a miss, never an error. Entries are written to a .tmp and renamed, read with weights_only=True (a cache directory is not a trusted source for a pickle), deleted when unreadable, and a cache that cannot be written warns once and generation carries on without it. _LazyEncoder is deliberately not a transparent proxy: upstream guards every use with `if self.text_encoder is not None`, so a proxy that looked like a module would be moved to the card by pipe.to() and offloaded by _place_pipeline, building the very weights it exists to avoid. pipe.text_encoder stays None and the holder sits beside it in _state, which is why both paths already do the right thing. Found while wiring it: a lazily realised encoder can be quantised, and the offload swap would have shuttled bitsandbytes weights to the host. The swap guard now checks the model, not just the offload mode. /health reports the cache and whether the encoder was ever built — laziness is invisible otherwise — and GET /prompt-cache returns its stats. No new dependencies. Unverified on the card: the round trip, eviction, corrupt entries and the key are covered by 54 tests against real tensors, but no generation has run through it yet. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Sweeping the pod images for the gap the LongCat one had: Triton compiles its CUDA driver module at RUNTIME, and only vllm and engines-kt carried a compiler. Thirteen other torch-bearing profiles did not. The risk is not theoretical and not confined to block-sparse attention. core.txt — which every capability image installs — carries bitsandbytes, and bitsandbytes reaches Triton's autotuner on import; that is how the NF4 verification failed first time round, with "libcuda.so cannot found!" and then a failed gcc invocation. On a pod the first quantised forward would have died with "Failed to find C compiler", exactly as Surya did locally before v0.2.62. So gcc and libc6-dev go into Dockerfile.capability-base, where every profile image inherits them, rather than into thirteen profiles one at a time. capability-base-light is deliberately left without one: it ships no torch, so it has no Triton to build for. That is precisely why the LongCat image installs its own — it adds a 3.10 venv with torch and triton to a parent that has neither — and there is now a test saying so, so a later cleanup does not "simplify" it away. Also: the quantised text-encoder load now reads torch.cuda.current_device() instead of hardcoding 0. The worker masks the card with CUDA_VISIBLE_DEVICES so 0 was right here, but assuming it would not survive an install that does its device selection differently. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
bitsandbytes NF4 works — verified on the 3090, in the LongCat venv, loading the real UMT5-XXL: 144 Linear4bit modules, is_loaded_in_4bit=True, finite embeddings of the right shape. But it does not cost what v0.2.77 claimed. bitsandbytes replaces Linear layers only. This encoder's 256k-token embedding table and one linear per layer stay at the load dtype, leaving 6.13 GB that cannot shrink. Measured on the card, against ~11.0 GB at bf16: nf4 + double quant 8.00 GB (the bits-per-weight estimate said 2.8) int8 9.76 GB (it said 5.6) So the estimate now comes from those measurements rather than the bit width, because under-reserving by 3x is an OOM at load time. What the card must hold: bf16 encoder, offloaded 13.9 GB 9.7 GB headroom nf4 encoder, resident 22.1 GB 1.4 GB headroom int8 encoder, resident 24.0 GB does not fit bf16 encoder, resident 25.2 GB does not fit Which inverts the recommendation this was built for: an NF4 encoder does fit a 24 GB card, but leaves less room for activations than a 29-frame clip already used, so it would OOM on anything longer. Offload plus the per-request embedding cache from v0.2.76 keeps 9.7 GB free and touches the encoder once, so it stays the better configuration here. The setting remains, correctly sized, for cards where it is the right trade. The model page now states the measured figures too, since "nf4 — ~2.8 GB" would have invited exactly the configuration that OOMs. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Configurable means configurable from the UI, not only from models.json. The model page now carries the four settings added over the last commits — text encoder quantisation, block-sparse attention, the base checkpoint an avatar family borrows tokenizer/text_encoder/vae from, and the context-parallel tile — each with the trade-off in the option text, because "nf4" means nothing without "~2.8 GB, stays resident" beside it. They are in api_model_configure's whitelist too, or the page would offer settings it then dropped on save. And the RunPod image: block-sparse attention is pure Triton, Triton compiles its CUDA driver module the first time it runs, and neither capability base image carries a compiler. A pod would have failed with "Failed to find C compiler" on the first denoising step — exactly how Surya failed in the local image before v0.2.62. gcc and libc6-dev added, and the compiler test now covers this Dockerfile alongside the two local ones, since checking only some of them is how the gap got there in the first place. Only this one pod image installs requirements-longcat.txt, so it is the whole blast radius of the accelerate and bitsandbytes pins; the main venv already carried bitsandbytes for the diffusers 4-bit path, in a separate venv, so there is no conflict. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The DiT has upstream's INT8 path. The UMT5-XXL text encoder does not, and at bf16 it is ~11 GB — 45% of why a 13.6 GB INT8 DiT still overflows a 24 GB card (13.6 + 11.2 + VAE = 25.1 GB before a single activation). Offload worked around it by making the DiT and the encoder take turns; quantising the encoder removes the reason to. Measured against the real checkpoint, what the card must hold: bf16 encoder, offloaded 13.9 GB bf16 encoder, resident 25.2 GB <- the configuration that OOMs int8 encoder, resident 19.5 GB nf4 encoder, resident 16.7 GB So `text_encoder_quant` is a setting: none | int8 | nf4 | fp4, per model or server-wide, validated at startup rather than after a seven-minute load. bitsandbytes is pinned in requirements-longcat.txt and asserted by both the pod image build and the local image's smoke test. A quantised encoder is RESIDENT and never offloaded: bitsandbytes quantises as the weights land on CUDA and is not built to shuttle them back — and at 2.8 GB it does not need to. That is detected from the MODEL, not the config, so a pipeline loaded quantised cannot be offloaded by a config that changed since, and defensively, so a component that cannot be walked is simply not quantised. Inserting these helpers by anchor dropped one of them between @contextlib.contextmanager and the function it was meant to decorate, so _encoder_is_quantised returned a context manager — truthy, making every pipeline look quantised — and _bsa_for quietly stopped being one. Both now have a test asserting their shape, because the symptom was five unrelated failures. Unverified: bitsandbytes has not executed on this GPU. A throwaway container cannot reach the card here (torch reports no CUDA devices under both --gpus all and --runtime=nvidia, though the production container's own CUDA is fine), so the NF4 path is proven only by its unit tests until a service run exercises it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
_run_segments calls generate_t2v once PER SEGMENT, and every one of those calls self.encode_prompt. A one-minute video at 15 fps is 12 segments, so that was 12 identical encodes of the same prompt — and under offload each one also moved the 13.6 GB DiT off the card and the ~11 GB text encoder on and back again. About 35 GB over PCIe per segment, twelve times, for an answer that cannot change. The embeddings are a few MB, so they are now cached on the arguments that produce them, bounded to the last four. Segment 2 onward never touches the encoder: for the rest of a long generation the card holds nothing but the DiT and the VAE. Caching applies without offload too, where it saves the redundant forwards if not the transfers. The wrapper passes its arguments through verbatim instead of naming the first two — naming them baked in an assumption about upstream's signature that the avatar pipeline need not share, and it collided on the first test that used a differently-shaped one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
File "codai/api/video.py", line 4176, in _generate_longcat payload["audio"] = _as_b64(_audio) NameError: name '_as_b64' is not defined _as_b64 was defined INSIDE _generate_h3, MiniMax H3's handler, and _generate_longcat referenced it across that scope. So every LongCat path that carries binary input raised NameError before reaching the worker: the avatar audio track (at2v), the reference image (ai2v and i2v), and continuation frames. Text-to-video is the only path that never touches it, which is why t2v was the only one that ever worked. It cost a 20-minute wait for the GPU to find out, because the NameError fires after the model is loaded and the swap gate has handed over. Now module level, beside the _decode_b64_or_url it wraps. The test walks the AST for a call site in a function that neither defines it nor is nested in one that does, so the same mistake in another backend fails here instead of on a rented GPU. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.2.69 turned the dense backend OFF when enabling BSA, reasoning that _process_attn tests enable_bsa first so anything else would be a silent second opinion. That is true of SELF-attention and wrong about the model: attention.py:248, _process_cross_attn elif self.enable_xformers: ... else: raise RuntimeError("Unsupported attention operations.") Cross-attention has no enable_bsa branch at all. With the dense backend off, every cross-attention call in all 48 blocks raised — the first real BSA run died on step 0 of the first generation. So enable_bsa is now set alongside the chosen dense backend: BSA wins for self-attention on video because it is tested first, and cross-attention keeps a path to run on. Measured on the 3090, 29 frames, warm steady-state per step: dense 480x832 (30 latent rows) 8.13 s/it BSA 448x832 (28 latent rows) 6.34 s/it Adjusting for the 6.7% token difference the two geometries force (BSA needs sides divisible by 64), that is roughly 10-15% per step — well short of the 16x the 0.9375 sparsity suggests, because at this size attention is not the dominant cost; the weight-only INT8 linears are. BSA should matter more at 93 frames and 720p, where attention grows quadratically. It also pays a one-time Triton compile of about four minutes per service. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Turning bsa on produced a bare AssertionError out of the first denoising step: File ".../block_sparse_attention/bsa_interface.py", line 639 assert Tq % tq == 0 and Hq % hq == 0 and Wq % wq == 0 flash_attn_bsa_3d tiles the latent into 3D chunks and needs it to divide evenly on every axis. With the checkpoint's shipped chunk of (4, 4, 4) that means sides divisible by 64 — the VAE's 16-pixel latent cell times 4 — and a latent depth divisible by 4, which puts num_frames at 13, 29, 45, 61, 77, 93 … and NOT at the step*k+1 the 4n+1 generation rule would suggest. Upstream's own default 480x832 fails it (480/16 = 30), which is presumably why the flag ships off: nothing in that AssertionError mentions geometry. bsa_problems() now says which axis is wrong and what to try instead, with the chunk read from the checkpoint's bsa_params rather than assumed. The service checks it per pass and, when the shape cannot be tiled, runs dense attention for that pass and logs why: turning the flag on in a config must not break requests that already work, and a silent fallback would leave someone wondering why nothing got faster. The flag is restored afterwards, so the next request with a tileable shape still gets it. The suggestion arithmetic was wrong first time round — it offered 33, which is exactly the frame count that fails — and the test that checks every suggestion is itself tileable is what caught it. Co-Authored-By:
-
- 06 Oct, 2026 18 commits
-
-
Stefy Lanza (nextime / spora ) authored
LongCat-Video-Avatar-1.5 is a transformer and an audio encoder: base_model/, base_model_int8/, lora/dmd_lora, scheduler/, whisper-large-v3/. It has NO tokenizer, text_encoder or vae. Its README says so in its own frontmatter (`base_model: meituan-longcat/LongCat-Video`) and upstream's instructions download both repos side by side. Validated against the base checkpoint's layout it reported four missing directories that were never part of the repo, and the loader would then have asked from_pretrained for a text_encoder/ that does not exist — and for subfolder="dit", where the avatar transformer is base_model/. So the layout is family-aware now. An avatar checkpoint contributes its scheduler, its transformer (base_model/ at bf16, base_model_int8/ at int8) and its LoRA; tokenizer/text_encoder/vae are borrowed from a base checkpoint, named by `base_model` per model or falling back to the upstream repo id, which resolve_checkpoint() turns into a local snapshot. A base checkpoint that is absent or incomplete is reported AS the base checkpoint, rather than as a tokenizer missing from the avatar repo, which sends you looking in the wrong place. Either transformer alone is enough to validate: base_model/ is 31.7 GB of bf16 that cannot fit a 24 GB card, so it is deliberately not downloaded here. Verified against the real checkpoint: family avatar-1.5, shared components resolving to the base snapshot, 4 INT8 shards with 2241 keys matching the index and 633 quantised Linear layers (the base DiT has 437 — the difference is the audio cross-attention), dmd_lora present. All three validators return []. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The estimator asked a 24 GB card for 74.3 GB and evicted every other model on the box before each generation. It sizes a model by summing its whole HF cache entry, and LongCat's holds three things that are never all resident: the bf16 DiT (51 GB, stored fp32), the INT8 DiT built beside it (13.6 GB), and an fp32 UMT5-XXL text encoder (22 GB). _longcat_resident_gb now counts only the DiT that `variant` selects, scales each component from its OWN stored dtype — one ratio cannot serve both an INT8 DiT that loads as it is stored and an fp32 encoder cast to bf16 — and, when offload is on, takes max(DiT, encoder) + VAE, because they take turns on the card. The dtype comes from the safetensors header, which is one small read per component and never touches the weights. Returning 0.0 leaves an unsizable model to the generic path rather than claiming it needs nothing. Measured against the real checkpoint: 13.9 GB for int8+offload, where the run actually used 13.6 GB. It was 74.3 GB. And accelerate, which upstream does not pin: without it transformers falls back to low_cpu_mem_usage=False and materialises the whole fp32 encoder in host RAM before casting it to bf16. Pinned in requirements-longcat.txt and asserted by BOTH the pod image build and the local image's smoke test, so a rebuild cannot quietly drop it. What accelerate does NOT fix, measured after installing it: the load time. The 5 shards still take ~63 s each, because /AI is ext4 on a spinning disk that reads at 28 MB/s cold — 22 GB of fp32 is ~5 minutes of pure I/O. accelerate removes the host-RAM spike, not the wait. Halving the bytes read (a bf16 copy of the encoder) or moving the checkpoint to NVMe is what would shorten it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two defects between a loading model and a finished clip, both found by running it. A repo id is not a directory. models.json holds what the model page registered — "meituan-longcat/LongCat-Video" — and resolve_model_path handed it straight to the service, which opens config.json, the VAE and the INT8 shards by path. Every load died with "checkpoint directory does not exist" before reading a byte. resolve_checkpoint() maps a repo id onto its newest snapshot in the hub cache, by hand rather than through huggingface_hub so the server's 3.13 venv and the service's 3.10 venv cannot disagree about where the weights are, and so a checkpoint that was never fetched stays an error instead of becoming a silent 74 GB download. And the offload added in v0.2.67 was only half of one. It held the text encoder off the card until encode_prompt and then brought it over WITH the DiT still resident: 13.6 GB of INT8 DiT plus ~11 GB of UMT5-XXL is 24.8 GB on a 24 GB card. It OOM'd on the first prompt of the first generation. Model offload means only the active component is on the card, so they take turns now — the DiT steps off for the encode and comes back for the loop, in a finally, because a failed prompt that stranded the DiT on the host would leave every later generation running at PCIe speed with nothing to say why. Peak is max(), not sum. The placement test passed on the broken version: it asserted the encoder arrived and left, never that the DiT was not there too. It asserts co-residency now, and fails on the old code. Measured end to end on the 3090: 33 frames at 480x832, distilled stage, 16 steps at 9.4 s/it, 13.6 GB resident, HTTP 200 with the mp4. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Attention is the expensive part at video token counts — ~37k tokens for 93 frames at 480x832, and it is O(N^2) in that. The checkpoint already carries bsa_params with sparsity 0.9375, roughly a 16x cut, and upstream's implementation is pure Triton, so none of it needs the flash-attn source build. So it is now a setting: longcat.bsa server-wide, "bsa" per model, --bsa on the service. Off by default because it is an approximation and video shows approximation as temporal flicker, not one soft frame. "auto" stays off when triton is missing; "on" says so instead, because someone who asked for it would otherwise be left wondering why nothing got faster. BSA replaces the dense backend rather than joining it, since Attention tests enable_bsa first. And cp_split_hw, found while wiring this: the DiT's forward opens with `if self.cp_split_hw[0] * self.cp_split_hw[1] > 1:` and has no None guard, but dit/config.json ships "cp_split_hw": null — so a DiT built from the config as published raises TypeError on its first forward. Our only assignment made it worse by storing cp_size, a process count, where a [h, w] pair belongs: an int is not subscriptable either. It defaults to [1, 1] now, derives [1, cp_size] under context parallelism, and takes an explicit tile that is checked against the group size. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
dit/config.json asks for enable_flashattn2, but requirements-longcat.txt ships xformers — flash-attn 2.7.4.post1 is a source build needing ninja and a CUDA toolchain. Upstream's Attention has no fallback: anything unselected reaches `raise RuntimeError("Unsupported attention operations.")`, and the FA2 branch imports flash_attn, which is not in the venv. --attention was only ever written into the environment. Nothing read LONGCAT_ATTENTION, so the checkpoint's config won every time. Now both load paths — bf16 through from_pretrained and INT8 through the config dict — are built with the backend that is actually importable, preferring what was asked for and saying so when it falls back. None installed is an error here, with the reason, rather than 48 blocks into the first step of a load that already cost minutes. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LongCatVideoPipeline's docstring says "This model inherits from [`DiffusionPipeline`]". The class declaration inherits nothing, so it has no enable_model_cpu_offload and no enable_sequential_cpu_offload. The guarded getattr found neither, logged a warning and fell through to pipe.to("cuda") — so every load put all three components on the card and the offload mode in the model config was decorative. That is the 45 GB ask against a 24 GB card. An INT8 DiT does not fix it on its own: 13.6 GB of DiT plus ~11.2 GB for the UMT5-XXL encoder (5.6B, stored fp32, loaded bf16) plus the VAE is ~25 GB before activations. But the encoder runs once, to encode the prompt, and is dead weight for every denoising step after. So offload now keeps it on the host and brings it over only for encode_prompt, set on the instance because all four generate_* methods call self.encode_prompt. The DiT and VAE stay resident, which leaves the loop — the part that has to be fast — on the card. It is moved back in a finally: stranding 11 GB after a failed encode would make the retry the thing that runs out of memory. 'sequential' is served as model offload and says so, rather than implying a deeper mode that upstream has no machinery for. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three left over from earlier fixes in this work: - the frame tests asserted 93 was a valid TRAINING length. SimpleTuner's rule is 8n+1, not generation's 4n+1, and it rounds down silently — which is why frame_problems() rejects 93 and says a good clip can still be a bad sample. Reversed: the valid cases are 8n+1 now, and 93 has its own case asserting the message explains itself. - the trainer stopped dumping "Last output" when it started picking the error lines out of a bounded tail; the test still looked for the old wording. - the vendoring test counted the __pycache__ the container writes into the bind-mounted copy as "vendored non-source files". Generated, not vendored. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Building the INT8 DiT needed ~42 GB because it materialised the bf16 model first; loading it needed ~55 GB for the same reason, plus every shard accumulated into one dict and then copied in. Both took the host down. Both now stream. The quantiser reads tensors one at a time through safe_open and flushes 4 GB shards, enumerating the Linear layers on a meta-device skeleton instead of a real model. _load_quantized_base_dit builds its skeleton on meta too, then assigns each shard into it as it is read. Measured on the 13.6B DiT: 51 GB bf16 -> 13.6 GB INT8, ~4 GB peak to write, and the load is 20 s at 0.6 GB RSS (safetensors mmap) with all 437 quantised Linears present and nothing stranded on meta. A checkpoint that leaves tensors on the meta device is now an error naming them, rather than a model that fails later somewhere unrelated. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Run against the real weights for the first time, it exhausted all 54 GB of RAM and all 4 GB of swap. MemAvailable reached 0 and I killed it to keep the server alive — the production container was two minutes from being the thing the OOM killer chose. It had loaded all six fp32 shards (13m42s) and reached "quantising every Linear to INT8". The peak is not the bf16 model alone: the INT8 copies are built while the originals are still referenced, and save_quantized_state_dict then materialises a full state dict on top. Roughly 2x bf16, not 1x. Three changes: - It now REFUSES when the host cannot take it, with the numbers: "~42 GB needed, 34 GB available". Estimated from the shards on disk (fp32, so bf16 is half) times 1.5, plus 4 GB for the interpreter and torch. min_free_gb overrides it for someone who knows better. A job that can kill the server should not be reachable from a button without this. - The load streams through a meta-device skeleton (low_cpu_mem_usage=True) instead of building a second full copy while loading. - gc.collect() between the phases, and the model is dropped before the save returns — the replaced bf16 weights are ~13 GB of garbage at exactly the moment the save wants to allocate. What this means on THIS host, honestly: 42 GB needed against 54 GB total with the server using ~20 GB. It will now refuse rather than try. Quantising here needs the server stopped, or a lower-peak implementation that never materialises the whole model (quantise shard by shard, write as it goes) — which is the real fix and is not in this commit. 22 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Asked to stop an offloaded render, I killed the service process by hand — and had to, because nothing else could. Every diffusers video path calls task_registry.raise_if_cancelled() between denoise steps and shows up on the Tasks page; LongCat was never registered at all. So the LONGEST-running video job in the system was the only one with no stop button: a 93-frame bf16 render offloading across PCIe can go for a very long time. Its poll loop claimed otherwise. The docstring said it would "carry cancellation and the thermal gate across the process boundary" and the body did neither — it only mirrored progress. A comment describing intent rather than behaviour. It is now registered like the others, reports step progress to the task, and polls for cancellation every 2s. Because the work happens in ANOTHER PROCESS, cancelling cannot raise inside it: the service is stopped instead, which is exactly what eviction already does to it, so that path is well exercised. The request then ends 499 rather than returning a partial video as success, and the task is closed on every exit — cancelled, error or done. I also owe a correction: I told the user there was no cancel for /v1/video/generations. There is, for every other model — I searched for a /v1/video/* endpoint and missed the task registry. The gap was LongCat specifically, which is code I wrote. 7 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.2.62 added gcc to Dockerfile.oci. This install builds with --venv, and build_oci_image.sh uses Dockerfile.oci-venv for that — so the image came out without a compiler and surya would have gone on falling back exactly as before. Caught by checking inside the built image rather than trusting the build's exit code. The test had the same blind spot: it asserted against one Dockerfile. Both can ship, so it now checks both, parametrised by file. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Why a remote host was sending PaddleOCR requests: surya never ran. Every attempt ended as [ocr] engine 'surya' could not serve (vLLM exited (code 1) before becoming ready) … falling back to paddle -> doctr and the root cause, four frames down in the vLLM subprocess output: RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl. Triton JIT-compiles its CUDA driver module at RUNTIME, on first engine start. The image has no cc, gcc or clang — confirmed by looking inside it; the single dpkg match is gcc-*-base, runtime libs, not a compiler. So gcc and libc6-dev are now installed in the shipped stage, with the reason recorded beside them: a compiler in a runtime image looks like an obvious thing to delete. This affects EVERY vLLM engine, not just OCR. What made it invisible: the OCR fallback chain worked. All 203 /v1/ocr requests in the window returned 200 — on paddle, 218 times, after surya failed. Nothing in the HTTP log said anything was wrong, and the only outward sign was a remote host preferring an engine it should not have needed. The RunPod vLLM capability profile has carried gcc from the start. The main image never did. The test checks both so they cannot drift apart again. Needs a rebuild to take effect — a compiler cannot be bind-mounted in like the Python fixes. 4 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three clip generations failed while OCR and other work was running, with: RuntimeError: the venv at /AI/guffcache/longcat_venv was built but still does not report Python 3.10 with torch 2.6 / transformers 4.41 The venv reports exactly that. It holds torch 2.6.0+cu124, transformers 4.41.0, numpy 1.26.4, on Python 3.10.18 — read from its own dist-info, and the probe passes when run by hand. The check imports torch from a 6.2 GB venv. Timed on an IDLE box: 106 seconds, against a 120 second limit. Under the load that necessarily accompanies a generation request it expired, TimeoutExpired was caught by a bare `except Exception: return False`, and the caller turned "I could not finish asking" into "the answer is no" — then rebuilt, re-probed, and failed again. Both probes are now tri-state: True verified, False a real mismatch, None could not be determined. An unverified venv is USED; only a verified-bad one is rebuilt or rejected. The limit is 900s, and a genuine failure now prints what the probe actually said instead of asserting versions nobody checked. Nothing was wrong with the OCR/paddle side, and there were no VRAM, eviction, swap or queue failures in the window — the 1626 apparent "503"s are those digits occurring inside OCR'd Italian legal documents in the request debug dumps, and the 1792 "ERROR"s are mostly the same plus vLLM warnings. The only real exceptions in two hours were this one and the LongCat training chain. 32 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two rules, and I had conflated them. Generation obeys the VAE's 4n+1 — 93, 173 and 333 are whole LongCat segments, which is what the dataset builder and the township templates use. TRAINING is stricter: SimpleTuner's longcat_video family rounds DOWN to 8n+1 (frames % 8 == 1) and says nothing about it. So a job asking for 93 trained at 89, the config recorded 93, and frame_problems() validated the generation rule and passed it. Three places agreeing on a number that was not the one being used. training_frames() now applies the same rounding SimpleTuner does — verified against its adjust_video_frames() for every count from 0 to 400 — and the job is adjusted BEFORE validation, so a legal clip length is never rejected for being legal in the other sense. The adjustment is emitted to the progress stream and the log: num_frames 93 -> 89 (8n+1) frame_problems() now states the training rule and explains why the number looked valid, rather than asserting a constraint the trainer does not use. Found by reading SimpleTuner's model.py while answering a question about whether clips are needed at all — not from a failure. It would have shipped as a quiet discrepancy between the configured and actual frame count. 23 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The CONFIG_PATH fix worked — SimpleTuner read the config, mapped it to CLI args and got as far as argparse, which then said: argparse.ArgumentError: the following arguments are required: --optimizer It has no default for it. The config now sets adamw_bf16, overridable per job. That choice matters here rather than being arbitrary: Adam normally carries two fp32 moments per parameter, which for a 13.6B base is more optimiser state than the card has. base_model_precision quantises the weights; adamw_bf16 keeps the state in bf16. Together they are what make this trainable on one consumer GPU. Also fixed how a SimpleTuner failure is reported. It raised with the last six lines of output, which for a traceback is frame noise — the line naming the cause was above the cut, which is why the first look at this error showed argparse internals and not the message. It now prefers lines that look like a cause (Error/Exception/required:/not found) and falls back to the raw tail. Verified against the installed package: every key the config emits exists in SimpleTuner's field registry, and adamw_bf16 is in optimizer_choices. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The venv built, SimpleTuner 4.9.3 installed, and the trainer RAN — the first time anything past the setup has executed. It then failed on how it was invoked: ValueError: JSON configuration file not found. Paths tried: config/config.json SimpleTuner takes no --config. Its loader picks a configuration BACKEND and asks that backend where to look; the JSON one reads CONFIG_PATH (a file, or a directory holding config.json) and otherwise defaults to ./config/config.json relative to the working directory. The --config we passed was simply ignored, so it fell through to the default and found nothing. Now the backend is named (SIMPLETUNER_CONFIG_BACKEND=json) and the path handed over in CONFIG_PATH, read from the installed package's own json_file.py rather than guessed. The same config is also written to <workdir>/config/config.json, so a loader change that ignored CONFIG_PATH would still find it. Worth recording what this run proved, since all of it was unexercised an hour ago: the vendored source resolves, the training venv rebuilds itself with a 3.12+ interpreter, SimpleTuner installs from PyPI into it, and the job reaches SimpleTuner's own argument parsing. The dataset built from the township clips has not been read yet — that is the next thing in the way. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 3.10 attempt did not just fail — it left /AI/guffcache/longcat_venv-train behind, pyvenv.cfg saying version = 3.10.18. ensure_train_built skips creation when the directory exists, so the next run would have pip-installed into that same 3.10 and failed identically, forever. Fixing the interpreter choice in v0.2.56 was not enough on its own for anyone who had already tried once. An existing training venv is now checked against what SimpleTuner needs (>=3.12,<3.15) and rebuilt when it does not qualify, saying so in the log. The version is READ from pyvenv.cfg rather than obtained by running the interpreter: the Python that built it may be gone, and a failed exec is indistinguishable from a venv that is merely incomplete. Found by looking at the directory before telling the operator to press the button again, which is the only reason it is in this commit rather than in their next log. 24 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The new diagnostics earned their keep immediately. Last run said only "returned non-zero exit status 1"; this one said: ERROR: Could not find a version that satisfies the requirement simpletuner (from versions: none) ERROR: No matching distribution found for simpletuner which reads like the package does not exist. It does. The line above it is the real one: "Ignored the following versions that require a different python version: … Requires-Python >=3.12,<3.14". SimpleTuner 4.9.3 needs >=3.12,<3.15, pip was being run under 3.10, so every release was filtered out and the remainder was none. My error. The 3.10 pin belongs to the INFERENCE venv — torch 2.6+cu124, transformers 4.41, numpy 1.26 — and I applied it to the training venv, which wants the opposite. They are separate venvs precisely because their requirements conflict; sharing an interpreter finder was the bug. ensure_train_built now picks an interpreter SimpleTuner can install into: a configured longcat.train_python, else this process's own (3.13, which satisfies >=3.12,<3.15), else python3.13/3.12 on PATH. So the training venv — unlike the inference one — needs nothing bundled and nothing provisioned. The manual instructions in the non-auto branch said python3.10 too; they now say 3.12+. Also renamed _py310_env to _clean_py_env, since it now guards a 3.13 child as well as the 3.10 one. The venv itself WAS created this time, so the earlier "No module named 'encodings'" did not recur — whether the stripped environment fixed it or it was transient, I still cannot say. 17 tests, including that ensure_train_built never reaches for the 3.10 finder. Co-Authored-By:
-