- 07 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authoredbitsandbytes NF4 works — verified on the 3090, in the LongCat venv, loading the real UMT5-XXL: 144 Linear4bit modules, is_loaded_in_4bit=True, finite embeddings of the right shape. But it does not cost what v0.2.77 claimed. bitsandbytes replaces Linear layers only. This encoder's 256k-token embedding table and one linear per layer stay at the load dtype, leaving 6.13 GB that cannot shrink. Measured on the card, against ~11.0 GB at bf16: nf4 + double quant 8.00 GB (the bits-per-weight estimate said 2.8) int8 9.76 GB (it said 5.6) So the estimate now comes from those measurements rather than the bit width, because under-reserving by 3x is an OOM at load time. What the card must hold: bf16 encoder, offloaded 13.9 GB 9.7 GB headroom nf4 encoder, resident 22.1 GB 1.4 GB headroom int8 encoder, resident 24.0 GB does not fit bf16 encoder, resident 25.2 GB does not fit Which inverts the recommendation this was built for: an NF4 encoder does fit a 24 GB card, but leaves less room for activations than a 29-frame clip already used, so it would OOM on anything longer. Offload plus the per-request embedding cache from v0.2.76 keeps 9.7 GB free and touches the encoder once, so it stays the better configuration here. The setting remains, correctly sized, for cards where it is the right trade. The model page now states the measured figures too, since "nf4 — ~2.8 GB" would have invited exactly the configuration that OOMs. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0185zEBiy3KWB37xNGSrxL7w
-
Stefy Lanza (nextime / spora ) authored
Configurable means configurable from the UI, not only from models.json. The model page now carries the four settings added over the last commits — text encoder quantisation, block-sparse attention, the base checkpoint an avatar family borrows tokenizer/text_encoder/vae from, and the context-parallel tile — each with the trade-off in the option text, because "nf4" means nothing without "~2.8 GB, stays resident" beside it. They are in api_model_configure's whitelist too, or the page would offer settings it then dropped on save. And the RunPod image: block-sparse attention is pure Triton, Triton compiles its CUDA driver module the first time it runs, and neither capability base image carries a compiler. A pod would have failed with "Failed to find C compiler" on the first denoising step — exactly how Surya failed in the local image before v0.2.62. gcc and libc6-dev added, and the compiler test now covers this Dockerfile alongside the two local ones, since checking only some of them is how the gap got there in the first place. Only this one pod image installs requirements-longcat.txt, so it is the whole blast radius of the accelerate and bitsandbytes pins; the main venv already carried bitsandbytes for the diffusers 4-bit path, in a separate venv, so there is no conflict. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The DiT has upstream's INT8 path. The UMT5-XXL text encoder does not, and at bf16 it is ~11 GB — 45% of why a 13.6 GB INT8 DiT still overflows a 24 GB card (13.6 + 11.2 + VAE = 25.1 GB before a single activation). Offload worked around it by making the DiT and the encoder take turns; quantising the encoder removes the reason to. Measured against the real checkpoint, what the card must hold: bf16 encoder, offloaded 13.9 GB bf16 encoder, resident 25.2 GB <- the configuration that OOMs int8 encoder, resident 19.5 GB nf4 encoder, resident 16.7 GB So `text_encoder_quant` is a setting: none | int8 | nf4 | fp4, per model or server-wide, validated at startup rather than after a seven-minute load. bitsandbytes is pinned in requirements-longcat.txt and asserted by both the pod image build and the local image's smoke test. A quantised encoder is RESIDENT and never offloaded: bitsandbytes quantises as the weights land on CUDA and is not built to shuttle them back — and at 2.8 GB it does not need to. That is detected from the MODEL, not the config, so a pipeline loaded quantised cannot be offloaded by a config that changed since, and defensively, so a component that cannot be walked is simply not quantised. Inserting these helpers by anchor dropped one of them between @contextlib.contextmanager and the function it was meant to decorate, so _encoder_is_quantised returned a context manager — truthy, making every pipeline look quantised — and _bsa_for quietly stopped being one. Both now have a test asserting their shape, because the symptom was five unrelated failures. Unverified: bitsandbytes has not executed on this GPU. A throwaway container cannot reach the card here (torch reports no CUDA devices under both --gpus all and --runtime=nvidia, though the production container's own CUDA is fine), so the NF4 path is proven only by its unit tests until a service run exercises it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
_run_segments calls generate_t2v once PER SEGMENT, and every one of those calls self.encode_prompt. A one-minute video at 15 fps is 12 segments, so that was 12 identical encodes of the same prompt — and under offload each one also moved the 13.6 GB DiT off the card and the ~11 GB text encoder on and back again. About 35 GB over PCIe per segment, twelve times, for an answer that cannot change. The embeddings are a few MB, so they are now cached on the arguments that produce them, bounded to the last four. Segment 2 onward never touches the encoder: for the rest of a long generation the card holds nothing but the DiT and the VAE. Caching applies without offload too, where it saves the redundant forwards if not the transfers. The wrapper passes its arguments through verbatim instead of naming the first two — naming them baked in an assumption about upstream's signature that the avatar pipeline need not share, and it collided on the first test that used a differently-shaped one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
File "codai/api/video.py", line 4176, in _generate_longcat payload["audio"] = _as_b64(_audio) NameError: name '_as_b64' is not defined _as_b64 was defined INSIDE _generate_h3, MiniMax H3's handler, and _generate_longcat referenced it across that scope. So every LongCat path that carries binary input raised NameError before reaching the worker: the avatar audio track (at2v), the reference image (ai2v and i2v), and continuation frames. Text-to-video is the only path that never touches it, which is why t2v was the only one that ever worked. It cost a 20-minute wait for the GPU to find out, because the NameError fires after the model is loaded and the swap gate has handed over. Now module level, beside the _decode_b64_or_url it wraps. The test walks the AST for a call site in a function that neither defines it nor is nested in one that does, so the same mistake in another backend fails here instead of on a rented GPU. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.2.69 turned the dense backend OFF when enabling BSA, reasoning that _process_attn tests enable_bsa first so anything else would be a silent second opinion. That is true of SELF-attention and wrong about the model: attention.py:248, _process_cross_attn elif self.enable_xformers: ... else: raise RuntimeError("Unsupported attention operations.") Cross-attention has no enable_bsa branch at all. With the dense backend off, every cross-attention call in all 48 blocks raised — the first real BSA run died on step 0 of the first generation. So enable_bsa is now set alongside the chosen dense backend: BSA wins for self-attention on video because it is tested first, and cross-attention keeps a path to run on. Measured on the 3090, 29 frames, warm steady-state per step: dense 480x832 (30 latent rows) 8.13 s/it BSA 448x832 (28 latent rows) 6.34 s/it Adjusting for the 6.7% token difference the two geometries force (BSA needs sides divisible by 64), that is roughly 10-15% per step — well short of the 16x the 0.9375 sparsity suggests, because at this size attention is not the dominant cost; the weight-only INT8 linears are. BSA should matter more at 93 frames and 720p, where attention grows quadratically. It also pays a one-time Triton compile of about four minutes per service. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Turning bsa on produced a bare AssertionError out of the first denoising step: File ".../block_sparse_attention/bsa_interface.py", line 639 assert Tq % tq == 0 and Hq % hq == 0 and Wq % wq == 0 flash_attn_bsa_3d tiles the latent into 3D chunks and needs it to divide evenly on every axis. With the checkpoint's shipped chunk of (4, 4, 4) that means sides divisible by 64 — the VAE's 16-pixel latent cell times 4 — and a latent depth divisible by 4, which puts num_frames at 13, 29, 45, 61, 77, 93 … and NOT at the step*k+1 the 4n+1 generation rule would suggest. Upstream's own default 480x832 fails it (480/16 = 30), which is presumably why the flag ships off: nothing in that AssertionError mentions geometry. bsa_problems() now says which axis is wrong and what to try instead, with the chunk read from the checkpoint's bsa_params rather than assumed. The service checks it per pass and, when the shape cannot be tiled, runs dense attention for that pass and logs why: turning the flag on in a config must not break requests that already work, and a silent fallback would leave someone wondering why nothing got faster. The flag is restored afterwards, so the next request with a tileable shape still gets it. The suggestion arithmetic was wrong first time round — it offered 33, which is exactly the frame count that fails — and the test that checks every suggestion is itself tileable is what caught it. Co-Authored-By:
-
- 06 Oct, 2026 29 commits
-
-
Stefy Lanza (nextime / spora ) authored
LongCat-Video-Avatar-1.5 is a transformer and an audio encoder: base_model/, base_model_int8/, lora/dmd_lora, scheduler/, whisper-large-v3/. It has NO tokenizer, text_encoder or vae. Its README says so in its own frontmatter (`base_model: meituan-longcat/LongCat-Video`) and upstream's instructions download both repos side by side. Validated against the base checkpoint's layout it reported four missing directories that were never part of the repo, and the loader would then have asked from_pretrained for a text_encoder/ that does not exist — and for subfolder="dit", where the avatar transformer is base_model/. So the layout is family-aware now. An avatar checkpoint contributes its scheduler, its transformer (base_model/ at bf16, base_model_int8/ at int8) and its LoRA; tokenizer/text_encoder/vae are borrowed from a base checkpoint, named by `base_model` per model or falling back to the upstream repo id, which resolve_checkpoint() turns into a local snapshot. A base checkpoint that is absent or incomplete is reported AS the base checkpoint, rather than as a tokenizer missing from the avatar repo, which sends you looking in the wrong place. Either transformer alone is enough to validate: base_model/ is 31.7 GB of bf16 that cannot fit a 24 GB card, so it is deliberately not downloaded here. Verified against the real checkpoint: family avatar-1.5, shared components resolving to the base snapshot, 4 INT8 shards with 2241 keys matching the index and 633 quantised Linear layers (the base DiT has 437 — the difference is the audio cross-attention), dmd_lora present. All three validators return []. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The estimator asked a 24 GB card for 74.3 GB and evicted every other model on the box before each generation. It sizes a model by summing its whole HF cache entry, and LongCat's holds three things that are never all resident: the bf16 DiT (51 GB, stored fp32), the INT8 DiT built beside it (13.6 GB), and an fp32 UMT5-XXL text encoder (22 GB). _longcat_resident_gb now counts only the DiT that `variant` selects, scales each component from its OWN stored dtype — one ratio cannot serve both an INT8 DiT that loads as it is stored and an fp32 encoder cast to bf16 — and, when offload is on, takes max(DiT, encoder) + VAE, because they take turns on the card. The dtype comes from the safetensors header, which is one small read per component and never touches the weights. Returning 0.0 leaves an unsizable model to the generic path rather than claiming it needs nothing. Measured against the real checkpoint: 13.9 GB for int8+offload, where the run actually used 13.6 GB. It was 74.3 GB. And accelerate, which upstream does not pin: without it transformers falls back to low_cpu_mem_usage=False and materialises the whole fp32 encoder in host RAM before casting it to bf16. Pinned in requirements-longcat.txt and asserted by BOTH the pod image build and the local image's smoke test, so a rebuild cannot quietly drop it. What accelerate does NOT fix, measured after installing it: the load time. The 5 shards still take ~63 s each, because /AI is ext4 on a spinning disk that reads at 28 MB/s cold — 22 GB of fp32 is ~5 minutes of pure I/O. accelerate removes the host-RAM spike, not the wait. Halving the bytes read (a bf16 copy of the encoder) or moving the checkpoint to NVMe is what would shorten it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two defects between a loading model and a finished clip, both found by running it. A repo id is not a directory. models.json holds what the model page registered — "meituan-longcat/LongCat-Video" — and resolve_model_path handed it straight to the service, which opens config.json, the VAE and the INT8 shards by path. Every load died with "checkpoint directory does not exist" before reading a byte. resolve_checkpoint() maps a repo id onto its newest snapshot in the hub cache, by hand rather than through huggingface_hub so the server's 3.13 venv and the service's 3.10 venv cannot disagree about where the weights are, and so a checkpoint that was never fetched stays an error instead of becoming a silent 74 GB download. And the offload added in v0.2.67 was only half of one. It held the text encoder off the card until encode_prompt and then brought it over WITH the DiT still resident: 13.6 GB of INT8 DiT plus ~11 GB of UMT5-XXL is 24.8 GB on a 24 GB card. It OOM'd on the first prompt of the first generation. Model offload means only the active component is on the card, so they take turns now — the DiT steps off for the encode and comes back for the loop, in a finally, because a failed prompt that stranded the DiT on the host would leave every later generation running at PCIe speed with nothing to say why. Peak is max(), not sum. The placement test passed on the broken version: it asserted the encoder arrived and left, never that the DiT was not there too. It asserts co-residency now, and fails on the old code. Measured end to end on the 3090: 33 frames at 480x832, distilled stage, 16 steps at 9.4 s/it, 13.6 GB resident, HTTP 200 with the mp4. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Attention is the expensive part at video token counts — ~37k tokens for 93 frames at 480x832, and it is O(N^2) in that. The checkpoint already carries bsa_params with sparsity 0.9375, roughly a 16x cut, and upstream's implementation is pure Triton, so none of it needs the flash-attn source build. So it is now a setting: longcat.bsa server-wide, "bsa" per model, --bsa on the service. Off by default because it is an approximation and video shows approximation as temporal flicker, not one soft frame. "auto" stays off when triton is missing; "on" says so instead, because someone who asked for it would otherwise be left wondering why nothing got faster. BSA replaces the dense backend rather than joining it, since Attention tests enable_bsa first. And cp_split_hw, found while wiring this: the DiT's forward opens with `if self.cp_split_hw[0] * self.cp_split_hw[1] > 1:` and has no None guard, but dit/config.json ships "cp_split_hw": null — so a DiT built from the config as published raises TypeError on its first forward. Our only assignment made it worse by storing cp_size, a process count, where a [h, w] pair belongs: an int is not subscriptable either. It defaults to [1, 1] now, derives [1, cp_size] under context parallelism, and takes an explicit tile that is checked against the group size. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
dit/config.json asks for enable_flashattn2, but requirements-longcat.txt ships xformers — flash-attn 2.7.4.post1 is a source build needing ninja and a CUDA toolchain. Upstream's Attention has no fallback: anything unselected reaches `raise RuntimeError("Unsupported attention operations.")`, and the FA2 branch imports flash_attn, which is not in the venv. --attention was only ever written into the environment. Nothing read LONGCAT_ATTENTION, so the checkpoint's config won every time. Now both load paths — bf16 through from_pretrained and INT8 through the config dict — are built with the backend that is actually importable, preferring what was asked for and saying so when it falls back. None installed is an error here, with the reason, rather than 48 blocks into the first step of a load that already cost minutes. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LongCatVideoPipeline's docstring says "This model inherits from [`DiffusionPipeline`]". The class declaration inherits nothing, so it has no enable_model_cpu_offload and no enable_sequential_cpu_offload. The guarded getattr found neither, logged a warning and fell through to pipe.to("cuda") — so every load put all three components on the card and the offload mode in the model config was decorative. That is the 45 GB ask against a 24 GB card. An INT8 DiT does not fix it on its own: 13.6 GB of DiT plus ~11.2 GB for the UMT5-XXL encoder (5.6B, stored fp32, loaded bf16) plus the VAE is ~25 GB before activations. But the encoder runs once, to encode the prompt, and is dead weight for every denoising step after. So offload now keeps it on the host and brings it over only for encode_prompt, set on the instance because all four generate_* methods call self.encode_prompt. The DiT and VAE stay resident, which leaves the loop — the part that has to be fast — on the card. It is moved back in a finally: stranding 11 GB after a failed encode would make the retry the thing that runs out of memory. 'sequential' is served as model offload and says so, rather than implying a deeper mode that upstream has no machinery for. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three left over from earlier fixes in this work: - the frame tests asserted 93 was a valid TRAINING length. SimpleTuner's rule is 8n+1, not generation's 4n+1, and it rounds down silently — which is why frame_problems() rejects 93 and says a good clip can still be a bad sample. Reversed: the valid cases are 8n+1 now, and 93 has its own case asserting the message explains itself. - the trainer stopped dumping "Last output" when it started picking the error lines out of a bounded tail; the test still looked for the old wording. - the vendoring test counted the __pycache__ the container writes into the bind-mounted copy as "vendored non-source files". Generated, not vendored. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Building the INT8 DiT needed ~42 GB because it materialised the bf16 model first; loading it needed ~55 GB for the same reason, plus every shard accumulated into one dict and then copied in. Both took the host down. Both now stream. The quantiser reads tensors one at a time through safe_open and flushes 4 GB shards, enumerating the Linear layers on a meta-device skeleton instead of a real model. _load_quantized_base_dit builds its skeleton on meta too, then assigns each shard into it as it is read. Measured on the 13.6B DiT: 51 GB bf16 -> 13.6 GB INT8, ~4 GB peak to write, and the load is 20 s at 0.6 GB RSS (safetensors mmap) with all 437 quantised Linears present and nothing stranded on meta. A checkpoint that leaves tensors on the meta device is now an error naming them, rather than a model that fails later somewhere unrelated. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Run against the real weights for the first time, it exhausted all 54 GB of RAM and all 4 GB of swap. MemAvailable reached 0 and I killed it to keep the server alive — the production container was two minutes from being the thing the OOM killer chose. It had loaded all six fp32 shards (13m42s) and reached "quantising every Linear to INT8". The peak is not the bf16 model alone: the INT8 copies are built while the originals are still referenced, and save_quantized_state_dict then materialises a full state dict on top. Roughly 2x bf16, not 1x. Three changes: - It now REFUSES when the host cannot take it, with the numbers: "~42 GB needed, 34 GB available". Estimated from the shards on disk (fp32, so bf16 is half) times 1.5, plus 4 GB for the interpreter and torch. min_free_gb overrides it for someone who knows better. A job that can kill the server should not be reachable from a button without this. - The load streams through a meta-device skeleton (low_cpu_mem_usage=True) instead of building a second full copy while loading. - gc.collect() between the phases, and the model is dropped before the save returns — the replaced bf16 weights are ~13 GB of garbage at exactly the moment the save wants to allocate. What this means on THIS host, honestly: 42 GB needed against 54 GB total with the server using ~20 GB. It will now refuse rather than try. Quantising here needs the server stopped, or a lower-peak implementation that never materialises the whole model (quantise shard by shard, write as it goes) — which is the real fix and is not in this commit. 22 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Asked to stop an offloaded render, I killed the service process by hand — and had to, because nothing else could. Every diffusers video path calls task_registry.raise_if_cancelled() between denoise steps and shows up on the Tasks page; LongCat was never registered at all. So the LONGEST-running video job in the system was the only one with no stop button: a 93-frame bf16 render offloading across PCIe can go for a very long time. Its poll loop claimed otherwise. The docstring said it would "carry cancellation and the thermal gate across the process boundary" and the body did neither — it only mirrored progress. A comment describing intent rather than behaviour. It is now registered like the others, reports step progress to the task, and polls for cancellation every 2s. Because the work happens in ANOTHER PROCESS, cancelling cannot raise inside it: the service is stopped instead, which is exactly what eviction already does to it, so that path is well exercised. The request then ends 499 rather than returning a partial video as success, and the task is closed on every exit — cancelled, error or done. I also owe a correction: I told the user there was no cancel for /v1/video/generations. There is, for every other model — I searched for a /v1/video/* endpoint and missed the task registry. The gap was LongCat specifically, which is code I wrote. 7 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.2.62 added gcc to Dockerfile.oci. This install builds with --venv, and build_oci_image.sh uses Dockerfile.oci-venv for that — so the image came out without a compiler and surya would have gone on falling back exactly as before. Caught by checking inside the built image rather than trusting the build's exit code. The test had the same blind spot: it asserted against one Dockerfile. Both can ship, so it now checks both, parametrised by file. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Why a remote host was sending PaddleOCR requests: surya never ran. Every attempt ended as [ocr] engine 'surya' could not serve (vLLM exited (code 1) before becoming ready) … falling back to paddle -> doctr and the root cause, four frames down in the vLLM subprocess output: RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl. Triton JIT-compiles its CUDA driver module at RUNTIME, on first engine start. The image has no cc, gcc or clang — confirmed by looking inside it; the single dpkg match is gcc-*-base, runtime libs, not a compiler. So gcc and libc6-dev are now installed in the shipped stage, with the reason recorded beside them: a compiler in a runtime image looks like an obvious thing to delete. This affects EVERY vLLM engine, not just OCR. What made it invisible: the OCR fallback chain worked. All 203 /v1/ocr requests in the window returned 200 — on paddle, 218 times, after surya failed. Nothing in the HTTP log said anything was wrong, and the only outward sign was a remote host preferring an engine it should not have needed. The RunPod vLLM capability profile has carried gcc from the start. The main image never did. The test checks both so they cannot drift apart again. Needs a rebuild to take effect — a compiler cannot be bind-mounted in like the Python fixes. 4 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three clip generations failed while OCR and other work was running, with: RuntimeError: the venv at /AI/guffcache/longcat_venv was built but still does not report Python 3.10 with torch 2.6 / transformers 4.41 The venv reports exactly that. It holds torch 2.6.0+cu124, transformers 4.41.0, numpy 1.26.4, on Python 3.10.18 — read from its own dist-info, and the probe passes when run by hand. The check imports torch from a 6.2 GB venv. Timed on an IDLE box: 106 seconds, against a 120 second limit. Under the load that necessarily accompanies a generation request it expired, TimeoutExpired was caught by a bare `except Exception: return False`, and the caller turned "I could not finish asking" into "the answer is no" — then rebuilt, re-probed, and failed again. Both probes are now tri-state: True verified, False a real mismatch, None could not be determined. An unverified venv is USED; only a verified-bad one is rebuilt or rejected. The limit is 900s, and a genuine failure now prints what the probe actually said instead of asserting versions nobody checked. Nothing was wrong with the OCR/paddle side, and there were no VRAM, eviction, swap or queue failures in the window — the 1626 apparent "503"s are those digits occurring inside OCR'd Italian legal documents in the request debug dumps, and the 1792 "ERROR"s are mostly the same plus vLLM warnings. The only real exceptions in two hours were this one and the LongCat training chain. 32 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two rules, and I had conflated them. Generation obeys the VAE's 4n+1 — 93, 173 and 333 are whole LongCat segments, which is what the dataset builder and the township templates use. TRAINING is stricter: SimpleTuner's longcat_video family rounds DOWN to 8n+1 (frames % 8 == 1) and says nothing about it. So a job asking for 93 trained at 89, the config recorded 93, and frame_problems() validated the generation rule and passed it. Three places agreeing on a number that was not the one being used. training_frames() now applies the same rounding SimpleTuner does — verified against its adjust_video_frames() for every count from 0 to 400 — and the job is adjusted BEFORE validation, so a legal clip length is never rejected for being legal in the other sense. The adjustment is emitted to the progress stream and the log: num_frames 93 -> 89 (8n+1) frame_problems() now states the training rule and explains why the number looked valid, rather than asserting a constraint the trainer does not use. Found by reading SimpleTuner's model.py while answering a question about whether clips are needed at all — not from a failure. It would have shipped as a quiet discrepancy between the configured and actual frame count. 23 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The CONFIG_PATH fix worked — SimpleTuner read the config, mapped it to CLI args and got as far as argparse, which then said: argparse.ArgumentError: the following arguments are required: --optimizer It has no default for it. The config now sets adamw_bf16, overridable per job. That choice matters here rather than being arbitrary: Adam normally carries two fp32 moments per parameter, which for a 13.6B base is more optimiser state than the card has. base_model_precision quantises the weights; adamw_bf16 keeps the state in bf16. Together they are what make this trainable on one consumer GPU. Also fixed how a SimpleTuner failure is reported. It raised with the last six lines of output, which for a traceback is frame noise — the line naming the cause was above the cut, which is why the first look at this error showed argparse internals and not the message. It now prefers lines that look like a cause (Error/Exception/required:/not found) and falls back to the raw tail. Verified against the installed package: every key the config emits exists in SimpleTuner's field registry, and adamw_bf16 is in optimizer_choices. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The venv built, SimpleTuner 4.9.3 installed, and the trainer RAN — the first time anything past the setup has executed. It then failed on how it was invoked: ValueError: JSON configuration file not found. Paths tried: config/config.json SimpleTuner takes no --config. Its loader picks a configuration BACKEND and asks that backend where to look; the JSON one reads CONFIG_PATH (a file, or a directory holding config.json) and otherwise defaults to ./config/config.json relative to the working directory. The --config we passed was simply ignored, so it fell through to the default and found nothing. Now the backend is named (SIMPLETUNER_CONFIG_BACKEND=json) and the path handed over in CONFIG_PATH, read from the installed package's own json_file.py rather than guessed. The same config is also written to <workdir>/config/config.json, so a loader change that ignored CONFIG_PATH would still find it. Worth recording what this run proved, since all of it was unexercised an hour ago: the vendored source resolves, the training venv rebuilds itself with a 3.12+ interpreter, SimpleTuner installs from PyPI into it, and the job reaches SimpleTuner's own argument parsing. The dataset built from the township clips has not been read yet — that is the next thing in the way. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 3.10 attempt did not just fail — it left /AI/guffcache/longcat_venv-train behind, pyvenv.cfg saying version = 3.10.18. ensure_train_built skips creation when the directory exists, so the next run would have pip-installed into that same 3.10 and failed identically, forever. Fixing the interpreter choice in v0.2.56 was not enough on its own for anyone who had already tried once. An existing training venv is now checked against what SimpleTuner needs (>=3.12,<3.15) and rebuilt when it does not qualify, saying so in the log. The version is READ from pyvenv.cfg rather than obtained by running the interpreter: the Python that built it may be gone, and a failed exec is indistinguishable from a venv that is merely incomplete. Found by looking at the directory before telling the operator to press the button again, which is the only reason it is in this commit rather than in their next log. 24 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The new diagnostics earned their keep immediately. Last run said only "returned non-zero exit status 1"; this one said: ERROR: Could not find a version that satisfies the requirement simpletuner (from versions: none) ERROR: No matching distribution found for simpletuner which reads like the package does not exist. It does. The line above it is the real one: "Ignored the following versions that require a different python version: … Requires-Python >=3.12,<3.14". SimpleTuner 4.9.3 needs >=3.12,<3.15, pip was being run under 3.10, so every release was filtered out and the remainder was none. My error. The 3.10 pin belongs to the INFERENCE venv — torch 2.6+cu124, transformers 4.41, numpy 1.26 — and I applied it to the training venv, which wants the opposite. They are separate venvs precisely because their requirements conflict; sharing an interpreter finder was the bug. ensure_train_built now picks an interpreter SimpleTuner can install into: a configured longcat.train_python, else this process's own (3.13, which satisfies >=3.12,<3.15), else python3.13/3.12 on PATH. So the training venv — unlike the inference one — needs nothing bundled and nothing provisioned. The manual instructions in the non-auto branch said python3.10 too; they now say 3.12+. Also renamed _py310_env to _clean_py_env, since it now guards a 3.13 child as well as the 3.10 one. The venv itself WAS created this time, so the earlier "No module named 'encodings'" did not recur — whether the stripped environment fixed it or it was transient, I still cannot say. 17 tests, including that ensure_train_built never reaches for the 3.10 finder. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
training failed: Command '[…py310…, '-m', 'venv', '/AI/guffcache/longcat_venv-train']' returned non-zero exit status 1. An exit status is not a diagnosis. The actual reason was in the log only because the child's stderr happened to land there: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding ModuleNotFoundError: No module named 'encodings' Every step of the venv build went through subprocess.run(check=True), which reports a number and discards what the process said. They now go through a runner that raises with the child's output, truncated at both ends if it is long. The next failure here names itself instead of needing a log archaeologist. The environment is also stripped for anything that spawns the 3.10 interpreter — PYTHONHOME, PYTHONPATH, VIRTUAL_ENV, PYTHONEXECUTABLE and friends — because that is exactly the class of variable that produces the error above, and the failure is fatal and silent in the parent. HF cache paths, CUDA_VISIBLE_DEVICES and everything else still reach the child. This covers the venv build, the pip installs, the service launch and the train/quantise workers; the service used dict(os.environ) directly before. Worth being straight about: I could NOT reproduce the failure. The same command from a throwaway container on the same image, same user, same mount, exits 0 and writes a complete venv, and the engine process has no PYTHONHOME, PYTHONPATH or VIRTUAL_ENV in its environment. So this is not a confirmed fix for that specific failure — it is the removal of the reason I cannot tell you what went wrong, plus hardening against the one cause the error text points at. The retry will either work or finally say why. 12 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Caught by running the validators against the real checkpoint the moment it finished downloading: variant_problems('int8') -> ["INT8 weights exist only for avatar-1.5; this checkpoint reads as the base model"] That guard predates the quantise action and contradicts it: the model page can now BUILD base_model_int8/ for any checkpoint, so the button would have produced the weights and this would have refused to load them. The feature shipped one commit ago with a gate that made it useless. What decides it now is whether the weights are present, not which family wrote them — upstream still ships INT8 only for avatar-1.5, and that is still accepted without a local build, but a base checkpoint with base_model_int8/ is served too. Without them the refusal names the remedy ("use Build INT8 weights on the model page first") instead of stating a family rule the operator cannot act on. The checkpoint directory is passed in for that, since the gate could not otherwise see what had been built. Checked against the real 78 GB download: checkpoint_problems is clean, the vendored source passes source_problems, bf16 is ungated, int8 is refused before a build and accepted after, and upstream's avatar-1.5 path is unchanged. 16 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 13.6B bf16 DiT is ~27 GB: it does not fit a 24 GB card and streams across PCIe every step. The INT8 copy fits. So the model page gets a "Build INT8 weights" button beside the existing "INT8 DiT" switch — the switch said which weights to USE, and nothing could produce them. Deliberately an action, never a default. It is lossy and slow, the bf16 weights are never touched, and un-ticking the switch goes straight back to them. A test asserts nothing turns use_int8 on by itself. It runs in LongCat's isolated py3.10 venv (upstream's helpers, torch 2.6), so it is a worker script like training, not in-process — tools/longcat_quantize.py, driven by longcat_worker.quantize_int8() on the same job/JSON-lines protocol, so the existing progress plumbing needed no special case. Quantisation happens on the CPU on purpose: the point is to produce a model that fits the card, so getting there must not need the card. One thing upstream's code could not do for us. load_quantized_dit() hardcodes the AVATAR transformer class — correct for its avatar-only INT8 build, wrong for a quantised BASE checkpoint. The avatar families keep using it; the base family gets _load_quantized_base_dit(), the same procedure (build from the saved config, swap every non-skipped Linear for QuantizedLinear, load the sharded state dict) against LongCatVideoTransformer3DModel. Upstream's QuantizedLinear and skip patterns are reused rather than restated, so the layout cannot drift from what wrote it. The existing /admin/api/model-quantize routes a longcat-backed model here instead of GPTQ/AWQ, which are for transformers this is not; quantize-status reports the job the same way, so the page polls one endpoint. A model whose weights are not downloaded yet is refused rather than silently started. Not yet run against real weights — the 83 GB base checkpoint is still downloading. 43 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`longcat_video` — the pipeline, DiT, VAE, scheduler and INT8 quantisation helpers — lives inside the upstream repo and is not on PyPI, so resolve_source_dir looked for a checkout at ~/.coderai/LongCat-Video or a path baked at build time. Neither exists in a published image, which means anyone pulling it had to clone from GitHub before LongCat would run at all: a third-party network dependency in the middle of someone else's install, on whatever `main` happened to be that day. The 26 source files are now vendored under third_party/, MIT, pinned to commit 6b3f4b85. 496 KB, no weights, no wheels, nothing compiled. The resolver prefers them over every fallback, and an explicit longcat_source still wins for anyone running their own checkout. They ship automatically: the image does COPY . /opt/coderai/app, and third_party is not in .dockerignore — asserted by a test, since that is the sort of thing a later ignore rule breaks silently. The ISOLATED VENV is unaffected and still builds on first use from requirements-longcat.txt. Only the source is vendored; torch 2.6+cu124 and the rest are installed, not carried. While confirming the fp8 question: upstream's quantization.py implements INT8 only — QuantizedLinear holds weight_int8 + weight_scale, and there is no fp8 code anywhere in the package. The fp8 checkpoints on HF carry no loader either, and szwagros/LongCat-Video-Avatar-1.5-fp8's own card is a verbatim copy of upstream's and documents `--use_int8`, never fp8. So a VRAM-resident DiT goes through INT8, not fp8. 88 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
✗ training failed: _lora_dir() missing 1 required positional argument: 'name' Progress, in that everything before it worked: the Train button built the dataset, the request carried dataset_config, the server accepted it and dispatched to the LongCat path. It then died on the first line of _train_externally that does real work — proof that function had never once executed. Two bugs in one line. _lora_dir(name) already resolves the per-LoRA directory AND routes a non-main DDP rank to scratch; the call treated it as the parent and joined the name on again, so even with the argument supplied it would have written <loras>/<name>/<name> and bypassed the DDP handling. The test calls the real _train_externally with only the worker stubbed — no GPU, no checkpoint, no venv — and checks it reaches the worker, that output_dir equals _lora_dir(name) with the name not doubled, that the job carries steps/rank/seed/ num_frames/dataset_config, that both refusal paths still refuse, and that the worker's **kwargs progress callback matches the receiver's signature (a mismatch there would only show up mid-training). Verified to FAIL on the real defect: restoring the old line turns 4 of the 6 red with exactly the TypeError above. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The log shows the 401 is gone — the latest attempt was accepted (200 OK, job-551586975a13) and reached the trainer. It then failed on the SAME error as before the key was fixed: RuntimeError: longcat LoRA training needs `dataset_config` Because there are two training paths and only one of them was wired. Step 4b (the batch stage) builds a SimpleTuner dataset from the rendered clips. The per-profile Train button on the Characters page — which is what was being pressed — assembled its kwargs independently and never passed dataset_config, so the server refused the job exactly as it is supposed to. Both paths now build the same dataset through the same function. When it cannot (no clips rendered yet, clips too short, too few), the button says so and names the remedy instead of letting the server answer with a stack trace. Surveyed against the real township output before claiming it works: 151 clips and outcomes carry prompts, and at the default 93-frame / 8-clip threshold all 6 fighters and all 4 environments qualify — khumalo has 19 usable clips, dlamini 11. Still true, and unchanged by this: no LongCat training has ever RUN. This gets the job past the server's gate; what SimpleTuner then does with it is unproven. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 401 on video LoRA training was not a missing key. It was a DELETED one. Loading a template persisted by writing the template over township_config.json: _j.dump(tcfg, f, indent=2, sort_keys=True) A template is a PARTIAL — the LongCat starter carries 10 options, the config carries 60. So loading it turned the operator's config into those 10 keys and destroyed everything else, api_key and base_url included. Every subsequent call to CoderAI then failed with a bare 401 that said nothing about why. Found by comparing the live file against its own backups: township_config.json was 238 bytes with exactly the LongCat starter's 10 keys, timestamped during this session; every backup beside it was ~1700 bytes with 60 keys and a 75-character api_key. The token itself was never gone — it is still 'fighters' in auth.json. It now reads the existing config, updates it with the template, and writes the result. A template's values win; nothing it is silent about is touched. With no readable config yet, the template is the config, as before. This is mine, from 310ffc79 ("loading a template persists, not just applies") in v0.2.39. The persistence was right; replacing rather than merging was not. The operator's config has been restored in place from the newest full backup with the LongCat values kept on top: 60 keys, api_key present again. Two tests: that a merge keeps what the template omits, and that the handler no longer dumps the template over the file. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Training a video LoRA from the Characters page returned: POST /v1/loras/train -> 401: Invalid API key. Provide a valid Bearer token. CoderAI gates every /v1 endpoint through the same dependency and makes NO exemption for loopback, so the bundled tools — which call 127.0.0.1 from inside the same container — need a Bearer token exactly like a remote client. The container launches this tool without --api-key and its saved config had none, so there was no token to send. Nothing was wrong with the training request itself. Fourteen call sites each built their client from default_args.api_key directly. They now go through one resolver with a stated order: the running options (set by --api-key, a saved config, or the Connection card) → the config file on disk → CODERAI_API_KEY in the environment, which is how a launcher can supply it without a config edit. A blank value is treated as absent rather than sent as "Bearer ", which would have failed in a more confusing way. A 401 now says what to do — where to put a key in THIS tool — instead of echoing a server message that tells an operator nothing about this UI. The immediate fix needs no rebuild: paste the key into the Connection card on the Run page and press Save config. That writes township_config.json and updates the running options, so the next action authenticates. 11 tests, including that no call site reads default_args.api_key any more. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Uploading a photo to a fighter was never missing — the Characters page has had a file picker and a working /profile/upload-image endpoint all along. It LOOKED missing because the page's whole <script> block has been dead since v0.2.41 (fixed in 11b0f446), so the button did nothing at all. Anyone on 0.2.41-0.2.46 sees a picker that ignores them. What was genuinely absent is the other half: pasting a link. That now feeds the SAME endpoint, so a fetched picture goes through the same magic-byte check (the extension is never trusted), the same ref_NN numbering, and the same patch_profile sync to CoderAI. The caller gets one answer covering both sources. I wrote a second endpoint for this first and deleted it: /profile/upload-image already did everything except fetch, and a parallel route would have been a second place to keep the validation, the numbering and the sync correct. The fetch is guarded the same way as the character studio's, for the same reason — the request is made by the tool, which can reach coderai on 127.0.0.1 and the whole LAN: - every hop must resolve to a public address, checked by RESOLVING the host rather than matching the string; - redirects followed by hand, five hops, each re-checked (a public URL that 302s to 127.0.0.1 is the usual way past a one-time check); - http/https only; 64 MB ceiling enforced while streaming; - the content must decode as PNG/JPEG/WebP/GIF by magic bytes. --allow-private-fetch opts out, for pulling from your own NAS. Verified against a real profile: an uploaded JPEG became ref_01.jpg, a public link became ref_02.png, and a link to 127.0.0.1:8776 was refused with the reason. "synced": false is reported honestly when CoderAI is unreachable rather than claiming the profile was updated. 131 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Paste a URL to a photo and the studio downloads it into the same uploads/ directory a picked file goes to. Nothing downstream changes: face extraction, the profile, the keyframes and LoRA training cannot tell the two apart, which is the point — "extract the character from it" is the pipeline that already exists. The part worth reviewing is that the STUDIO makes the request, not the browser. On this box that process can reach coderai on 127.0.0.1:8776, every admin API it exposes, and the whole LAN. So: - every hop must resolve to a public address. Private, loopback, link-local, reserved, multicast and unspecified are all refused, by resolving the host and checking the ADDRESS rather than pattern-matching the string; - redirects are followed by hand, five hops maximum, re-checking each one. A public URL that 302s to 127.0.0.1 is the standard way past a check done once; - only http/https. ftp://, file:// and the rest are refused by scheme; - 256 MB ceiling, enforced against Content-Length AND while streaming, since a server can lie about the first; - the content must be an image or a video, by URL suffix or, for the many image URLs with no suffix at all, by content type. --allow-private-fetch turns the address rule off for someone pulling from their own NAS. It is a flag rather than a default so it is the operator's decision. Verified against a running studio: loopback, 192.168/16, 169.254.169.254 and ftp:// are all refused with the reason; a real public image over the internet and a local one with the flag both land in uploads/ with the extension resolved from the content type when the URL had none. The tests also parse the page's script with node — the township page shipped a dead <script> for four versions because nothing ever parsed what it served, and this page now has a reason not to repeat that. 125 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The template picker rendered as a box you could not choose from. The cause is much worse than the symptom: the main <script> block has not parsed since v0.2.41, so NOTHING in it ran — status polling, the template buttons, the per-stage buttons, Save config, Stop. The page still rendered perfectly, which is why screenshots looked right and nothing in the logs complained. One character did it. The page is emitted from a Python f-string, so side.textContent = running ? ('Running\n'+(label||'')) : ... puts a REAL newline inside the JS string literal. The literal never closes, the block dies with a SyntaxError, and every function it defines is gone. It needed \\n, which is what it has now. Found by probing the server: the browser never requested /templates, nor /status, nor anything else the init block does — and a marker written as the first statement of that block never ran either. That ruled out the picker and pointed at the parse. Two things in the picker itself, both real: - tplRefresh() was only ever called AFTER a save or a delete, from the first templates commit (c1a65e1d). A freshly loaded page never listed what was on disk, so the picker was empty even with the script working — and my v0.2.40 claim that "the picker is never an empty box" was simply wrong. - The select had no options at all before its first fill, which is why it read as an input field rather than a dropdown. It now ships a placeholder and is width- constrained. Verified against the real templates directory: all four script blocks parse, and the picker lists "big frames", "LongCat - few long scenes" and "Wan - many short scenes" with their dates and option counts. The test starts the real server, fetches the real page, and parses every script block with node. It is verified to FAIL on the defect: reintroducing the bare \n makes it fail with the SyntaxError. A dependency-free scanner was written first and deleted — it reported the known-broken block as clean, and a check that passes on a file proven broken is worse than no check. 115 tests. Co-Authored-By:
-
- 05 Oct, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
The page was slow to open on a large configuration. Measuring first, because the obvious suspect was wrong: the disk scans are not the problem — scan_cache_dir over 59 repos is 0.42s and the GGUF walk 0.53s, and _scan_caches() is 0.08s once warm. The server-side stale-while-revalidate cache was already doing its job. What it was, in order of cost: THE CACHE DIED WITH THE PROCESS. It starts empty in every new one, and _cached() recomputes synchronously when it has nothing — so the first read after a restart pays the whole scan while someone sits looking at a spinner. Measured on the real configuration: 3.69s cold, 0.001s once the previous result is read back from disk. It is now persisted (atomically, under <config>/cache/) and seeded on first use. The stored SIGNATURE is deliberately not restored: it is a list of mtimes, and trusting it across a restart would serve a possibly hours-old list without revalidating. Leaving it unset means the stored result is returned instantly AND a refresh starts behind it. A missing, truncated or future-version file just falls back to scanning. THE PAGE WAITED ON ITSELF. First paint ran refreshLoadedStatus, then refreshQuantJobs, then the list — three round trips in a chain — while _loadEngineInfo did two more in another chain and then rendered the list a SECOND time. Inside the list load, /admin/api/cached-models and /admin/api/models also went out one after the other. Now: the two status reads go out together, the two list reads go out together, and concurrent loadCachedModels() calls collapse into one in-flight load, so first paint does one fetch and one render instead of three. A call made after it settles still refetches, which every post-action refresh relies on. IT PAINTED NOTHING UNTIL ALL OF THAT FINISHED. The last response is kept in sessionStorage and painted before any network call, then revalidated in place. It is a listing of what is on disk, so seconds of staleness cost nothing — and a quota error or private mode just means the page behaves as it does today. One thing the tests had to be corrected on: the first version asserted the background rescan had not run, which failed because _cached() starts the refresh thread and then reads state["data"] — with a trivial scan function the thread wins the race. That race is harmless (fresher data), and it was the assertion that was wrong: the property worth holding is that the CALLER is not blocked, so the test now uses a slow scan and asserts the read returns in under 0.2s with the stored value, and that the revalidation really did start. 112 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
"Free disk" on the colibri GLM-5.2 model returned 200 OK three times and freed 4 KB. The 400 GB was in /AI/offloads/colibri/models--mastouri--…, and the HF cache held that model as a SYMLINK into it — so _do_delete_model deleted the link's entry, found nothing else it knew about, and truthfully reported success. Two separate failures, fixed separately. IT DID NOT SAY HOW MUCH. Freeing 4 KB and freeing 400 GB were the same {"success": true}. Every branch now measures what it removed and returns freed_bytes/freed_human; the UI prints it, and says so plainly when the answer is zero ("Nothing was on disk for this model — freed 0 B. Its files may live somewhere coderai has not been told about."). This is the part that would have made the problem visible on the first click instead of the third. IT ONLY LOOKED IN TWO PLACES. codai/models/weight_stores.py adds the rest: - The configured offload directory AND its siblings, one level. Engines are pointed at sibling roots by convention — coderai offloads to …/offloads/nvidia and colibri's weights landed in …/offloads/colibri. The scan widens where we LOOK; it never widens what we delete, which stays an exact models--<repo> name match. Nothing is ever removed by prefix or pattern. - register_weight_root() for an engine that does not follow that convention. Deliberately not called from colibri_worker/ds4_worker: at import time config_manager is not loaded yet, so the offload directory is '' and the registration would be dead code. The sibling scan runs at DELETE time, off config, and covers both. Measuring the size honestly took two passes: - symlinks are not followed or counted (an HF snapshot is a tree of links into blobs/, so following them doubles the number); - hardlinks are counted once by (device, inode), since a cache that links rather than copies would otherwise report the weights once per link. And purging a store leaves the cache entry that pointed INTO it dangling, which still looks like a present model to anything that only stats the name — so dangling links to a purged store are pruned too. Verified against the real model before touching it: a dry run found exactly one path, 399.8 GB, and nothing else among the nine roots it searched. The delete then freed it — /dev/nvme0n1p4 went from ~96% to 1% used, 444 GB free. (Worth recording: I first read df on /AI and reported the space had NOT come back, before noticing /AI/offloads is its own mount. Same wrong-filesystem mistake as earlier in this session. The deletion had worked the whole time.) 100 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Step 4b failed in production: RuntimeError: longcat LoRA training needs `dataset_config` — a path to a SimpleTuner data-backend config describing your captioned video clips The refusal is right: LongCat trains through SimpleTuner, which wants captioned VIDEO clips, and synthesising a dataset from a fighter's reference stills would train something other than a video LoRA. But the UI offered no way to supply one, so the button could not succeed — and the tool was already sitting on exactly the corpus SimpleTuner asks for. Every clip it renders was rendered FROM a prompt, and that prompt is kept beside it in videos/prompts.json. So it now assembles the dataset rather than demanding it. Per fighter and per location, against a LongCat video model: <out>/video_datasets/character_<name>__<slug>/ clips/0000.mp4 -> ../../../videos/match_a_vs_b_clip03.mp4 clips/0000.txt the prompt that produced those pixels cache/vae cache/text data_backend.json sent as dataset_config manifest.json which render each clip came from Decisions worth stating: - SYMLINKS, not copies. A fighter appears in dozens of clips; a dataset per fighter would otherwise duplicate the whole render per fighter. - Captions come from the PLAN, never from filenames, so a caption is always the text that actually produced the pixels. - _2x/_2xfps derivatives are never picked up. They are the upscaler's and the frame interpolator's output; training on them teaches their artefacts. - An outcome clip is training data for BOTH fighters — they are both on screen. - Clips shorter than the training frame count are filtered out HERE, because SimpleTuner's loader drops them silently. A dataset where every clip is too short refuses with the numbers: "all 12 clips are shorter than 93 frames (longest 70)". Likewise "only 3 usable clips, need 8" and "no rendered clips yet (run the Videos step first)" — a skip with a reason, not a stack trace. - Rebuilt from scratch each time, so a symlink to a deleted render cannot fail deep inside SimpleTuner's loader instead of here. Two knobs on the Run page, in the config and on the CLI: Training clip frames (default 93 — one whole LongCat segment, and 4n+1 as the VAE requires) and Minimum usable clips (default 8; upstream suggests 50-100, this is a floor so a part-rendered show does not quietly train on three). The backend schema is upstream's documented one, including exactly one text_embeds backend carrying default: true. The client sends dataset_config only when it has one; the server has always accepted the field and the in-process Wan path ignores it. Checked against the real township render: 151 sources (102 clips + 49 outcomes), 19 usable for the busiest fighter at 93 frames. 82 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found in a production log, during a FULL VRAM CLEANUP ahead of a LoRA training job: Warning in external VRAM releaser: release_f5_engine() takes 0 positional arguments but 1 was given The manager calls every external releaser as `fn(needed_gb)` and sums what they return. release_f5_engine() declared no parameter, so every call raised TypeError straight into the manager's warning path: the cached F5-TTS engine held its VRAM through every eviction, including the exclusive ones that exist precisely because training needs the whole card. It failed quietly — the warning is one line in a cleanup that otherwise reports success. It now takes needed_gb and returns the VRAM delta it observed, like the other four releasers. The test checks the shape statically, across every register_external_vram_releaser call in the tree, so the next releaser cannot be registered with a signature the manager cannot call. Written against the real contract: one positional argument, and a value returned. (It first asserted a default was required too, which flagged _cosite_vram_releaser — but the manager always passes the argument, so that was the test inventing a rule.) 11 tests. Added with -f: .gitignore ignores test_*.py by default. Co-Authored-By:
-