- 06 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored✗ training failed: _lora_dir() missing 1 required positional argument: 'name' Progress, in that everything before it worked: the Train button built the dataset, the request carried dataset_config, the server accepted it and dispatched to the LongCat path. It then died on the first line of _train_externally that does real work — proof that function had never once executed. Two bugs in one line. _lora_dir(name) already resolves the per-LoRA directory AND routes a non-main DDP rank to scratch; the call treated it as the parent and joined the name on again, so even with the argument supplied it would have written <loras>/<name>/<name> and bypassed the DDP handling. The test calls the real _train_externally with only the worker stubbed — no GPU, no checkpoint, no venv — and checks it reaches the worker, that output_dir equals _lora_dir(name) with the name not doubled, that the job carries steps/rank/seed/ num_frames/dataset_config, that both refusal paths still refuse, and that the worker's **kwargs progress callback matches the receiver's signature (a mismatch there would only show up mid-training). Verified to FAIL on the real defect: restoring the old line turns 4 of the 6 red with exactly the TypeError above. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
The log shows the 401 is gone — the latest attempt was accepted (200 OK, job-551586975a13) and reached the trainer. It then failed on the SAME error as before the key was fixed: RuntimeError: longcat LoRA training needs `dataset_config` Because there are two training paths and only one of them was wired. Step 4b (the batch stage) builds a SimpleTuner dataset from the rendered clips. The per-profile Train button on the Characters page — which is what was being pressed — assembled its kwargs independently and never passed dataset_config, so the server refused the job exactly as it is supposed to. Both paths now build the same dataset through the same function. When it cannot (no clips rendered yet, clips too short, too few), the button says so and names the remedy instead of letting the server answer with a stack trace. Surveyed against the real township output before claiming it works: 151 clips and outcomes carry prompts, and at the default 93-frame / 8-clip threshold all 6 fighters and all 4 environments qualify — khumalo has 19 usable clips, dlamini 11. Still true, and unchanged by this: no LongCat training has ever RUN. This gets the job past the server's gate; what SimpleTuner then does with it is unproven. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 401 on video LoRA training was not a missing key. It was a DELETED one. Loading a template persisted by writing the template over township_config.json: _j.dump(tcfg, f, indent=2, sort_keys=True) A template is a PARTIAL — the LongCat starter carries 10 options, the config carries 60. So loading it turned the operator's config into those 10 keys and destroyed everything else, api_key and base_url included. Every subsequent call to CoderAI then failed with a bare 401 that said nothing about why. Found by comparing the live file against its own backups: township_config.json was 238 bytes with exactly the LongCat starter's 10 keys, timestamped during this session; every backup beside it was ~1700 bytes with 60 keys and a 75-character api_key. The token itself was never gone — it is still 'fighters' in auth.json. It now reads the existing config, updates it with the template, and writes the result. A template's values win; nothing it is silent about is touched. With no readable config yet, the template is the config, as before. This is mine, from 310ffc79 ("loading a template persists, not just applies") in v0.2.39. The persistence was right; replacing rather than merging was not. The operator's config has been restored in place from the newest full backup with the LongCat values kept on top: 60 keys, api_key present again. Two tests: that a merge keeps what the template omits, and that the handler no longer dumps the template over the file. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Training a video LoRA from the Characters page returned: POST /v1/loras/train -> 401: Invalid API key. Provide a valid Bearer token. CoderAI gates every /v1 endpoint through the same dependency and makes NO exemption for loopback, so the bundled tools — which call 127.0.0.1 from inside the same container — need a Bearer token exactly like a remote client. The container launches this tool without --api-key and its saved config had none, so there was no token to send. Nothing was wrong with the training request itself. Fourteen call sites each built their client from default_args.api_key directly. They now go through one resolver with a stated order: the running options (set by --api-key, a saved config, or the Connection card) → the config file on disk → CODERAI_API_KEY in the environment, which is how a launcher can supply it without a config edit. A blank value is treated as absent rather than sent as "Bearer ", which would have failed in a more confusing way. A 401 now says what to do — where to put a key in THIS tool — instead of echoing a server message that tells an operator nothing about this UI. The immediate fix needs no rebuild: paste the key into the Connection card on the Run page and press Save config. That writes township_config.json and updates the running options, so the next action authenticates. 11 tests, including that no call site reads default_args.api_key any more. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Uploading a photo to a fighter was never missing — the Characters page has had a file picker and a working /profile/upload-image endpoint all along. It LOOKED missing because the page's whole <script> block has been dead since v0.2.41 (fixed in 11b0f446), so the button did nothing at all. Anyone on 0.2.41-0.2.46 sees a picker that ignores them. What was genuinely absent is the other half: pasting a link. That now feeds the SAME endpoint, so a fetched picture goes through the same magic-byte check (the extension is never trusted), the same ref_NN numbering, and the same patch_profile sync to CoderAI. The caller gets one answer covering both sources. I wrote a second endpoint for this first and deleted it: /profile/upload-image already did everything except fetch, and a parallel route would have been a second place to keep the validation, the numbering and the sync correct. The fetch is guarded the same way as the character studio's, for the same reason — the request is made by the tool, which can reach coderai on 127.0.0.1 and the whole LAN: - every hop must resolve to a public address, checked by RESOLVING the host rather than matching the string; - redirects followed by hand, five hops, each re-checked (a public URL that 302s to 127.0.0.1 is the usual way past a one-time check); - http/https only; 64 MB ceiling enforced while streaming; - the content must decode as PNG/JPEG/WebP/GIF by magic bytes. --allow-private-fetch opts out, for pulling from your own NAS. Verified against a real profile: an uploaded JPEG became ref_01.jpg, a public link became ref_02.png, and a link to 127.0.0.1:8776 was refused with the reason. "synced": false is reported honestly when CoderAI is unreachable rather than claiming the profile was updated. 131 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Paste a URL to a photo and the studio downloads it into the same uploads/ directory a picked file goes to. Nothing downstream changes: face extraction, the profile, the keyframes and LoRA training cannot tell the two apart, which is the point — "extract the character from it" is the pipeline that already exists. The part worth reviewing is that the STUDIO makes the request, not the browser. On this box that process can reach coderai on 127.0.0.1:8776, every admin API it exposes, and the whole LAN. So: - every hop must resolve to a public address. Private, loopback, link-local, reserved, multicast and unspecified are all refused, by resolving the host and checking the ADDRESS rather than pattern-matching the string; - redirects are followed by hand, five hops maximum, re-checking each one. A public URL that 302s to 127.0.0.1 is the standard way past a check done once; - only http/https. ftp://, file:// and the rest are refused by scheme; - 256 MB ceiling, enforced against Content-Length AND while streaming, since a server can lie about the first; - the content must be an image or a video, by URL suffix or, for the many image URLs with no suffix at all, by content type. --allow-private-fetch turns the address rule off for someone pulling from their own NAS. It is a flag rather than a default so it is the operator's decision. Verified against a running studio: loopback, 192.168/16, 169.254.169.254 and ftp:// are all refused with the reason; a real public image over the internet and a local one with the flag both land in uploads/ with the extension resolved from the content type when the URL had none. The tests also parse the page's script with node — the township page shipped a dead <script> for four versions because nothing ever parsed what it served, and this page now has a reason not to repeat that. 125 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The template picker rendered as a box you could not choose from. The cause is much worse than the symptom: the main <script> block has not parsed since v0.2.41, so NOTHING in it ran — status polling, the template buttons, the per-stage buttons, Save config, Stop. The page still rendered perfectly, which is why screenshots looked right and nothing in the logs complained. One character did it. The page is emitted from a Python f-string, so side.textContent = running ? ('Running\n'+(label||'')) : ... puts a REAL newline inside the JS string literal. The literal never closes, the block dies with a SyntaxError, and every function it defines is gone. It needed \\n, which is what it has now. Found by probing the server: the browser never requested /templates, nor /status, nor anything else the init block does — and a marker written as the first statement of that block never ran either. That ruled out the picker and pointed at the parse. Two things in the picker itself, both real: - tplRefresh() was only ever called AFTER a save or a delete, from the first templates commit (c1a65e1d). A freshly loaded page never listed what was on disk, so the picker was empty even with the script working — and my v0.2.40 claim that "the picker is never an empty box" was simply wrong. - The select had no options at all before its first fill, which is why it read as an input field rather than a dropdown. It now ships a placeholder and is width- constrained. Verified against the real templates directory: all four script blocks parse, and the picker lists "big frames", "LongCat - few long scenes" and "Wan - many short scenes" with their dates and option counts. The test starts the real server, fetches the real page, and parses every script block with node. It is verified to FAIL on the defect: reintroducing the bare \n makes it fail with the SyntaxError. A dependency-free scanner was written first and deleted — it reported the known-broken block as clean, and a check that passes on a file proven broken is worse than no check. 115 tests. Co-Authored-By:
-
- 05 Oct, 2026 21 commits
-
-
Stefy Lanza (nextime / spora ) authored
The page was slow to open on a large configuration. Measuring first, because the obvious suspect was wrong: the disk scans are not the problem — scan_cache_dir over 59 repos is 0.42s and the GGUF walk 0.53s, and _scan_caches() is 0.08s once warm. The server-side stale-while-revalidate cache was already doing its job. What it was, in order of cost: THE CACHE DIED WITH THE PROCESS. It starts empty in every new one, and _cached() recomputes synchronously when it has nothing — so the first read after a restart pays the whole scan while someone sits looking at a spinner. Measured on the real configuration: 3.69s cold, 0.001s once the previous result is read back from disk. It is now persisted (atomically, under <config>/cache/) and seeded on first use. The stored SIGNATURE is deliberately not restored: it is a list of mtimes, and trusting it across a restart would serve a possibly hours-old list without revalidating. Leaving it unset means the stored result is returned instantly AND a refresh starts behind it. A missing, truncated or future-version file just falls back to scanning. THE PAGE WAITED ON ITSELF. First paint ran refreshLoadedStatus, then refreshQuantJobs, then the list — three round trips in a chain — while _loadEngineInfo did two more in another chain and then rendered the list a SECOND time. Inside the list load, /admin/api/cached-models and /admin/api/models also went out one after the other. Now: the two status reads go out together, the two list reads go out together, and concurrent loadCachedModels() calls collapse into one in-flight load, so first paint does one fetch and one render instead of three. A call made after it settles still refetches, which every post-action refresh relies on. IT PAINTED NOTHING UNTIL ALL OF THAT FINISHED. The last response is kept in sessionStorage and painted before any network call, then revalidated in place. It is a listing of what is on disk, so seconds of staleness cost nothing — and a quota error or private mode just means the page behaves as it does today. One thing the tests had to be corrected on: the first version asserted the background rescan had not run, which failed because _cached() starts the refresh thread and then reads state["data"] — with a trivial scan function the thread wins the race. That race is harmless (fresher data), and it was the assertion that was wrong: the property worth holding is that the CALLER is not blocked, so the test now uses a slow scan and asserts the read returns in under 0.2s with the stored value, and that the revalidation really did start. 112 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
"Free disk" on the colibri GLM-5.2 model returned 200 OK three times and freed 4 KB. The 400 GB was in /AI/offloads/colibri/models--mastouri--…, and the HF cache held that model as a SYMLINK into it — so _do_delete_model deleted the link's entry, found nothing else it knew about, and truthfully reported success. Two separate failures, fixed separately. IT DID NOT SAY HOW MUCH. Freeing 4 KB and freeing 400 GB were the same {"success": true}. Every branch now measures what it removed and returns freed_bytes/freed_human; the UI prints it, and says so plainly when the answer is zero ("Nothing was on disk for this model — freed 0 B. Its files may live somewhere coderai has not been told about."). This is the part that would have made the problem visible on the first click instead of the third. IT ONLY LOOKED IN TWO PLACES. codai/models/weight_stores.py adds the rest: - The configured offload directory AND its siblings, one level. Engines are pointed at sibling roots by convention — coderai offloads to …/offloads/nvidia and colibri's weights landed in …/offloads/colibri. The scan widens where we LOOK; it never widens what we delete, which stays an exact models--<repo> name match. Nothing is ever removed by prefix or pattern. - register_weight_root() for an engine that does not follow that convention. Deliberately not called from colibri_worker/ds4_worker: at import time config_manager is not loaded yet, so the offload directory is '' and the registration would be dead code. The sibling scan runs at DELETE time, off config, and covers both. Measuring the size honestly took two passes: - symlinks are not followed or counted (an HF snapshot is a tree of links into blobs/, so following them doubles the number); - hardlinks are counted once by (device, inode), since a cache that links rather than copies would otherwise report the weights once per link. And purging a store leaves the cache entry that pointed INTO it dangling, which still looks like a present model to anything that only stats the name — so dangling links to a purged store are pruned too. Verified against the real model before touching it: a dry run found exactly one path, 399.8 GB, and nothing else among the nine roots it searched. The delete then freed it — /dev/nvme0n1p4 went from ~96% to 1% used, 444 GB free. (Worth recording: I first read df on /AI and reported the space had NOT come back, before noticing /AI/offloads is its own mount. Same wrong-filesystem mistake as earlier in this session. The deletion had worked the whole time.) 100 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Step 4b failed in production: RuntimeError: longcat LoRA training needs `dataset_config` — a path to a SimpleTuner data-backend config describing your captioned video clips The refusal is right: LongCat trains through SimpleTuner, which wants captioned VIDEO clips, and synthesising a dataset from a fighter's reference stills would train something other than a video LoRA. But the UI offered no way to supply one, so the button could not succeed — and the tool was already sitting on exactly the corpus SimpleTuner asks for. Every clip it renders was rendered FROM a prompt, and that prompt is kept beside it in videos/prompts.json. So it now assembles the dataset rather than demanding it. Per fighter and per location, against a LongCat video model: <out>/video_datasets/character_<name>__<slug>/ clips/0000.mp4 -> ../../../videos/match_a_vs_b_clip03.mp4 clips/0000.txt the prompt that produced those pixels cache/vae cache/text data_backend.json sent as dataset_config manifest.json which render each clip came from Decisions worth stating: - SYMLINKS, not copies. A fighter appears in dozens of clips; a dataset per fighter would otherwise duplicate the whole render per fighter. - Captions come from the PLAN, never from filenames, so a caption is always the text that actually produced the pixels. - _2x/_2xfps derivatives are never picked up. They are the upscaler's and the frame interpolator's output; training on them teaches their artefacts. - An outcome clip is training data for BOTH fighters — they are both on screen. - Clips shorter than the training frame count are filtered out HERE, because SimpleTuner's loader drops them silently. A dataset where every clip is too short refuses with the numbers: "all 12 clips are shorter than 93 frames (longest 70)". Likewise "only 3 usable clips, need 8" and "no rendered clips yet (run the Videos step first)" — a skip with a reason, not a stack trace. - Rebuilt from scratch each time, so a symlink to a deleted render cannot fail deep inside SimpleTuner's loader instead of here. Two knobs on the Run page, in the config and on the CLI: Training clip frames (default 93 — one whole LongCat segment, and 4n+1 as the VAE requires) and Minimum usable clips (default 8; upstream suggests 50-100, this is a floor so a part-rendered show does not quietly train on three). The backend schema is upstream's documented one, including exactly one text_embeds backend carrying default: true. The client sends dataset_config only when it has one; the server has always accepted the field and the in-process Wan path ignores it. Checked against the real township render: 151 sources (102 clips + 49 outcomes), 19 usable for the busiest fighter at 93 frames. 82 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found in a production log, during a FULL VRAM CLEANUP ahead of a LoRA training job: Warning in external VRAM releaser: release_f5_engine() takes 0 positional arguments but 1 was given The manager calls every external releaser as `fn(needed_gb)` and sums what they return. release_f5_engine() declared no parameter, so every call raised TypeError straight into the manager's warning path: the cached F5-TTS engine held its VRAM through every eviction, including the exclusive ones that exist precisely because training needs the whole card. It failed quietly — the warning is one line in a cleanup that otherwise reports success. It now takes needed_gb and returns the VRAM delta it observed, like the other four releasers. The test checks the shape statically, across every register_external_vram_releaser call in the tree, so the next releaser cannot be registered with a signature the manager cannot call. Written against the real contract: one positional argument, and a value returned. (It first asserted a default was required too, which flagged _cosite_vram_releaser — but the manager always passes the argument, so that was the test inventing a rule.) 11 tests. Added with -f: .gitignore ignores test_*.py by default. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 0.2.40 "restyle" rewrote the stylesheet into tokens and kept every class and every piece of markup, so it rendered as the same page in a slightly different grey. It also could not have shown: the markup carried 270 inline style attributes, 32 of which re-shrank a button below the type scale it had just been given, and 47 of which hardcoded a colour. This changes the structure. THE SHELL. Three regions that do not move, instead of one 1180px column: - A left sidebar with the six sections, each one an icon plus a label, and the active one carried by aria-current. Under ~900px it becomes a strip of icons above the content rather than eating a third of a phone. - A sticky top bar holding the page title and, on the Run page, the controls that start, stop and persist a run. Starting a render used to mean scrolling past 66 settings to reach the button. The status pill sits with them. - A log dock fixed to the bottom. Watching a render used to mean scrolling to the end of the page and staying there. It remembers whether it was open, and opens itself the moment a run starts, which is the one time it is worth reading. Templates and the per-stage buttons move from under the settings to a card above them, in the order they are used: pick a template, see it in the form, run a stage. The Refresh links on Characters/Environments/Matches move to the top bar, and the per-page <h1> goes with them, since the bar already names the page. THE FORM. Above 1220px the cards flow in two columns, which roughly halves the height of a 66-setting form; the four wide cards span both. Each card is a container query root, so a row inside a half-width card collapses to the card's width rather than the window's. A label that wraps to two lines no longer pushes its field out of line with the one beside it. A BUG THIS SURFACED. The wardrobe inputs declared no type=, and [type=text] does not match a missing attribute, so every outfit field had been rendering as a white browser-default box on a dark page. The selector now covers them. The inline styles are mostly gone: 46 button size attributes folded into .btn-sm /.btn-xs, and 144 hardcoded hex values replaced by tokens, including four hues (violet, sage, tan, deep green) that the markup had been spelling out by hand and that are now declared once. THE STARTERS, answering what they left out. Both only carried video settings, so switching to LongCat left the outcome stings at Wan's 96-150 frames — neither 4n+1 nor segment-aligned, so the server had to round them. Both now also set the outcome range (LongCat: 93, one whole segment, to 173), frame interpolation (off for LongCat, which does not need it at 15fps), the upscale factor, the keyframe size and the match count (3 for LongCat, where a scene is ~4x the frames). Both cover the same keys, so switching between them is deterministic. What they still do not set, deliberately: the character and environment stages. Those are image-side work — the video model never sees the pool — so a template that pinned the pool size or the reference-image budget would reset an operator's setup for no reason. All six pages rendered and screenshotted at 1600px and 820px. 53 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
THE UI. One style block, ~40 hardcoded hex values, 0.72-0.85rem type throughout, caps section labels, no focus ring, and an infinite striped animation with no reduced-motion guard. Rewritten as a token system, keeping every existing class so no page markup had to change: - Palette from the subject rather than a template: cool graphite (not a tinted black), title-belt gold for the single primary action, and red/blue corner as a SEMANTIC pair — every bout is fighter 1 vs fighter 2, so those two carry meaning here. - Numbers set in a monospace with tabular figures: frame counts, odds and durations are compared down a column. - 16px fields everywhere, not just on mobile — .85rem was cramped on a desktop too, and 16px is what stops iOS zooming on focus. - Numeric inputs capped at 11rem. A frame count is three digits; a box stretched across half the viewport makes the form longer to scan and harder to aim at. - Section headings instead of tracked-out caps labels, which were harder to read and made every group shout. - Sticky nav, with the active page carried by aria-current — it drives the underline AND tells a screen reader where you are, where the old inline colour did neither. - :focus-visible everywhere, and prefers-reduced-motion collapses the animated bar. Reviewed against renders of all six pages; the density pass alone fits roughly one extra section per screen on a form with 66 settings. TWO STARTER TEMPLATES, seeded on first use and never overwritten, so the picker is never an empty box. They describe the same show rendered by models with opposite strengths: Wan 8fps, 50-70 frame clips -> a 45s short is 6-8 scenes LongCat 15fps, 173-333 frames -> the same short is 2-4 scenes LongCat's numbers are its own: 15fps is the rate it was trained at (8 was a Wan-era compromise), 173 = 93+80 and 333 = 93+80x3 are whole segment counts, and all of them are 4n+1 — the VAE's rule, which the server also checks before a long run. Since the server owns the segment loop for LongCat, those long scenes are one request each rather than chained sub-renders with a re-encoded join and accumulating drift. 49 tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading applied the template to the live session but never wrote the active config file, which made "switch live" only half true: - the launcher starts the tool with --config <out_dir>/township_config.json, so the template silently reverted on the next restart; - _ref_gen_res_steps deliberately RE-READS that file from disk for keyframe_size and keyframe_steps (so Run-page edits apply without a restart), which meant those two options came from the stale file even in the same session — the loaded template disagreeing with itself. The load now writes the active config too, reports the path back as `persisted`, and says so in the log if the write fails rather than leaving a switch that looks applied and reverts later. Verified live: save a template with keyframe_size 1024x576, load it, and the active config file on disk carries it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A saved config was one JSON file; --config/--save let you keep several by juggling paths. A template is the same content under a NAME, in a known place, with list, load and delete — so "the LongCat long-shot setup" or "the quick 480p draft" is something you pick rather than a path you remember. Two properties beyond convenience: - A template is a COMPLETE configuration — every CONFIG_FIELDS key, 66 of them — not a patch. Loading one therefore cannot leave an option from the previous run in place. - The name is a SECURITY boundary, because names arrive from the CLI and from a browser. _safe_template_name strips separators and traversal and refuses a name with nothing usable; a test asserts that whatever is asked for, the resolved path stays inside the templates directory. "../../etc/passwd" becomes "etc-passwd". They live at <out_dir>/templates/<name>.json, because that is the directory the launcher already maps and persists — a template written to a container-local path would vanish on the next run. CLI: -T/--template NAME (layered over --config; explicit arguments still win, so a template can be a base you tweak per run), --save-template, --list-templates (with date, video model and option count), --delete-template. --config/--save are untouched. Web: GET /templates, POST /templates/load, POST /templates/delete, and a picker on the Run page with Load / Save as… / Delete. Saving is handled INSIDE /save-config rather than in its own route, so the large form-to-config mapping is not duplicated and cannot drift — a test pins that. Loading applies to default_args exactly as /save-config does, since the Run page renders from it and a load that only wrote a file would appear to do nothing until restart; it asks for confirmation first, because replacing every option is not an undoable click. A corrupt template is listed with its error rather than hiding the rest of the picker. 39 tests. Verified live against the web UI: save, list, load, delete, a missing name, and a traversal attempt contained. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Plan item I, the last of the LongCat work. The tool chains clips because the server generates one clip per request and VACE frame-tail conditioning is the only continuation a Wan model has. That machinery exists to fight drift — the dual-expert and VACE-extension notes are both about chained parts going wrong — and every join is a re-encode. LongCat was pretrained on continuation, so: - the client-side split is LIFTED for it (_chunk_max becomes unbounded): a clip's whole frame budget goes in ONE request and the server runs the segment loop. Chaining it from here would reintroduce exactly the drift the chaining was invented to avoid. Wan/VACE keeps the cap it always had; - between DELIBERATE shots the tool still chains, because a cut is a cut — but LongCat is tail-capable too, so it takes the same cond_frames handover. What differs is underneath, not the call; - the tail length follows the model: 13 of 93 frames, which is what LongCat's continuation pretraining used, against VACE's 5. Handing it 5 would condition it on less than it expects. The constant is asserted against tools/longcat_common.py so the two cannot drift apart; - the log says which continuation produced a shot, because diagnosing a seam starts there. Server side, so a client asks for the frames it wants rather than reimplementing the loop: num_frames is the PER-CALL count, and a request for more than one call's worth now derives total_frames and lets the segment loop cover it. An explicit num_segments or duration_seconds still wins. Passing 400 straight through would have asked the pipeline for a 400-frame single call. The uploader and the odds generation are untouched — they are downstream of the frames. 17 tests. No match has been rendered with it: that needs the weights. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Plan item H.2. Also fixes a real bug shipped in 0.2.35. THE BUG: load_quantized_dit was imported from longcat_video.modules.longcat_video_dit, where it does not exist — it is in longcat_video.modules.quantization (beside QuantizedLinear and quantize_model), and it returns the AVATAR transformer, consistent with INT8 being an avatar-1.5-only build. The wrong import would have raised at the exact moment someone asked for INT8. TRAINING. coderai does not implement this loop, and the reasons are the design: 1. LongCat's venv is standalone (Python 3.10 / torch 2.6 / transformers 4.41), so a trainer there cannot `from codai.api import loras` the way tools/lora_train_worker.py does — that overlay venv inherits the parent's site-packages, this one does not. 2. The upstream repo has NO way to create trainable LoRA layers. Its DiT exposes load_lora() / enable_loras() / disable_all_loras() and nothing that initialises one, and the adapters it loads (cfg_step_lora, refinement_lora, dmd_lora) use its own key layout. Reimplementing that layout unverified would produce adapters the pipeline cannot load: support in appearance only. SimpleTuner implements LongCat-Video properly — model_family "longcat_video", LoRA and quantised LoRA via int8-quanto / int4-quanto / fp8-torchao with no extra installs — so coderai generates its config and drives it: - lora_archs gains the arch with an `external` marker; load_classes refuses with that reason instead of an AttributeError that reads like a diffusers version problem. `target: video` still resolves Wan weights to Wan, so existing configs are untouched. - tools/longcat_train.py writes the config, runs SimpleTuner, parses its steps into the same JSON-lines protocol the overlay trainer uses, and reports the adapter it actually wrote rather than assuming one appeared. - Two VAE constraints are checked BEFORE a long run starts: (num_frames - 1) divisible by 4 — the same 4n+1 rule coderai already applies to Wan — and both sides divisible by 16. - A venv of its own, because SimpleTuner pins its own torch and would fight the inference venv's. train_auto_build is off by default, and the refusal carries the manual command. - QLoRA defaults ON (int8-quanto): a 13.6B transformer plus optimiser state does not fit a consumer card at bf16, so the quantised base is the normal path. `dataset_config` is REQUIRED and refused when absent or missing: SimpleTuner wants captioned video clips, and synthesising a dataset from the `images` field other targets use would train something other than a video LoRA, silently. 34 tests. No training run has been executed — the driver and its constraints are tested, the run against a real dataset is not. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 7, the last of the server-side work. Two things that are more than flags. CONTEXT PARALLELISM is N processes under torchrun, not N threads, and every rank must enter the same pipeline call or the collectives deadlock. So: - NCCL and init_context_parallel() run BEFORE the model loads — that call is what makes the DiT split its spatial dims, so load_pipeline cannot come first; - rank 0 owns the HTTP socket and BROADCASTS each job before entering the pipeline; the other ranks sit in cp_worker_loop waiting for one and discard their copy of the result. A rank-0-only server that simply called generate() would hang on the first collective; - the group is told to shut down when the server stops, or the non-zero ranks outlive it; - cp_size 1 (the default) initialises no distributed group at all; - asking for more ranks than there are visible GPUs is refused up front, because NCCL's own failure for that names nothing useful. THE POD IMAGE could not use the generic builder: Dockerfile.capability is python:3.12 on the cu128 torch index, and LongCat needs 3.10 on cu124 — two interpreters in one image, the light core running coderai and a 3.10 venv running the model. So video-longcat carries its own Dockerfile (the escape hatch the engines profile uses), builds the venv from the SAME requirements-longcat.txt a local install uses, asserts 3.10 / torch 2.6 / transformers 4.41 at build time rather than discovering a different resolution on a rented pod, clones the pipeline source (it is not on PyPI) and checks it imports, and bakes no weights. model_capability() now routes a `backend: longcat` entry to that image. Left alone it would land on the plain 'video' image, which cannot run LongCat at all — a pod that boots, passes its health check, accepts the request and then has nothing to generate with, exactly the trap the OCR profile documents for Surya. An explicit `capability` override still wins. 24 new tests, 212 across the seven LongCat suites. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 6. Read from upstream's avatar demo rather than assumed, because almost nothing carries over from the base model: - avatar is a DIFFERENT pipeline class, LongCatVideoAvatarPipeline, with its own methods generate_at2v / generate_ai2v / generate_avc; - it takes TWO guidance scales — text and audio — where the base pipeline takes one; - its distilled pass is dmd_lora at 8 steps with lora_network_dim=128/alpha=64 and dit.enable_loras(["dmd"]), NOT cfg_step_lora at 16; - use_int8 is a separate load_quantized_dit(subfolder="base_model_int8"), not a dtype; - both INT8 and the DMD LoRA exist only for avatar-1.5, which also REQUIRES distilled sampling; - the audio is librosa.load(sr=16000) then pipe.get_audio_embedding(...), and the encoder differs per family (chinese-wav2vec2-base for Avatar, whisper-large-v3 for Avatar-1.5). Unsupported combinations are refused BEFORE loading, because the alternative is quietly serving something that is not what was asked for: INT8 or DMD on anything but avatar-1.5, avatar-1.5 without distilled sampling, a base-model mode on avatar weights or the reverse, and GGUF, which needs ComfyUI's gguf loader and says so. FP8 loads the DiT as float8_e4m3fn. Community layouts differ between publishers (Kijai's scaled format is not a plain e4m3 dump), so a repo that needs its own loader fails with that reason rather than producing noise. Also: librosa is a real new dependency and went into requirements-longcat.txt, not only into the code that imports it. And the avatar audio comes from the request's `audio_file` field — `add_audio` is a BOOLEAN for the post-processing path, and using it would have passed True into the base64 encoder. 35 new tests, 188 across the six LongCat suites. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 5. The requirement is "full configuration in the interface", and the trap is silent: api_model_configure REBUILDS the entry from scratch and copies only an explicit whitelist, so a key with a UI control but no whitelist entry is written once and deleted on the next save from the models page. It looks saved until you reload. That is precisely why H3's documented h3_venv / in_process / dtype keys never survive, and it is not inherited here. - `longcat` is now an option in the hard-coded cfg-backend select — without that the backend cannot be pinned from the UI at all, which is the other half of why H3's `backend: "h3"` switch is unreachable. - A cfg-longcat field group, shown for that backend: variant (bf16/fp8/int8/gguf), default quality, offload, segments, conditioning frames, context-parallel GPUs, repo checkout, venv, and the avatar fields (reference frame index, mask range, INT8, distilled sampling, KV-cache offload). Populated on open and read back on save. - All thirteen keys added to the whitelist, with a comment saying what happens without it. - The settings now DO something: the dispatch reads num_segments, num_cond_frames, offload_kv_cache and quality off the entry as defaults, with the request still winning. A knob that is saved but never read is worse than no knob. - docs/longcat-video.md: why the venv exists, the two prerequisites, the download repos and why file_pattern matters (~83 GB), the checkpoint layout, the stages and presets, the segment arithmetic, the VRAM/eviction/segment-boundary behaviour, and an explicit list of what is NOT done — including that no generation has been verified against real weights. 62 tests, parameterised over every field so a future control added without a whitelist entry fails rather than silently losing data. Not in this step: there is no known-model catalogue in the codebase to pre-register the LongCat repos into — adding a model is operator-driven by model_id — so the repos and their file patterns are documented instead of hardcoded. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 4. It also CORRECTS step 2, whose generate() was written against an assumed interface. The real one, read from upstream's demo scripts, differs in four ways that all mattered: - there is no single callable: generate_t2v(height, width, …), generate_i2v(image, resolution, …), generate_vc(video, num_cond_frames, use_kv_cache, offload_kv_cache, …) and generate_refine(stage1_video, spatial_refine_only, …) are separate methods, and t2v takes height/width where i2v and vc take a resolution NAME; - each returns [0]-indexed frames normalised to [0,1], not an object with .frames — upstream converts with (tensor * 255).clamp(0,255).to(uint8); - the distilled stage is a use_distill=True FLAG on the same method, not a LoRA to fuse by hand, so _apply_stage_lora is gone; - the 50-step passes take a negative_prompt and the distilled one does not (its guidance is 1.0, so it would do nothing). On top of that: - The segment loop is upstream's: a call emits num_frames but its first num_cond_frames re-render the previous tail, so only the remainder is new and the fresh frames become the next call's conditioning. torch_gc() between segments, as upstream does. - Modes map onto the tasks: t2v, i2v/ti2v -> i2v, extend/vc/v2v -> vc. Continuation reuses the EXISTING cond_frames field, so a client that already chains clips through the VACE path needs no new vocabulary — and gets native pretrained continuation instead, which is what stops the colour drift chained conditioning suffers. - Long video: num_segments or duration_seconds, and the SERVER owns the loop. The response carries the tail a caller needs to continue by hand if it prefers to. - The segment boundary is now a real handover point. POST /yield asks a generation to stop at the next boundary; the releaser calls it before waiting, so an eviction arriving mid-request costs one segment instead of the whole generation, and the request still returns — shorter, with a warning that reaches the caller — rather than failing. Unsupported modes (the audio-driven avatar tasks) are still refused with a 400 rather than downgraded to t2v, which would read as a quality bug rather than a gap. 23 new tests pinning the upstream call contract so it cannot drift back, plus the segment arithmetic and the yield behaviour. Still no real generation: that needs the venv, the repo checkout and the ~83 GB checkpoint. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 3: make LongCat a citizen of the memory and swap machinery rather than a model that merely runs. Three gaps had to close. 1. THE ESTIMATE. _get_model_used_vram_gb had no branch for a `longcat:` key, so it fell through to scanning the checkpoint — ~83 GB across variants, which would have asked the manager to evict the whole box on every request. Added beside the ds4/colibri/vllm early returns: measured if known, else the starting estimate. 2. THE MEASUREMENT. No official VRAM figure exists for LongCat and the real peak moves with the variant, the stage and the offload mode, so the constant is a starting point and the service reports what it actually reached. It has to come from inside the worker: the pipeline loads lazily on the first request, so a host-side free-VRAM delta would also be counting whatever else was loading on the card. torch's max_memory_reserved is this process's own arenas — what the card genuinely cannot give to anything else. The HIGH-WATER mark is kept, because a draft run touches far less than a 720p refinement and reserving the smaller figure would under-evict later, and it is keyed per config_id/variant so an fp8 sibling cannot inherit a bf16 measurement. 3. GIVING THE CARD BACK. register_external_vram_releaser with the documented fn(needed_gb) -> float contract (not voice_clone's zero-argument one, which raises TypeError inside eviction and is therefore never reclaimed). The drain matters more here than anywhere: acquire_stt_backend POPS the model pool, so _is_key_busy is permanently False for this model and Pass 1 of release_idle_vram would tear the subprocess down mid-generation and lose a multi-minute request. The releaser waits for in-flight work, bounded by longcat.evict_drain_timeout_s (300 s). Also: video can now take a slot in the front per-model queue. Every acquire site was guarded by `_task_kind(path) == "text"`, so max_instances on a video model did nothing at the front and nothing bounded concurrent generations but the engine and the swap gate. _queues_at_front() enables it for video when — and only when — the operator has set max_instances: doing it unconditionally would change behaviour for every existing video model and start returning 503 at queue_max_size, which is not something to impose on deployments that never asked. Applied at all four sites (direct, keepalive, both brokered), and the `is_text` local it replaced was left dead, so it is gone. 21 tests, including the one that matters most: an eviction arriving mid-generation waits and the request still completes. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 2: text-to-video through LongCat-Video, single GPU, bf16. The engine exists now; i2v, continuation/long-video and the avatar variants are the next increment and are REFUSED with a 400 rather than silently downgraded. Step 1 missed a prerequisite, fixed here: LongCatVideoPipeline lives in the repo's own `longcat_video` package, not on PyPI (`from longcat_video.pipeline_longcat_video import LongCatVideoPipeline`), and it vendors its own Wan VAE and scheduler. So the repo SOURCE is bundled like Wav2Lip/SadTalker — code only, no weights — and the service prepends it to sys.path. - tools/longcat_common.py: the contracts both sides share, stdlib-only so it runs on 3.10 and 3.13 alike and is loaded BY PATH from each. Checkpoint layout (tokenizer/text_encoder/vae/scheduler/dit + lora/*.safetensors), the stage table, the presets, and the segment arithmetic — 93 frames per call of which 13 re-render the previous tail, so a segment ADDS 80, which is what makes upstream's "11 segments for a 1-minute video" come out at 59.5 s. Getting that wrong silently truncates long output. - tools/longcat_service.py: runs IN the 3.10 venv, serves /health /progress /generate /unload. Fuses the right LoRA per stage (stages 2 and 3 ARE their LoRAs — the distilled pass is 16 steps because of cfg_step_lora), and refuses 'refinement' with no earlier frames instead of generating something else. - codai/api/longcat_worker.py: venv/source resolution, process lifecycle, and VRAM registration through acquire_stt_backend so eviction reclaims it by stopping the subprocess — the only way to be sure a 13.6B DiT's pages are gone. ensure_built REFUSES by default with the exact manual command, because auto_build pulls torch 2.6 plus its CUDA runtime; the venv probe asserts 3.10/2.6/4.41 rather than a bare `import torch`, since the image rewrites pyvenv.cfg and a mis-pointed venv would import the main venv's torch 2.11. service_key includes config_id/variant: sibling configs on one path are DIFFERENT residencies and must not share a service or a measured footprint. - Detection honours the backend pin, then alias/served-name/path. The dispatch branch in video.py unwraps `_raw_cfg` first, because build_runtime_kwargs emits a FIXED key list for video models with no backend, alias or path in it — which is exactly why H3's documented `backend: "h3"` switch is dead. Not inherited. - The worker path reports progress and is cancellable: the service exposes /progress and the engine polls it onto the existing bar. H3's worker path has neither. - VideoGenerationRequest gains `quality` (draft|fast|best) and `stage` (base|distill|refinement), the preset and resumable-per-stage halves of the staging requirement. 31 tests. NOT covered, and so not claimed: a real generation — that needs the venv, the repo checkout and the ~83 GB checkpoint. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 1 of the LongCat-Video integration: provision the runtime. No engine code yet. LongCat is the strictest isolation in the tree — it pins Python 3.10, torch 2.6+cu124, transformers 4.41 and numpy 1.26 against the main venv's 3.13 / 2.11+cu130 / 5.x / 2.4, four simultaneous conflicts, plus opencv-python against the bundled opencv-contrib. Installing it into the main venv would walk transformers and numpy backwards and break the stack, the failure already recorded for parler-tts. Crucially, coderai cannot build this venv itself: every venv bootstrap in codai/ uses `sys.executable -m venv`, which is 3.13. The only 3.10 in the tree is the standalone interpreter hand-built for the lip-sync tools and bundled as py310/, so LongCat rides that same interpreter in a venv of its own (its torch differs, so they cannot share one). - requirements-longcat.txt: upstream's pins off the cu124 index. flash-attn is deliberately NOT pinned — it is a source build needing ninja and a CUDA toolchain, and upstream accepts xformers, which ships a wheel; longcat.attention = "flash" is the opt-in. streamlit (upstream's demo UI) is left out. - build_oci_image.sh: declares LONGCAT_VENV and bundles it. The py3.10 interpreter is now discovered from EITHER py3.10 venv's pyvenv.cfg — it was read only from the lip-sync one, so a host with LongCat and no lip-sync venv would have bundled a venv with no interpreter to run it. - Dockerfile.oci-venv: copies longcat_venv and the pyvenv.cfg/bin-symlink rewrite now loops over both venvs. A bundled venv still carries the build host's `home =`, so without that its bin/python points at an interpreter the image does not have. - smoke_test_services.sh: asserts 3.10 + torch 2.6 + transformers 4.41, not just that torch imports — a venv re-pointed at the WRONG interpreter passes a bare import while being useless. - LongcatConfig (enabled, service_url, venv, python, auto_build, host, port, gpu, attention, ready_timeout, extra_args, extra_env), attached to Config, serialised by save_config, reconstructed in load() and exposed/parsed by the admin settings API. auto_build ships OFF: it pulls torch 2.6 plus its CUDA runtime. 16 tests. The config round-trip one caught a real hole: load() reconstructs each section explicitly, so without its line the operator could set LongCat up in the interface and lose it on the next restart. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod asked for surya or olmOCR was downgraded to docTR UNCONDITIONALLY. The reasoning was sound — both are VLMs needing a model server, locally they ride coderai's own vLLM engine, and a pod image will not get one (that venv alone is ~10.5 GB against a ~12 GB ceiling, and it would be a second torch beside the one docTR and Surya already need) — but the conclusion was too broad. In their SERVED modes these engines only speak HTTP, and PIL and requests are already in the image. So a pod can run them whenever the operator has pointed them at a reachable endpoint, including at a vLLM pod, which is the composition to prefer over a fat image. - _ocr_external_server(engine) decides it: olmocr_serve="server" + olmocr_server_url, or surya_serve="llamacpp" + surya_server_url. The local-only modes ("vllm", "model") still downgrade, and the log now names the setting to fix instead of only the symptom. - The mode, URL and model ids are seeded to the pod. config.py gained the env overrides for surya_serve/surya_server_url and fallback_engines; only olmOCR's existed. - fallback_engines is pinned to the engines the image actually carries: "auto" would hand a failed page to an engine with no server on that pod, which the fallback added in 0.2.28 made newly reachable. - Engine selection is alias-aware (`alias or served or path`, the convention used throughout the manager). An entry aliased "olmocr" over an HF repo path was read as an unknown engine and silently downgraded. Aliases are how one model carries several configs and how several models share an endpoint, so this path has to honour them. - The licence decision now survives a downgrade: `wanted` was reassigned to "doctr" before the `wanted == "surya"` check, making that branch's own documented intent ("asking for surya by name IS the decision") unreachable. olmOCR needs no pip dependency anywhere — it imports only PIL and requests — which is why nothing flagged its absence from the profiles; ocr.txt now records that and why a pod has no vLLM. 19 new tests; 278 pass across the OCR, front-gate, vLLM-share, cluster, remote-execution, placement and routing suites. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Escalation alone only ratchets upward, so an opening bid that is too generous was permanent: 0.60 of a 24 GB card for an engine that needs 0.25 is 8 GB no other model could have. A successful boot states how much of the budget was left for the cache, and `footprint = budget - cache` is the only direct measurement of an engine's real cost there is — so it is now read back and used. - _pump_logs captures "Available KV cache memory: N GiB" as it STREAMS. By the time the server answers its health check that line has scrolled out of the 80-line tail behind vLLM's route listing, so it cannot be recovered afterwards. - On a boot with clear slack (> 1 GiB beyond the ~2 GiB target), a smaller share is computed from the measured footprint and recorded for the next start. - The learned store is now a record per key, {"ok": booted, "failed": too small}, and the downward half may never propose a share at or below "failed" + a step. That floor is what stops it oscillating between a budget that works and one that does not. - Over-provisioning now costs one boot instead of the life of the process, and the flat float format the previous version wrote still loads. 6 more tests (shrinking, the tight-fit no-op, the anti-oscillation floor, a missing cache figure, the old format, and the footprint round-trip): 16 in the file, 255 across the OCR/gate/cluster/remote-execution/placement files together. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The 0.60 shipped in 0.2.26 was measured from one model on one card, which is the same mistake as the 0.35 before it: a footprint is not knowable up front. It depends on the model, the card's size, the context and the multimodal limits vLLM profiles for — and the 0.2.26 boot proved the point, since capping max_num_batched_tokens moved the footprint only 11.13 -> 10.88 GiB (all of it CUDA graphs, 0.30 -> 0.05) while the ~9.4 GiB peak stayed put: vLLM sizes surya-ocr-2's encoder cache at 114,688 tokens and profiles it with one VIDEO item at maximum feature size. So the share is now an opening bid and the launcher measures the rest: - ensure_service escalates on the KV-exhaustion signature (_KV_EXHAUSTED: "No available memory for the cache blocks", a negative "Available KV cache memory", or vLLM's free-memory complaint), +0.12 per attempt to at most 4 attempts, capped at the 0.92 ceiling, asking the evictor for the larger prelaunch figure before each retry. - An unrelated failure (missing model, broken venv) is NOT retried — escalating past it would burn minutes of boot per attempt for nothing. - What booted is persisted to ~/.coderai/vllm_gmu_learned.json keyed by model|card-total|ctx|max_num_batched_tokens, so later starts skip rediscovery, a different model/card/limit set discovers its own, and a configured share LOWER than what is known to work is ignored rather than reintroducing a dead budget. - A read-only config dir costs rediscovery, not a failure. The multimodal limits are deliberately NOT capped by default: doing so would stop that instance serving video at all, and the point is that any request type keeps working. docs/ocr.md records the measurement and how to opt in via vllm.extra_args. 10 new tests covering escalation, the step, the ceiling, non-retryable failures, persistence, scoping and an unwritable store. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live measurement on the 24 GB 3090 settled what the 0.35 share could never do. Two boots of surya-ocr-2: a 9.57 GiB budget reported `Available KV cache memory: -1.54 GiB`, a 14.31 GiB budget reported `+3.18 GiB` — a fixed footprint of 11.11 and 11.13 GiB, i.e. ~11.1 GiB (weights 1.37 + CUDA graphs 0.30 + ~9.4 GiB activation peak) before a single KV block. 0.35 of this card is 8.4 GiB: under the floor, so it could not start no matter what else was resident, and the 06:41 boot that finally succeeded did so only because 6.1 GB held by other models inflated the old add-back to an absolute 0.604. Those same two data points show vLLM charges only its OWN allocations against total × gmu — if other residents counted, they would imply activation peaks of 8.3 and 3.6 GiB rather than one consistent 11.1. So: - effective_gmu no longer adds back what others hold. It made the budget depend on unrelated residents and asked for share*total for this instance while `used` stayed held by someone else — at 0.60 that would have been 20+ GiB of a 24 GB card. The evict-first pass (prelaunch_free_gb) is what clears the card, which is the honest mechanism; the log line now states the share, its GB and what else is resident. - vlm_gpu_memory_utilization default 0.35 -> 0.60 (14.4 GiB, ~3 GiB of KV cache). - New ocr.vlm_max_num_batched_tokens (4096), vlm_max_num_seqs (16) and vlm_max_model_len (0 = inherit), plumbed through ensure_service/_launch_cmd via base.vlm_serve_limits(). This is the better lever: the 9.4 GiB peak is profiled for the batch size, and a VLM OCR engine otherwise inherits the LLM backend's ctx=18432. Chunked prefill is on, so capping tokens per batch shrinks the peak without shrinking the usable context — lower it enough and OCR fits beside an LLM instead of needing 60% of the card. 9 new tests (the caps, the launch command, the footprint floor); the share tests that asserted the add-back now assert its absence, with the reason. Co-Authored-By:
-
- 04 Oct, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr bypassed the gate entirely because it is not in the router's _INFERENCE_PATHS — and it should not be, since that predicate drives the per-model queue, pins, spillover and keepalive, none of which fit OCR. But OCR is GPU work: an OCR request could start a surya vLLM boot while an LLM was mid-generation on the same card, and vice versa. The gate is what turns that contention into turns. - is_gpu_inference_path() = is_inference_path() + /v1/ocr and /v1/ocr/batch, and that is what the swap gate keys on now. is_inference_path() is unchanged, so no other front machinery starts applying to OCR. - All OCR shares ONE gate owner ("ocr"): the OCR manager holds one set of pools and evicts between them itself, so paddle and surya are the same tenant for the purpose of who owns the card. It also means the front need not read the `engine` field out of a multipart body carrying a 100 MB PDF. - _computed_off_this_gpu(): the gate AND the local wait queue are skipped when the work is computed elsewhere — a cluster node (engine.remote), a RunPod pod (model backend), or an external API endpoint (every enabled OCR engine pointing off-box). Such a request holds no local VRAM, so a turn on this card's gate would add latency and swaps for nothing. Applied at all four queue sites (direct, keepalive, both brokered) and both gate sites. 15 new tests, the front proxy's first: path classification, the owner key, each off-box case, and that an OCR tenant and a model tenant alternate without starving each other. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Eviction now does the same clean swap the model manager does for a busy model (release_idle_vram waits for a request boundary rather than evicting mid-request). OcrManager._release_vram tore pools down immediately, which lost whatever page was being recognised — and with it the caller's document. Taking turns on one card means the handover waits for the handover point. - _Pool tracks in-flight pages (threading-guarded: incremented on the event loop, read from the manager's eviction thread) and drain_and_release() waits for them to finish before tearing down, bounded by ocr.evict_drain_timeout_s (default 60 s) so a wedged engine cannot block a model load forever. - A rebuild waits out an in-progress drain, instead of re-evicting the model that just took its turn and loading into VRAM the drain is about to free. - Evicted stays temporary: the next OCR request rebuilds and asks for room again, so OCR and models can alternate as many times as the traffic asks for. 4 new tests, including the two that matter: an in-flight page survives an eviction that arrives mid-recognition, and three consecutive OCR/model turns each rebuild and re-reserve. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
surya-on-vLLM was failing every ~20 minutes in production with "Available KV cache memory: -2.81 GiB", and the cause was the 0.35 share itself. vLLM's --gpu-memory-utilization is a fraction of the card's TOTAL memory and every other process' allocation counts against it, so with 3.6 GB of resident bge-m3 on a 24 GB 3090 a 0.35 budget left surya about 5 GB — less than its weights (1.37) plus activation peak at max_model_len=18432 plus CUDA graphs (0.30). Each attempt burned ~3 minutes of boot and profiling, then died, and the next OCR request started over. - vlm_gpu_memory_utilization is now the engine's OWN share and is translated at launch against what else holds the card (effective_gmu): 0.35 with 3.6 GB resident becomes 0.50, i.e. a 12 GB budget of which 8.4 GB is the engine's. Capped at 0.92 so the driver keeps its room. - The evict-first demand (prelaunch_free_gb) now covers the share plus that headroom, so a genuinely crowded card gets models evicted rather than the share clamped. - stop_service_for reports what the instance actually reserved, recorded at launch, instead of the caller's share — under-reporting sent the evictor hunting for VRAM that was already free. - Stopping the managed vLLM is no longer gated on the engine's current serve mode: a mode changed while a service was up leaked its share of the card until a restart. - ocr.fallback_engines (default "auto"): an engine that cannot serve hands the document to the other enabled engines instead of returning 503, fewest-ways-to-fail first. The response's "engine" field names whoever actually read the pages. This also covers a pool evicted mid-document — the failed pages are re-read rather than failing. 21 new tests; the two older vLLM-worker tests now patch the package attribute as well as sys.modules, without which the fake worker silently did not apply. Co-Authored-By:
-
- 02 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
`docker import` cannot express "empty list", so a base committed by the in-image upgrader (run_oci.sh passes --change "CMD []") came back as null in the flattened image and the equality check refused to retag — over a difference `docker run` cannot see. Normalise empty Entrypoint/Cmd to absent on both sides. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New engine `olmocr` — allenai/olmOCR-2 (Apache-2.0), a document VLM for the pages the detector-based engines read badly: old scans, maths, multi-column, dense tables. Equations as LaTeX, tables as HTML, reading order sorted; no bounding boxes, so paddle/surya stay the choice when something has to point at the page. It is served three ways (`ocr.olmocr_serve`): "model" (default) routes through coderai's OWN model manager, so a GGUF+mmproj or HF entry in models.json gets VRAM accounting, eviction and the thermal governor — and it works without CUDA; "vllm" gives it its own vLLM instance; "server" attaches to an OpenAI endpoint already running it. olmOCR's trained no-anchoring-v4 prompt is sent verbatim, the page is rendered at 1288 px, the YAML front matter lands in the new `OcrPage.meta` (omitted for engines that report none), and a page the model calls rotated is turned and asked once more. Four fixes the production Surya failures exposed — 169 consecutive failed boots between 08:00 and 09:35 today, every one against the same 11.34 of 23.56 GiB free against a 14.13 GiB demand, each costing the caller a 35 s hang: - The pool evicted for an engine AFTER loading it. For an engine that boots a server claiming a fixed share of the card, the load is what fails. Engines now declare `prelaunch_vram_gb()` and the pool frees that much first. - That reservation is the WHOLE POOL's, asked for once. Per instance it would have asked the manager to free 24 x 8.4 GB on this install's 24-instance surya pool, and an engine that reserved up front was then evicted for a second time, after its server had already taken the memory. - A failed build was retried on every single request. `ocr.build_retry_cooldown_s` (60 s) now fails fast with the reason the boot gave. - `ocr.vlm_gpu_memory_utilization` (default 0.35): the OCR VLM's own share of the card instead of the one `vllm.gpu_memory_utilization` asks for on behalf of an LLM. 0.35 of 24 GB fits surya-2 / olmOCR-2 and leaves the resident engines alone; 0.6 was more than the card had left. Per-instance footprints corrected to match: a surya worker in a served mode speaks only HTTP (0.05 GB, not 1 GB x 24), and an olmocr instance is one in-flight page, holding nothing of its own. Eviction in the other direction is unchanged and tested: the model manager's external releaser tears down the pools AND stops both VLM engines' vLLM services, and the pools rebuild on the next request. 18 new tests (no model, no GPU, no network). Docs: docs/ocr.md. Co-Authored-By:
-
- 30 Sep, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
Motus request 4. The most common reel format is a person talking to camera, and only this server can build it: composition synthesises the narration, so it holds the exact audio of each scene, and the lip-sync engines and identity-locked video generation are here too. - codai/compose/presenter.py: per scene — a face (a character profile's front reference, or a portrait), something to move in (a still, the caller's clip, or a short identity-locked shot generated through the ordinary video endpoint so its model config/acceleration/keyframe identity all apply), then an engine driven by THAT scene's narration, held to the scene's exact length and the canvas size. wav2lip and sadtalker are known by name; anything else installed under its own name can be selected by that name, so a newer engine needs no API change. engine=auto prefers a video-driven engine when there is a clip and a portrait engine otherwise. - Layouts in render.py: full (replaces the scene's visuals — the scene may then have none), pip (a masked circle/rounded/rect bubble with an optional border, inside the caption safe area) and split (stacked halves). With a presenter a scene's visuals are joined without an internal transition and composited as one clip; without one nothing changes. - Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so only the scenes that need it move (captions.py carries margin_v through). - Failure never loses a render: no engine, no face, a crash or a failed composite falls back to the scene's visuals (a gradient when there are none) and warns, naming the scene. New stage "presenter"; result.scenes[i].presenter says which engine and layout, or null. - Lip-sync is the only GPU step in a composition, so it takes a lease from the central scheduler instead of fighting a generation for the card. - POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a one-scene full-screen composition, for a character-page preview, capped at 15 s; GET /v1/video/presenter/engines tells a client what is installed so it can grey the feature out instead of meeting a 501. Both stay local in the capability gateway (a video pod has the shims but not this install's characters). - 400 for an unknown character / invalid layout, position, size, shape, motion, or a pip scene with no visuals; 501 with what to install when no engine is present. - docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed, compositing verified on real pixels). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Motus request 3. The character and voice extractors decoded media with their own helpers, which took base64, data: URIs and http(s) URLs but not the /v1/files/... paths compose accepts — so a client had to inline every source clip as base64 (a third larger) even when the server already held the file it had uploaded minutes earlier. - codai/compose/media.py: one resolver for every accepted reference — blob_path() for a sha256: upload id, local_path() for that or a /v1/files path (traversal-checked), fetch_bytes() for the endpoints that want bytes; a file we hold is read from disk, never fetched over the loopback. - characters._decode_source and voice_clone._decode_b64_or_url now delegate to it, which also gives POST /v1/characters and PATCH add_images the same references (they only base64-decoded before), and _decode_audio takes the stored extension from the CONTENT rather than the caller's claim (a data:audio/mpeg URI holding a wav was written as .mpeg). - uploads.find_blob() reuses media.blob_path so there is one lookup, not two. - A bare server path stays refused; a missing blob is a 400 naming the problem. - docs/video-compose.md lists the endpoints that share the resolver; tests/test_compose.py +2 (every reference form, and a character saved from an upload id). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`coderai-docker --upgrade` was unusable: git.nexlab.net's certificate expired on 2026-09-15, the clone failed, and the script gave up — although the same production branch sits on the public GitHub mirror. Nobody running the published image has a stake in that host, so an unreachable default is now a detour, not a dead end. - packaging/linux/launcher/coderai-upgrade: clone_from() per remote; a connection /TLS/DNS failure on the DEFAULT remote retries the mirror and logs both the reason and the source it ended up using. A repo the operator named (--upgrade-repo / CODERAI_UPGRADE_REPO) or an SSH-key clone is never silently replaced, and a ref that does not exist stays an error — papering over that with the mirror would install the wrong code. - Two bugs found while testing it, both silent: * `if ! cmd; then rc=$?` captured the status of the negation (0 on failure), so the fallback branch could never run; * reqs_content() ran under `set -o pipefail` with a `[[ -f ]] && grep` group, so a tree without packaging/common/requirements-oci.txt (or with no deps at all) aborted the upgrade one line after "upgrading x -> y", with no message. - tests/test_upgrade_launcher.py: the real script against local git repos — no network — for the healthy default, the fallback, the explicit repo, a missing ref and an up-to-date install. The certificate itself is on the host serving git.nexlab.net (94.23.108.54, not lisa) and still needs renewing there. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CoderAI could make every piece of a short video and nothing turned the pieces into a reel — assembly needs ffmpeg and two facts only the server has (how long the narration it just synthesised is, and when each word falls), so a detached client cannot do it. Requested by the Motus project over the HTTP API. - codai/api/compose.py: scenes -> one uploadable video. Scene length = narration + padding (min_duration floor), visuals split it evenly; async job with stage/progress/message/warnings, cancel (the ffmpeg child is killed), a queue that reports its position, and a blocking form for tests. Narration, music and word timings come from this server's own endpoints, called on a loop this module owns — never the request's loop, which may be closed before the render needs it. 400 names the offending scene/field; 501 when a sub-feature has no model (music.generate, timing=stt, TTS for narration) or ffmpeg is missing. - codai/compose/render.py: one normalised clip per visual (so a failed visual becomes a gradient and a warning instead of a lost render), concat or xfade that preserves the total duration, narration padded per scene and loudness-normalised, music looped/faded/ducked with sidechaincompress, burned captions + overlays with the final H.264 High/AAC/+faststart encode, thumbnail. - codai/compose/captions.py: word timings from STT aligned onto the CALLER's words (a mis-heard word never reaches the screen) or estimated; line breaking; ASS with six presets inside a 10%/6% safe area; SRT/VTT with the same text. - codai/compose/media.py: resolve a URL / /v1/files path / data URI / base64 (never a server path), probe with ffprobe, resolve fonts through fontconfig. - codai/api/uploads.py: POST /v1/files/upload, content-addressed by sha256 with GET /v1/files/blob/{hash} so a client can skip a re-upload; type decided by content, not by the client's claim. - compose and uploads are pinned local in the capability gateway (a pod has ffmpeg but not this install's models or files); both advertised in the capability document; caption fonts (Montserrat/Inter/Noto CJK) added to the three image builds. - docs/video-compose.md; tests/test_compose.py (30, real ffmpeg, no model/GPU). Co-Authored-By:
-
- 20 Sep, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus /metrics + per-key usage, node logs and re-push on return, GGUF row split, cosign-signed images - codai/cluster/discovery.py: _coderai._tcp announce + browse (zeroconf), HMAC fingerprint of cluster.token as membership; members auto-join as nodes, listed nodes with a blank key get the shared token; nodes and engines accept the cluster token (front /cluster/*, engine BearerAuth); env overrides for capability images (CODERAI_CLUSTER_TOKEN/DISCOVERY/NODE_NAME/ADVERTISE_HOST); coderai-docker --host-network - frontproxy/affinity.py + router prefer=: a conversation goes back to the engine/node that served it when the model is resident in several places; engine_for_model now picks the least busy holder - frontproxy/metrics.py: /metrics (requests per key/model/kind/status/engine, engines/nodes up, VRAM, in-flight, RPC, discovery, RunPod spend), usage tables on the Cluster page, server.metrics_public - node logs: /cluster/logs on the node, /admin/api/cluster/logs on the head, log viewer on the Cluster page; a node that comes back gets its assigned models pushed again - per-model split_mode=row (llama.cpp tensor parallel over local + RPC devices); vLLM TP may span nodes (documented) - packaging/sign-images.sh + packaging/cosign.pub: all 21 GHCR images signed - docs/cluster.md §§ discovery, 7 prefix routing, 9 operating, 10 accelerators evaluation; tests/test_cluster_ops.py (11) Co-Authored-By:
-
- 19 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-