- 10 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authoredAdds tools/character_studio.py, a demo app over the character-profile endpoints: extract a character from real photos/videos (POST /v1/characters/extract), review and prune the reference crops, then generate a brand-new video starring that person (POST /v1/video/generations with character_profiles + keyframe_identity). CLI core plus a `web` subcommand serving the same pipeline in the browser. The UI honours X-Forwarded-Prefix, so it works at the root or under a sub-path mount. Every generated mp4 gets its request body written alongside it, so the demo also doubles as a worked API example. Wired in as a bundled tool like the other UIs: supervisord program on port 7791 (autostart via CODERAI_TOOL_CHARACTER_STUDIO, on by default), entrypoint env defaults, run_oci.sh --enable/--disable-tool + --tool-arg names, nginx route at /character/, service smoke test, and the run banner/README URLs. Version bumped to 0.1.88 so the in-image upgrade path ships it. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
- 09 Sep, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
Ships the audio pipeline work (real music dub, demucs separation, DeepFilterNet restoration), the video fixes (camera_motion, dialog voice cloning, managed Wav2Lip, IP-Adapter identity conditioning + keyframe bridge), the huggingface_hub compat shim, and the documentation refresh. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The keyframe bridge dispatched its nested image request back through the front via execute_api_request. That deadlocks: by the time the engine handler runs, the front's GPU-swap gate has already made the VIDEO model the GPU owner for this request, so the nested request for a different (image) model queues behind the very request that is blocked waiting for it. Observed on a real Wan run: [gpu-swap] queued 'stabilityai/stable-diffusion-xl-base-1.0' behind owner 'wan-ai/wan2.2-i2v-a14b-diffusers' with the GPU sitting at 0% until the request was killed. Call codai.api.images.create_image_generation directly instead. The engine already holds the GPU for this request, and its own model manager handles the load/evict — which is also what we want, since the keyframe must be rendered before the video weights land. Verified end to end on real hardware (Wan2.2-I2V-A14B, 4-bit, model-CPU-offload): t2v request + character_profiles -> "rendering IP-Adapter keyframe" -> "IP-Adapter loaded" + 72 modules aligned to bf16 -> keyframe in 10 steps -> "running ti2v from it" -> Wan loaded (24.45 GB VRAM measured) -> HTTP 200 in 1642s with a real 480x480 h264 video, 17 frames, 2.125s, motion present between first and last frame. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Verified end to end on real hardware: same prompt, same seed, mean absolute pixel difference 46/255 between a generation with a character profile and one without. Before this, that difference was 0 — the profile changed nothing. IP-Adapter had never once loaded successfully. Four separate blockers, each hidden behind the previous one, all found by running it rather than reading it: 1. Attention slicing (71f88d14). diffusers' adapter loader rebuilds every attention processor; re-instantiating SlicedAttnProcessor without slice_size raised, the failure was caught and logged as a warning, and generation continued without conditioning. Slicing is now off across the load. 2. Quantized dtype cast. _load_ip_adapter_weights() ends with self.to(dtype=self.dtype) on the denoiser — a NO-OP cast to the dtype it already has — but bitsandbytes refuses any dtype argument on a quantized model. Tolerated for the duration of the load. 3. That tolerance then broke the forward pass: the suppressed cast was what brought the NEWLY added IP-Adapter modules up to the model dtype, so they stayed fp16 against a bf16 pipeline ("expected mat1 and mat2 to have the same dtype: BFloat16 != Half"). We now perform that cast ourselves on the non-quantized modules only — 72 of them, confirmed in the log. 4. Reference-image shape. A character profile carries several images, but diffusers matches the OUTER list against the number of ADAPTERS, not images: "Got 4 images and 1 IP Adapters". Multiple images for one adapter must be nested one level deeper. New shared helper ip_adapter_image_arg(), used by both the image and video paths. Also fixes a real production bug found while testing: pipelines are cached, and once an adapter is attached the UNet config demands image_embeds forever after, so the NEXT request without character references 500s with "...encoder_hid_dim_type set to 'ip_image_proj' which requires the keyword argument image_embeds". The adapter is now detached when a request doesn't need it (with slicing kept off across the unload too, which trips the same constructor). Verified lifecycle on one cached pipeline: with-profile -> without -> with again, all 200, and the two with-profile results are bit-identical (0.0 diff), so unload/reload restores conditioning exactly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by actually running it, not by reading it. Every IP-Adapter load was failing with SlicedAttnProcessor.__init__() missing 1 required positional argument: 'slice_size' and _ensure_ip_adapter_loaded() caught it, logged a warning and returned False — so character_profiles fell back to the prompt hint on the ONE path that was supposed to have real identity conditioning. The image came back looking plausible, which is why it was never noticed. Cause: we enable attention slicing for VRAM headroom, and diffusers' adapter loader rebuilds every attention processor; re-instantiating a SlicedAttnProcessor without its required slice_size raises. Slicing is now disabled for the load and left off while the adapter is in use (re-enabling would swap the IP-Adapter processors straight back out), and restored if the load fails anyway. Verified: the SlicedAttnProcessor error is gone. NOT yet working — the next blocker, now visible underneath: on a QUANTIZED pipeline the load fails with "Casting a quantized model to a new dtype is unsupported". So IP-Adapter still cannot attach to a 4-bit image model, and identity conditioning needs an unquantized image model (or a follow-up that loads the image encoder with an explicit dtype). Do not describe character identity as working end to end until that is resolved. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading the IP-Adapter (958d2e09) fixed the mechanism, but only AnimateDiff-class pipelines have one. On Wan / CogVideoX / LTX / SVD — the models people actually use — character_profiles still did nothing except prepend the profile name to the prompt, so identity was not working where it mattered. The image endpoint does have real IP-Adapter. So when a video model can't condition on references itself, render the first frame there against the same profiles and run the video as ti2v from that keyframe. This is the recipe tools/videogen.py already drives by hand, now done server-side. It runs BEFORE the video pipeline is loaded, so an image model and a video model are never resident at once, and it dispatches through the front (execute_api_request) so the image model may live on a different engine. New request fields: keyframe_identity (auto | always | never), keyframe_model, keyframe_steps. 'auto' skips the bridge when the model has native IP-Adapter or a first frame was already supplied. If the keyframe can't be rendered the request still proceeds — identity degrades to the prompt hint plus any per-character LoRA and the reason is reported in the response warnings — unless the caller asked for 'always', which is an explicit requirement and so returns 502 rather than silently producing a stranger. Verified offline across every branch: bridge fires for Wan and rewrites to ti2v with the keyframe; skipped for AnimateDiff; forced by 'always'; disabled by 'never'; no-op with no profiles or an existing first frame; degrades with a warning on image-endpoint failure and raises 502 under 'always'. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Correction to the previous commit. The OCI image already ships a complete lip-sync integration — packaging/linux/launcher/{wav2lip,sadtalker} shims on PATH, a dedicated lipsync_venv, baked Wav2Lip AND SadTalker source, first-use checkpoint download and a smoke test. The claim that the binary "essentially never exists" was wrong for the packaged deployment; _apply_lipsync() uses those shims and never reaches the managed path there. The managed install is therefore only for from-source installs (build.sh + venv_all), where nothing is on PATH and lip sync did silently degrade. More importantly it had claimed CODERAI_WAV2LIP_DIR — the same variable the shim uses for its writable working copy, seeded from the baked read-only source and run against the lipsync venv's pinned deps. Applying our numpy/librosa/torch rewrites to that tree would have corrupted a working packaged install. The managed copy now lives under CODERAI_WAV2LIP_MANAGED_DIR (~/.coderai/wav2lip-managed), whose basename also cannot collide with the shim's ~/.coderai/Wav2Lip on a case-insensitive filesystem. Re-verified after the move: fetch/patch/run still produces a valid synced mp4. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The last two gaps from the person-to-scene audit. IP-Adapter on the video path never worked. _apply_character_refs() set kw['ip_adapter_image'] without ever calling load_ip_adapter(), so a pipeline that does accept the kwarg (AnimateDiff) failed inside diffusers rather than transferring identity, and it also passed ip_adapter_scale as a call kwarg, which no pipeline accepts — a second latent TypeError. It now reuses the image path's _ensure_ip_adapter_loaded(), sets the scale through set_ip_adapter_scale(), and returns whether conditioning is actually active. Pipelines with no IP-Adapter at all (Wan/CogVideoX/LTX/SVD) now say so instead of looking like they worked. Lip sync required a 'wav2lip' or 'sadtalker' executable on PATH — a file that essentially never exists, since Wav2Lip ships neither a pip package nor a binary and its original checkpoint links are dead. codai/api/lipsync.py now owns the lifecycle the way ds4_worker owns ds4: it fetches code AND weights from the camenduru/Wav2Lip mirror on first use, patches what no longer runs on modern numpy/librosa/torch, creates the scratch dirs inference.py expects, and drives it as a subprocess. An operator-installed binary still takes precedence. Each patched file is compiled before being kept, so a bad rewrite is caught at patch time and reverted rather than surfacing as a SyntaxError from a subprocess (which is exactly how the first attempt failed: two librosa patterns compounded into 'sr=sr='). Lip-sync fallback is no longer silent. The reason travels up a per-request warnings list — not module state, which would cross-talk between concurrent requests — and appears as a `warnings` field on the video response, so a client can tell a synchronised result from a plain audio mux. Verified offline (live container untouched): Wav2Lip downloads, patches, and lip-syncs a real face to audio, producing a valid 768x512 h264+aac mp4 that differs from the source; the second run reuses the install with no download; the fallback populates warnings; IP-Adapter loads its weights and sets the scale for a supporting pipeline and cleanly declines for a Wan-like one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three defects found while auditing the 'person from a video -> scene with their voice' path. 1. camera_motion returned 500 on every request that used it. _apply_camera_motion() set kw['camera_motion'] unconditionally and nothing ever popped it, so pipe(**kw) raised TypeError — no diffusers video pipeline accepts that kwarg (the 'CogVideoX supports it natively' comment was wrong). tools/videogen.py sends the field, so that path was dead. Now the kwarg is passed only when the pipeline signature really takes it, otherwise the motion is folded into the prompt, which is the only conditioning these models have. 2. dialogs[].voice never cloned the voice it named. _generate_tts_for_line() loaded the profile's reference audio and then used it purely as a flag to force a generic voice (en-US-JennyNeural / af_sky) — F5-TTS was never invoked, so the selected identity was silently discarded. It now clones with F5-TTS when the profile has reference audio AND a transcript, and says why in the log when it falls back. 3. seed-vc and F5-TTS could not load their BigVGAN vocoder. huggingface_hub 1.x stopped forwarding proxies/resume_download to a subclass _from_pretrained; BigVGAN still declares them as required keyword-only args, so loading died with 'missing 2 required keyword-only arguments'. Pinning hub below 1.0 is not possible (transformers 5.x needs the new one), so codai/api/hub_compat.py supplies the dropped arguments for exactly the subclasses whose signature still requires them. This also un-breaks /v1/audio/convert. Also fixed on the way: - edge-tts never ran: get_event_loop().run_until_complete() inside an executor thread raises RuntimeError. In _generate_tts_for_line it was swallowed; in _generate_tts only ImportError was caught, so it propagated out of /v1/video/dub. Both now use asyncio.run() and verify the output file. - music dub adopted a failed Seed-VC result: converted_path was assigned before the write succeeded, so a near-empty conversion got remixed instead of falling back to the F5-TTS vocal. The result is now validated before adoption. Verified offline (live container untouched): camera_motion passes through only to a supporting pipeline and otherwise reaches the prompt; a dialog line clones with the profile's reference audio and transcript and refuses to clone without one; seed-vc loads BigVGAN and converts; the music dub falls back to the full vocal and reports the reason. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three related placeholders were shipping as working features: - audio_stems.separate_with_provider() and audio_clean.restore_with_provider() were one-line stubs that unconditionally raised 501, so /v1/audio/stems and /v1/audio/cleanup only ever worked in ffmpeg fallback_mode despite demucs and deepfilternet being declared dependencies and probed by detect_audio_backends. - run_full_music_dub() ran only the STT step and returned hardcoded paths (vocals.wav, final_mix.wav) plus 'status': 'placeholder' for the other four, while the endpoint reported 'status': 'available'. Now: - separate_with_provider drives demucs (demucs.api, CLI fallback), keeps the vocals stem and sums drums+bass+other into an instrumental bed for vocals-instrumental; 4-stem passes the four through. - restore_with_provider runs DeepFilterNet (VoiceFixer fallback) and then applies the hum/click/loudness DSP with ffmpeg, so those options are no longer lost. - audio-music-dub is a real six-stage chain: separate -> transcribe the ISOLATED vocal -> adapt lyrics (LLM prompted for singable output with matching syllable count, argostranslate fallback) -> re-sing with F5-TTS cloned from the original vocal -> optional Seed-VC f0-conditioned singing pass -> ffmpeg remix + limiter. New request fields: text_model, voice_name/ref_text, sing_convert, diffusion_steps, pitch_shift, speed, seed, fallback_mode. Missing optional deps now degrade to a reported 'skipped'/'failed' step with a reason and a top-level status of 'complete' vs 'degraded' plus warnings[], rather than silently producing placeholders. 501s name the package to install. Verified offline with a fake-model harness: the chain produces a real RIFF wav mix end to end, feeds the vocal stem (not the mix) to STT, and correctly reports degraded when seed-vc is unavailable. Co-Authored-By:
-
- 07 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
The README had not been touched since the OCR/multi-engine work in August and described roughly half of what the server does. Add the missing subsystems and correct the ones that overstated the code: - Inference engines: the per-model backend matrix (transformers/gguf/ds4/ colibri/k3/kt/vllm/runpod) and how one is selected. - RunPod: feature summary plus a full docs/runpod.md — pods vs serverless, per-model config keys, cold start, GPU fallback, stuck-boot retry, budgets and the ledger, the stale-pod reaper, spillover (with its real scope), observability and RunPod-only deployments. - STT backends, per-model languages/translation, word timestamps, diarization and enrolled speaker recognition; embeddings families (BGE-M3 multi-vector, GME-Qwen2-VL, GeoCLIP, VPR) and /v1/rerank; the OCR subsystem and its schema store; LoRA training. - API tables for /v1/ocr*, /v1/rerank, /v1/loras*, the audio/speaker routes and the two missing pipelines. - models.json reference incl. multi-config (config_id/alias), the front/engine split, optional-subsystem requirements files, and a docs/ index. Honesty fixes: /v1/pipelines/audio-music-dub is flagged as incomplete (only the transcription step runs); spillover is documented as serverless-target / direct-path only. Co-Authored-By:
-
- 06 Sep, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
For a coderai instance that only orchestrates RunPod (no local models), the node that hosts the proxy must advertise the runpod capability. The no-GPU path already builds its engine with backend "auto" (which has it), but an explicit engine_specs backend:"cpu" and the unknown-backend fallback did not — those would 503. - _DEFAULT_CAPS gains "cpu": {transformers, gguf, k3, runpod}. - Unknown-backend fallback now {transformers, gguf, runpod} (runpod needs no local hardware, so every node can host it). Verified by simulating pick_engine for a runpod-pinned model on auto/cpu/nvidia nodes — all route instead of 503. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The 300s boot cap was a hard-coded constant; large models need much more time for the image pull + weight download. Both budgets are now per-model config: - boot_timeout_s (default 300) — until the pod exposes its port; a stuck machine is still dropped fast and the next candidate tried. - load_timeout_s (default 600) — until vLLM answers /v1/models; raise for big models. Parsed in RunpodModelConfig (min 30s), used by _provision_one in place of the class constants (which remain the fallback), whitelisted in api_model_configure, and exposed on the model page with hints. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live run showed get_pod_logs hitting HTTP 400 on {rest}/pods/{id}/logs?tail=N, so a failed boot fell back to the console link with no output. Now tries the plausible routes in order (no params, ?limit=, containerLogs) and keeps the first that answers, reporting the collected errors otherwise. The console deep-link remains the always-available path. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Live testing showed RunPod sometimes ACCEPTS a pod create but the machine never starts (port never exposed) — 2 of 4 pods hung this way, burning the full 900s wait. Provisioning is now resilient and diagnosable: - Boot failure retries on the NEXT ranked GPU candidate (MAX_PROVISION_ATTEMPTS=3) instead of failing the request; _create_with_fallback returns the next index so each attempt uses a different machine/type. - Timeouts split + tightened: port-exposed 300s (was 900 — a pod that hasn't exposed its port in 5min never will), vLLM /v1/models health 600s (model load). Cuts worst-case billing on a bad machine from ~15min to ~5min per attempt. - Self-diagnosing: on any boot failure the pod's container/vLLM log is fetched and printed to coderai's log before teardown (the user's #1 cause is a vLLM OOM / launch error, only visible in that log), plus the console link. - Pod id + console URL are now logged immediately after create (previously only at "ready", so a stuck pod was untraceable). - Booting pods appear in pods_status/stats page as state="provisioning" with their console link, so a slow boot is visible live. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live-tested against a real RunPod account (Secure RTX 4090, Qwen2.5-0.5B): provision → boot → vLLM /v1/models health → idle auto-teardown + ledger ($0.021) all confirmed. Serverless + spillover proxy paths confirmed (reach RunPod, correct URL/auth). Fixes/additions from that run: - FIX: pods mode posted to <proxy>/chat/completions (missing /v1) → 404. The pod's vLLM serves under /v1 (health check hit /v1/models=200), so RunpodBackend now appends /v1 to the pod proxy URL (serverless base already ends in /openai/v1). - Stale-pod REAPER (safety): runpod_worker.reap_orphans() lists account pods tagged coderai-<deployment_id>-* and terminates any NOT tracked by a live pool (orphans from a crash/restart) — never touches in-flight provisions or another deployment's pods. Runs every ~30s in the scaler loop AND once at startup; started on the primary engine (where pods are hosted) via start_runpod_maintenance(). New config.runpod.deployment_id (default "default"); pods named with the tag. - Multi-GPU capacity FALLBACK: _rank_gpus() returns ranked candidates and _create_with_fallback() tries the next on a RunPod "no instances available" capacity error instead of failing (hit a real A5000 community capacity miss). - Pod LOGS: RunpodClient.get_pod_logs() (best-effort REST) + pod_console_url deep link; admin GET /admin/api/runpod/pod-logs; RunPod stats page gets a per-pod "view logs" modal + console link. PodHandle/pods_status carry console_url. - Settings: deployment_id field. Built coderai:base=5cffe23d1d20. Relaunch to activate (running image had the /v1 bug + no reaper). After relaunch, a pods chat returns a real completion. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Completes the RunPod backend for testing. Pods pool (runpod_worker.py): - RunpodPodPool: provision via _select_gpu (cheaper=lowest $/hr, faster=SECURE+ throughput, across allowed pools, spot-aware) → create_pod → wait_ready → vLLM health-check; least-loaded per-request acquire, provision-on-demand up to max_pods; background scaler thread tears down idle pods to min after idle_timeout_s, keeps min warm, reaps. get_pod_pool registry + pods_status(). - Budget enforcement at provision: global $/hr rate cap + per-model & global cumulative spend caps (rolling period) via the ledger; refuses over budget. Cost ledger (runpod_ledger.py): persistent per-model + global spend, trailing rolling windows (hour/day/week/month), totals() for stats. Final pod cost booked on teardown; live pod cost added on top for enforcement/stats (no double count). Backend (backends/runpod.py): pods mode wired to the pool (per-request acquire/ release; async stream acquires off the event loop); serverless unchanged. Config adds global_cost_limit_usd + global_cost_period. Global-gate bypass (app.py _model_capacity): runpod models get an effectively unlimited LOCAL queue cap — remote GPUs aren't bound by the local concurrency gate; RunPod's max_pods/workers are the real limit. Spillover (app.py): a local model can burst to a RunPod SERVERLESS target when local can't serve it — front reverse-proxies to the serverless OpenAI URL, guarded to fall back to local on any error. Wired at the direct path for on_no_gpu (no engine) + on_concurrency_full (queue full). NOTE partial: broker path, streaming-concurrency, pods-target and on_local_error triggers are follow-ups. UI: dedicated /admin/runpod stats page (pods, live $/hr vs caps, spend by period + model) + nav link; Tasks page RunPod box; Settings split into category tabs (JS-only, IDs/save untouched) + global spend cap fields. Verified: imports in-image (8b16914ee24b), ledger + GPU selection (cheaper/faster) unit-tested, all templates Jinja-parse. Live RunPod calls (provision/serverless) still need an end-to-end test with the user's account. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
End-to-end serverless: a model tagged backend:runpod (mode:serverless) proxies OpenAI chat/completions to a RunPod serverless endpoint. Runs on a remote GPU, holds NO local weights. - codai/api/runpod_worker.py: RunpodModelConfig + parse_model_runpod (mode/pools/ criteria/gpu/price/scaling/cost/serverless), serverless_base_url + auth_headers. - codai/backends/runpod.py: RunpodBackend — thin OpenAI proxy (chat/stream, Bearer auth for serverless), uses_local_vram=False; pods/auto raise a clear "next phase" error for now. - manager.py: runpod in _ENGINE_BACKENDS + _runpod_name_claims (pin-only) + runpod_should_handle + load_model dispatch branch. _model_is_runpod() gate: set_default_model skips local download/cache for runpod-served models. - router.py + registry.py: runpod is an explicit-pin capability; added to every engine's default caps (no local VRAM → any node hosts the proxy; prefers primary). - admin routes: whitelist per-model `runpod` + `runpod_spillover` blocks; skip local-VRAM estimate for runpod models. - models.html: RunPod backend option + full per-model sub-form (mode w/ explainer, serverless endpoint, served model, pools/criteria/gpu/price/spot, min/max pods, idle timeout, image/disk/port/env, spend limit+period) + prefill/collect. Serverless verified wired in-image (ef1fd7a3802c). Pods pool/scaler, cost ledger, spillover routing, and stats UI are later phases. Co-Authored-By:
-
- 05 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
First slice of the RunPod remote-GPU backend (rent cloud GPUs, proxy inference). Dormant unless enabled — no request path yet. - config.py: RunpodConfig (enabled/api_key/cloud_type/api_base/rest_base/ serverless_base/default_gpu_type/data_center/global_max_hourly_usd); registered, loaded, serialized. get_active_runpod_config() in manager.py. - codai/api/runpod_client.py: GraphQL client — ping, list_gpu_types (+prices), pick_gpu (cheapest under a $/hr ceiling, VRAM filter, spot-aware), create_pod (on-demand + interruptible/spot), get_pod, stop/terminate, list_pods, wait_ready. API key via ?api_key=, masked in all error strings. - admin: settings GET/save runpod block (key masked, overwritten only on a real new value); GET /admin/api/runpod/gpu-types (key check + catalog) and /admin/api/runpod/status (pods + live $/hr). - settings.html: RunPod card (key/cloud/default-gpu/global $/hr cap/DC/advanced bases) + "Test key & list GPUs" button. Decisions locked: host in primary engine (Layer-2, zero local VRAM); default vLLM-OpenAI pod image; per-model $/hr + global budget; per-model allow_spot; mode pods|serverless per model. TODO (request-path phase): RunPod requests must bypass the GLOBAL concurrency gate only (QueueManager.max_parallel_requests). Client GraphQL field names written from docs — pending live verification with a key. Co-Authored-By:
-
- 02 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
whisper-server models (whisper0/1/2/3) relied on whisper.cpp's --convert, whose ffmpeg call deadlocks on mp4 (undrained pipe) and hangs forever, head-of-line blocking the single-threaded server so every later request queues (app reported jobs "stuck in queue"; found a --convert ffmpeg stuck 4h+). - WhisperServerManager.start(): drop the --convert flag. - add _to_wav_16k_mono(): ffmpeg via subprocess.run (drains stdout/stderr, no pipe deadlock) + 120s timeout; transcribe() converts before POST so whisper.cpp only ever receives clean 16k mono wav. Bad file fails fast. - keep_resident honored for whisper-server models (vram group membership). Verified on f595eb2a6ece after relaunch: whisper3 + mp4 -> 200 (was infinite hang); wav control + repeat mp4 return sub-second (no head-of-line stall). Co-Authored-By:
-
- 01 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
Contiguous windows dropped words at each ~28s seam (Whisper's weak edge context + words straddling the boundary). Now use overlapping windows (29s wide, 24s hop = 5s overlap), advancing by the hop and transcribing each full window, then de-duplicate the overlap by keeping only words that start at/after the last kept word's end, and rebuild the transcript from the de-duped words. Verified on an ~11min clip: 0 gaps >2s across all seams, continuous word timestamps to 673s, recovered the previously-dropped seam words. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- Long-form fix: the HF pipeline's internal word-timestamp stitching crashes at the ~30s chunk boundary for CrisperWhisper (IndexError, dim-2 index==size). The service now windows manually — pipeline built without chunk_length_s; audio read via soundfile, split into <=28s windows (each a single Whisper pass), per-window word/segment timestamps offset by window start and concatenated. Verified on an ~11min clip: 1877 words, continuous timestamps to 673s, HTTP 200. - Self-heal: _RemoteCrisperWhisperBackend + _RemoteNemoBackend re-ensure the worker and retry once on a ConnectionError, so a restarted/killed worker no longer leaves a stale cached service_url in acquire_stt_backend. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
CrisperWhisper does verbatim ASR with precise word timestamps, but its custom generation config crashes on the server's transformers 5.x. Run it in its own venv worker (pinned torch 2.5.1+cu124 + transformers 4.46.3 — last pre-5.x with working Whisper word timestamps and a py3.13 tokenizers wheel): - codai/api/crisperwhisper_worker.py + tools/crisperwhisper_service.py (return_timestamps="word" + CrisperWhisper pause redistribution). - requirements-crisperwhisper.txt. - stt_backends.py: `crisperwhisper` is its own family -> _RemoteCrisperWhisperBackend (HTTP to the worker); generic `whisper-hf` still runs in the shared venv (text-only default + graceful timestamp fallback); HF-ASR path now converts any container (mp4/m4a/webm) to 16k mono wav first and uses model-correct return_timestamps (Whisper True/"word", CTC "chunk"/"word"). - transcriptions.py: crisperwhisper dispatched as a GPU worker. - manager skip-download + admin backend option for whisper-hf/crisperwhisper. Verified: mp4 -> verbatim text + per-word timestamps, HTTP 200, on the 3090. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Load the wav2vec2/HF-ASR pipeline in FP16 on CUDA (~3GB -> ~1.5GB), FP32 on CPU, with a fallback to FP32 if a checkpoint rejects half precision; overridable via a `dtype` config field. Lower the wav2vec2 VRAM reservation default 3->2GB to match. Enables co-residency of the full transcription/diarization/speaker stack with a 14B LLM on one 24GB 3090 (measured ~14.5GB used with a gemma-4-14b @8k sibling). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/audio/speaker-embeddings gains window (seconds) + step (default = window): returns one embedding per sliding window, each tagged start/end, instead of a single whole-file embedding. Implemented in the pyannote worker's /embed via _crop_wav over the cached embedder; client + endpoint pass the params through; whole-file mode and enroll/identify/verify are unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Transcription (/v1/audio/transcriptions): - Route multipart uploads by peeked model (front _peek_model) so they land on the model's assigned engine instead of scattering by capability. - New pluggable STT backends (codai/api/stt_backends.py): wav2vec2 (HF), vosk (CPU dirs), NVIDIA NeMo-Canary (isolated venv worker) + faster-whisper/whisper. - Per-model language selection + validation; Canary source->target translation. - OpenAI timestamp_granularities[]=word: top-level words[] in verbose_json (vosk/wav2vec2/faster-whisper/canary). - VRAM-eviction tracking (acquire_stt_backend) + keep_resident co-residency for the small set; /v1/models surfaces languages + backends. Diarization (/v1/audio/diarization + diarize=true): - pyannote.audio isolated venv worker (codai/api/pyannote_worker.py, tools/pyannote_service.py) on the 3090; speaker-labeled segments; graceful degradation. Pinned torch/pyannote/hf_hub/matplotlib in requirements-pyannote.txt. Speaker embeddings + recognition: - /v1/audio/speaker-embeddings (ecapa via speechbrain, pyannote/wespeaker). - Enroll/list/delete + identify/verify (codai/api/speaker_registry.py); persistent voiceprint store; diarization identify=true relabels turns with enrolled names. New isolated-venv requirements: requirements-nemo.txt, requirements-pyannote.txt (+ vosk in requirements.txt). Co-Authored-By:
-
- 27 Aug, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
record_vram_delta persisted measured_n_gpu_layers from every load, including VRAM-CONTENDED ones (another model resident → llama.cpp fit fewer layers). Since _learned_n_gpu_layers reuses that as a hard cap for auto (-1) configs, one unlucky contended load pinned a model half on CPU permanently — it never recovered even when the card was later free. Observed on the gemma-4-26B-A4B 32k config (alias lisa-32k): a contended load ratcheted 31→20 GPU layers and stuck, spilling ~5.8 GB of weights + KV to CPU on a 3090 with 11 GB free. Fix: in record_vram_delta, never overwrite a HIGHER known-good measured_n_gpu_layers with a LOWER value from an offloaded (CPU-spilled) load. A genuinely-too-big model still settles at its free-card maximum on an uncontended load, so keeping the higher value is safe; worst case is one offload-retry on a later truly-contended load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Fixes "embedding request debug on radeon disappeared" — a real regression where the radeon engine stopped emitting ALL log output the instant it finished loading its GGUF embedders (~13s after launch), while still serving, so its FULL REQUEST DEBUG + thermal debug vanished with no trace. Two changes: 1) codai/api/log.py — the log_requests middleware now ALSO appends each FULL REQUEST DEBUG block, from the engine's own process, straight to <dir(CODERAI_LOG_FILE)>/requests.log via a single atomic O_APPEND write (tagged [ts][engine] from CODERAI_ENGINE_NAME). This survives the engine's stdout stream dying, so request bodies are captured regardless. Concurrent engines never interleave mid-block. 2) codai/frontproxy/engine_supervisor.py — _pump_logs no longer dies silently: a bad line can't kill the pump, and on stream end it prints "[pump] log stream ended (<stdout EOF|pump error>); engine still ALIVE/exited" so a vanishing engine log stream is diagnosable instead of looking like "debug turned off". Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Follow-up to the same-path multi-config runtime fix. Two more path-keyed spots broke a same-file sibling addressed by its alias (e.g. lisa-32k): 1) GGUF dispatch: _resolve_local_gguf matched entries by path/basename only, so a sibling addressed by alias wasn't recognized as GGUF and fell to the HF loader ("lisa-32k is not a valid model identifier"). Now also match the entry's alias. 2) Measured footprint cross-contamination: persist_model_field matched entries by path, so one config's measured_vram_gb/n_gpu_layers overwrote the sibling's — the small-ctx variant inherited the large-ctx footprint (34 GB) and still offloaded to CPU. persist_model_field now targets by config_id when given; record_vram_delta passes cfg['config_id']; _model_cfg carries config_id/ config_name so the config knows its own entry identity. Net: lisa (178K) and lisa-32k (32K) are now fully independent — distinct config, GGUF load from the shared file, and separate measured footprints. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The web UI stores sibling configs for one model file (distinct config_id, e.g. a smaller-context variant), and the front already routes by alias (_route_key: "two configs of the same model with distinct aliases can be assigned to different engines"). But engine startup keyed every text-model config by path (_model_id -> path), so a second same-path entry OVERWROTE the first — BOTH aliases then resolved to the last-registered config (verified: lisa and lisa-32k both got n_ctx=32768, silently changing the primary too). Fix: in main.py text-model registration, key the PRIMARY config by path (stays path-addressable) and each SIBLING (same path, already seen) by its ALIAS, so the two configs are distinct and each alias resolves to its own n_ctx. A sibling without an alias is skipped (can't be addressed). The GGUF loader already reads the file from cfg['path'], so an alias-keyed sibling still loads the right file. Verified: lisa->178000, lisa-32k->32768, same underlying .gguf. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The reranker is registered in embedding_models, so request_model(model,"rerank") was rejected ("is a embedding model and cannot be used for rerank generation"). Resolve with model_type="embedding" (its actual category) so validation passes; rerank.py still loads it as a cross-encoder (AutoModelForSequenceClassification), re-loading if a cached embedder object is found under the same key. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
bitsandbytes 0.49.2's Params4bit.__new__ has no **kwargs and rejects the _is_hf_initialized kwarg that transformers 5.12 passes during 4-bit weight init, so every load_in_4bit model crashed with: Params4bit.__new__() got an unexpected keyword argument '_is_hf_initialized' In production this turned a single default-model chat request (Qwen3.5-9B, load_in_4bit) into a ~2h internal 60-retry storm. bnb 0.50.2 adds **kwargs to Params4bit.__new__ (absorbs _is_hf_initialized); verified end-to-end: a transformers load_in_4bit load now succeeds on cuda:0 (torch 2.11/cu130), and a GPU 4-bit quantize/dequantize round-trips. Image rebuilt with bnb 0.50.2. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reranker (e.g. BAAI/bge-reranker-v2-m3): - new codai/api/rerank.py: /v1/rerank scores query
↔ document pairs with a native AutoModelForSequenceClassification cross-encoder (no new dep), cached + VRAM/thermal-managed via request_model like embeddings. Returns {index, relevance_score(sigmoid), document?} sorted desc, top_n honored. - router: /v1/rerank added to _INFERENCE_PATHS so the front routes it by model capability (transformers → nvidia engine), like /v1/embeddings. - app: include rerank_router. - capabilities: add `reranking` + `ocr` fields; detect 'rerank'/'cross-encoder' names as reranking. /v1/models discoverability: - list_models() now appends enabled OCR engines (paddle/doctr/surya) as type='ocr' entries (a node subsystem invoked at /v1/ocr?engine=<id>), so clients can discover them. - a reranker registered in models.json (embedding_models, capabilities ['reranking']) surfaces automatically via list_models, like bge-m3. Co-Authored-By:
-
- 26 Aug, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
The per-token Python loops that built the sparse {token_id: weight} dict and the colbert vector lists held the GIL between forward passes, starving the GPU (steady ~10-36% util with brief bursts). Replace them with GPU tensor ops: - sparse: relu(sparse_linear·h) with invalid positions (specials/padding) zeroed, then max-pooled per token id via a single scatter_reduce(amax) into a (B, vocab) matrix; per-row nonzero extraction is O(nnz), not O(L). - colbert: one C-level tolist per row over its valid tokens (vs per-token). - valid mask computed on-device via torch.isin. Output byte-matches the previous per-token logic (dense/sparse/colbert within 1e-4). Benefit grows with batch size, so larger client batches translate to GPU work instead of Python-loop time. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
hf_hub_download(local_files_only=True) intermittently failed to locate the cached sparse_linear.pt inside the engine process, so _is_bge_m3 returned False and bge-m3 silently degraded to the dense-only sentence-transformers path. _resolve_repo_file now falls back through try_to_load_from_cache and a direct filesystem glob of the HF cache (models--org--name/snapshots/*/file), which resolves reliably regardless of process context. Detection keys off the sparse head file rather than the model_type string. (Debug scaffolding removed.) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- import build_from_pretrained_kwargs inside _load_bge_m3 (it's a local import of _load_embedding_model, not module-level) — the NameError made every bge-m3 load fall over so the model silently degraded to the dense-only path. - cast the sparse/colbert heads to the base model's dtype (bf16) so the head matmul over the last hidden state doesn't dtype-mismatch. - requires_grad_(False) on the heads (inference only) — no autograd graph per request. - log the bge-m3 backend selection at load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add BAAI/bge-m3 hybrid embedding (dense + sparse lexical, optional colbert) without the FlagEmbedding dependency (which pins older transformers/peft and would conflict with the transformers 5.x stack): the sparse/colbert heads are plain Linear layers over XLM-RoBERTa's last hidden state, reproduced natively with transformers + torch already in the venv. - new 'bge-m3' embedding backend: detected by xlm-roberta model_type + a sparse_linear.pt head (so plain bge-large etc. stay on the dense ST path); loads the base model + sparse_linear.pt (+ colbert_linear.pt if present). - _bge_m3_encode: dense = L2-normalized CLS; sparse = max-pooled relu(sparse_linear·h) per token id (specials dropped) as {token_id: weight}; colbert = L2-normalized colbert_linear·h per content token. - /v1/embeddings extended: request.embedding_types selects any of dense|sparse|colbert (overrides the model-config default); response items gain sparse_embedding {indices, values} and colbert_embedding. Dense stays in the standard `embedding` field (base64 honored). Configurable default via the model's embedding_types config. - cleanup()/eviction branch moves the base model + heads off GPU. Co-Authored-By:
-
- 22 Aug, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
The Surya-2 engine serves its VLM via a managed vLLM subprocess (vllm_worker.ensure_service). That subprocess is not a manager-tracked model, and the OCR VRAM releaser only tore down the lightweight worker pools (killing the HTTP workers) — it never stopped the vLLM holding the bulk VRAM (gpu_memory_utilization × card). So on-request eviction saw the VRAM as used, found nothing it could free, and (with no CPU fallback) a docTR/Paddle load on the same card could OOM. - vllm_worker: add stop_service_for(cfg, model_path, served_name) (stop ONE service, return estimated GB freed) + is_running() helper. - OcrManager._release_vram now also stops the managed Surya-2 vLLM (_stop_surya_vllm) when surya_serve == "vllm", so the registered external releaser actually reclaims its VRAM. Next OCR request re-boots it via ensure_service. Targeted by (model_path, served_name) so an LLM vLLM instance is left alone. Co-Authored-By:
-
- 21 Aug, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
OCR ran outside the shared GPU governors: /v1/ocr(/batch) and run_ocr_step call OcrManager.ocr_document directly, bypassing request_model, so OCR neither waited on thermal cooldown nor evicted/released VRAM against LLM/diffusion models. - thermal: ocr_document now awaits thermal.wait_until_safe (via to_thread) before OCR-ing pages, so OCR honours the same cooldown/front-pause gate as every other GPU workload. - VRAM (both directions): * evict-before-build: when a pool loads its instances, ask the model manager to free size*per-instance-VRAM first (_evict_models_for_vram), so OCR contends on equal footing instead of OOMing. * releaser: OcrManager registers _release_vram(needed_gb) via register_external_vram_releaser, tearing down built pools so loading an LLM/diffusion model can reclaim OCR VRAM. Pools rebuild lazily next request. - engine cleanup() (SYNC, eviction-thread safe) + vram_gb() footprint per engine: docTR ~0.7 GB (frees model + empty_cache), subprocess ~2.5 GB (kills worker). _Pool tracks all instances and release_sync() tears them down. - configure() now tears down stale pools on reconfigure (prompt VRAM free instead of GC-deferred). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Each engine's status payload reported only running/queued/paused tasks, so finished generations on a non-primary engine (e.g. radeon, reached via the API) never got to the front and vanished from the Tasks page. Now every engine also reports its most recent terminal tasks (capped at 10, newest-first); the front's _merge_engine_tasks already merges non-primary e.tasks (tagged by engine), so the page shows the last-N per engine. Co-Authored-By:
-