- 16 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe HF token fix worked: pyannote got past the gate and failed one layer deeper, instantiating the pipeline. Reproduced by running the real service with a GPU: ModuleNotFoundError: No module named 'matplotlib'. pyannote's segmentation tasks import matplotlib.pyplot at module load; it is not declared, and 'import pyannote.audio' never reaches it, so the build's import check passed and the venv shipped broken. This is the fourth venv whose bare import check passed while the real load failed. A <profile>.venv-<name>.check script now runs at build time in the venv's own python and imports what the REAL load reaches — pipeline modules, model classes — without a GPU or a token, which a build has neither of. The pyannote check fails on the shipped image and passes with the fix; the XTTS check passes as-is, so that venv is genuinely complete. Also: every isolated-venv worker reported failures as the LAST FIVE LINES of the subprocess tail. A Python traceback's last five lines are frame headers — the exception is the final line, and cutting to five dropped it whenever the trace was deep. Ten workers said where a process died and never why. Fixed in all of them. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
coqui-tts imports a transformers 4.x symbol that 5.x removed. Unlike parler and the OCR engines it had never been isolated locally: _CoquiBackend did a plain 'from TTS.api import TTS' and gave up when it was missing. On a pod that meant an image with a verified, working TTS venv answered 'pip install coqui-tts' — the venv was there and nothing knew to use it. xtts_worker finds the venv the way the others do (explicit env var, the baked /opt/coderai/venvs/TTS, then per-user) and drives tools/xtts_service.py in it over stdin/stdout. XttsSubprocess wears the TTS object's shape — speakers, tts(**kwargs), synthesizer.output_sample_rate, and the TypeError the backend relies on to retry without speed — so synthesize() runs unchanged whether the library is in this process or next door. Verified inside the pushed image: the venv is found and the service answers from it. The fall-through is unit-tested with a fake worker, and the error when no venv exists names the places it looked. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
pyannote is gated. On a pod it loaded from its baked venv, reached from_pretrained, and stopped: 'the model is gated/unavailable'. The local install has HF_TOKEN in its environment; a pod was never sent one — every gated model was unservable remotely, not just this one. It is resolved the way pyannote_worker already resolves it (request, env, token file) and travels as pod ENVIRONMENT only. A test proves it reaches the pod and appears nowhere else in the plan — not the seed list the pod parses, not the catalogue, not the args. Fixing this exposed a precedence bug: at provision time the plan's env was applied OVER the model's own env, so a token a model was explicitly configured with was silently replaced by whatever this process happened to have. The model's own env wins now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The failure this prevents was observable, not hypothetical: a pod boots, pulls the image, starts downloading, fills its 40 GB disk and dies at load_timeout_s having paid for every minute. The size of a HuggingFace repo is one metadata call, so it is asked before the pod exists, and the disk is grown to fit — weights plus ~25 GB for the unpacked image and the download cache. It counts what a load will actually FETCH, not the whole repo. SDXL's repo is 77 GB, of which a diffusers load pulls 28: the rest is the same weights again as Flax .msgpack, legacy .bin and ONNX exports. Sizing to the repo would have rented 100 GB for a 28 GB model. An explicit container_disk_gb that is already large enough is respected. And with a network volume attached the weights land there, not on the container disk, so the disk stays at its configured size — which is the point of the volume. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod refuses images above a size line — measured: 10.1 GB boots, 14.5 GB is 'Exited by Runpod' two seconds after rent on every machine, no other reason given. Container disk (tried 80 GB), layer format and the full image config were each ruled out; size was the only difference. All five specialised images were above the line. They were above it for one reason: each was built on the full core, so it carried 7 GB of CUDA, torch and triton the main venv never used on that pod, beside the venv that did the work. The speaker image was 14.8 GB, of which pyannote's own venv was 5.6. The light core is coderai's Python dependencies with no GPU stack: 1.2 GB. codai.api.app imports with no torch present (verified first), so the main interpreter still boots the API, serves /healthz and /boot, and drives the subprocess worker. A profile marked <name>.light builds on it, its main venv carries only what the API layer needs, and the build FAILS if a torch reaches that venv — because that is exactly the leak that would put the image back over the line. speaker: 14,796 MB -> 7,077 MB, complete, nothing on a volume. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A dead pod produced three hundred seconds of 'still no port' and no reason at all, because the one Pod field that carries the reason was never queried. It now travels into the progress line and the failure message, so an EXITED pod says what RunPod says about it instead of leaving the cause to be guessed. The strict-mode removal in boot.sh was one such guess: the image boots cleanly here with a minimal environment and with a GPU exposed, so it does not reproduce the exit. This is how the real cause gets found. Co-Authored-By:
-
- 15 Sep, 2026 29 commits
-
-
Stefy Lanza (nextime / spora ) authored
A speaker pod reported EXITED with uptime 0s from its very first poll. The loop waited out the full budget on it — three machines in a row — printing 'image pull in progress', which it never was: wait_ready treated only TERMINATED and FAILED as terminal. EXITED past a short restart grace now fails fast and says what it means: the image pulled and the process died on boot. The budget it waited out was 300s, not the 900s set on the pool class: the dataclass default was never updated. Two numbers for one thing, and the smaller one silently won. There is one now. Why the process died: boot.sh ran under set -u, and every line before the exec is a diagnostic — it exists to describe the pod, and a diagnostic must never be able to stop the boot it describes. The image boots fine locally; on RunPod an environment difference tripped a strict-mode abort before uvicorn was reached. That is the best-supported explanation, not a proven one — RunPod publishes no log API to confirm it — so the script is now unable to abort before its exec under any environment, verified against a bare sh with nothing set. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 150 GB warning's call site in _provision_one used a name that did not exist in that scope. The helper was unit-tested and passed; every pod test then failed in two seconds with "name 'entry' is not defined" before renting anything. A helper can be correct while the line that calls it is not. The entry is looked up the way the rest of the pool does it. And since pyflakes sees exactly this class of bug, a test now runs it over the modules a pod boot goes through, so an undefined name in that path fails the suite instead of the first real request. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The venv built and then failed its import check with ModuleNotFoundError. Verified the whole spec in a throwaway container first this time: with click, 'from TTS.api import TTS' works on torch 2.5.1+cu124 with transformers 4.46.3 — the stack this image exists to provide. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Everything around pods is tuned for images of ~10 GB and models of a few. The native MoE engines are a different shape: DeepSeek-V4 is ~154 GB and they go well past that, so a pod spends one to two hours filling its disk before it answers anything — on EVERY cold start, since a pod is disposable storage. The warning names the model, the size, and the fix: attach a network volume so the weights are fetched once and every later pod attaches to them instead. It appears at provision time, boxed so it cannot be missed in a log, and in the test run — which is where someone looks BEFORE committing to a configuration, while at provision time the money is already being spent. It says nothing when a volume is attached, and nothing for ordinary models. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Diarization had no probe, so a speaker pod could only ever report reachability — and reachability is what reported ok:true for an STT image that shipped nothing for speaker work at all. espeak provides the audio; the check is that the model returns at least one labelled turn, which a pod with no diarization stack cannot do. One speaker saying one sentence is a thin test musically and an honest one here. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod images were built to satisfy imports, not to serve models. The audit found coqui/XTTS-v2, every whisper.cpp model, diarization, speaker ID, SALAD place recognition and Surya all unservable on a pod, and the OCR image shipping no OCR engine at all. Three of the eight 'isolated' stacks are not isolated any more. Surya, speechbrain and pytorch_lightning all import cleanly beside torch 2.11 — their venvs were decided against an older torch and never re-checked, exactly like audiocraft's torch==2.1.0. They go straight into the ocr, stt and embeddings profiles: place recognition, speaker ID and Surya now work for a few hundred MB instead of three new images. Surya's subprocess engine is pointed at the pod's own interpreter, since building a venv to hold what is already installed would only cost rental time. Five stacks genuinely cannot share, each verified rather than assumed: pyannote (torchaudio.AudioMetaData is gone in 2.11), coqui-tts (a transformers 4.x symbol), CrisperWhisper (transformers 4.46 AND torch 2.5), NeMo (large, own torch) and paddle (its own CUDA runtime). Each gets an image carrying a baked venv, rather than 6 GB added to an image everyone pulls. The `speaker` capability no longer points at the STT image: that was the assumption that it was 'the STT deps plus a bit', and it shipped nothing for diarization or voiceprints at all. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The OCR pod image installed NO OCR engine — only transformers and pypdfium2. A pod could boot, pass its health check, accept a request and have nothing to do it with. Three gates in front of that each reported their own refusal first — 'OCR subsystem is disabled', then 'OCR engine surya is not enabled' — so it took three rounds to see that the engine was never there. docTR goes in, because it is the only one of the three that runs in-process. Surya caps pillow<11 and lives in an isolated venv by design; paddle wants paddlepaddle in one of its own. Either would put a second torch in the image — 6 GB to avoid a Pillow conflict — and the venv mechanism is there if a pod ever needs Surya's layout quality specifically. Each engine also has its own enable flag on top of the subsystem's, and two of the three default to off. A pod now enables the engine it was asked for, falls back to docTR when the request names something the image cannot serve, and still inherits the Surya licence decision from here rather than making it itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The isolated venv was solving a problem that was not real. audiocraft is pure Python: its torch==2.1.0 is a declared pin, not an ABI requirement, so --no-deps sidesteps it. The only genuine constraint was xformers, which it imports at module load — and 0.0.35 ships a py39-none wheel requiring torch>=2.10, which this image satisfies. I had ruled that out by grepping wheel filenames for cp312 and missing that abi3/py39 wheels cover 3.12. Verified in the built core image: torch 2.11.0+cu128 with xformers 0.0.35, MusicGen and AudioGen import, ops.memory_efficient_attention present. Cost in the main venv: 950 MB. The second venv cost 6.2 GB, because it carried a duplicate torch and a duplicate CUDA stack. The audio image drops from 15.8 GB to about 10.2 GB, which also removes the reason to split it in two. audio_backend (per model, or CODERAI_AUDIO_BACKEND for a pod) selects audiocraft or transformers explicitly, because they are not equivalent — melody conditioning and AudioGen exist only in the first. Asking for audiocraft when it is absent raises rather than quietly substituting something less capable. A model's own setting beats the environment it lands in, and the choice travels to a pod instead of being decided by what happens to be installed there. The isolated-venv worker stays in the tree: if audiocraft 1.3.0 turns out to misbehave against torch 2.11 at generation time, restoring one profile file brings it back. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three build cycles, three different missing modules — xformers, then torch at install time, then transformers. The list had been trimmed against what the MAIN venv provides, which is exactly backwards: an isolated venv starts with nothing, so it needs the whole declared set, minus only the torch pins we choose and the dev-only tools. Verified in a throwaway container against the real core image before building anything: audiocraft 1.3.0 on torch 2.5.1+cu121 with xformers 0.0.28.post3, and MusicGen and AudioGen both import. That check takes two minutes; the image build it replaces takes fifteen, which is why the last three rounds were so expensive. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Voice cloning failed with 'Could not load libtorchcodec', whose message blames FFmpeg and the torch version. It was neither: the CUDA build links against NVIDIA Performance Primitives (libnppicc.so.12), torch does not use NPP, and torchcodec arrives as a transitive dependency of f5-tts — so nothing ever installed it. Installing it is only half: torch preloads its own nvidia libs and NPP is not among them, so the library sits in site-packages and the loader never looks there. LD_LIBRARY_PATH covers the other half. Verified in the built image: torch 2.11.0+cu128, torchcodec 0.11.1+cu128, AudioDecoder imports. Two wrong theories preceded this — an FFmpeg major mismatch and a torch/torchcodec ABI mismatch — both suggested by that error message, and both disproved by loading each .so by hand with torch imported. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Second failure in that venv: 'No module named torch' while installing. xformers below 0.0.27 has no cp312 wheel, so pip tried to build it from source, and its setup.py imports torch — which does not exist yet in a fresh venv. The binding constraint was never audiocraft's torch==2.1.0 (unsatisfiable on this Python) but xformers, which audiocraft imports at module load: its cp312 wheels start at 0.0.27, and 0.0.28.post3 requires torch==2.5.1 exactly. So the venv is torch 2.5.1 + xformers 0.0.28.post3 — chosen by what actually has wheels for this interpreter rather than by what the libraries would prefer. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Dropped with the rest of its unsatisfiable pins, but audiocraft imports xformers at module load, so the venv built cleanly and then failed its import check with ModuleNotFoundError — which is what that check exists for: on a rented GPU this would have surfaced minutes into a generation instead. Its pin (<0.0.23) pairs with the torch it cannot have here; 0.0.25.post1 is the build that matches torch 2.2.2. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The gateway fix worked — the OCR request reached the pod, which booted, took it, and answered 'OCR subsystem is disabled (enable it in Settings → OCR)'. That is the right default for a fresh install and the wrong one for a machine that exists for the next four minutes and was rented specifically to do OCR. An ocr-capability pod now gets CODERAI_OCR_ENABLED, the engine it was asked for, and — when that engine is surya, or this deployment has already accepted it — the licence acknowledgement, because surya is GPL-gated and a pod cannot make that decision for itself. Asking for surya by name is the decision. Env only, never written to config.json: like the seed list, this describes one disposable instance, not a deployment. A test asserts it cannot leak to disk. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The audio build failed with 'No matching distribution found for torch==2.1.0': audiocraft's pin is unsatisfiable on Python 3.12 at all, because torch support for 3.12 starts at 2.2. So the venv uses the oldest torch this Python can run and installs audiocraft --no-deps, with the deps it actually uses listed explicitly. --no-deps is reckless in a shared venv and routine in an isolated one: nothing in there can reach the server's stack, and the build's import check fails the image if the version judgement turns out to be wrong. The step is now generic — any <profile>.venv-<name>.txt builds /opt/coderai/venvs/<name>, with an optional .nodeps beside it — rather than hardcoding audiocraft into the Dockerfile. The next library that pins an impossible torch will not need a new build step. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.2.14 was published for three profiles before the audiocraft venv landed, so that number now means different things for different images. Skipping it keeps a version honest: one tag, one tree. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The transformers fallback keeps MusicGen working on a pod, but it is not the same thing: audiocraft has melody conditioning and AudioGen, and transformers has neither. Supporting it is not a matter of adding a requirement — it pins torch==2.1.0, torchvision==0.16.0 and torchtext==0.16.0, and torch 2.1.0 has no build for the CUDA these images run, so installing it beside the server would leave the pod with no working GPU torch at all. So it gets its own venv and is driven as a subprocess, the same answer parler-tts and the OCR engines already use here. AudiocraftModel deliberately wears audiocraft's own interface (set_generation_params / generate / generate_with_chroma / sample_rate) so the generation path cannot tell whether the library is in this process, in a venv next door, or absent. The venv is baked into the audio image rather than built on first use: a pod is rented by the second, and pip-installing a library during a cold start costs more than carrying the layer. Order of preference: in-process audiocraft, then the isolated venv, then transformers — which needs no venv but cannot do melody. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
None of these were placement or pod bugs: each image booted, pulled its model and got as far as executing. The remote machinery worked; the payloads did not. voice — torchcodec arrived as a transitive dependency of f5-tts, PyPI served the CUDA-13 build, and it died with 'libnvrtc.so.13: cannot open shared object file' after loading the model. The same class of bug as the torch pin: a CUDA-coupled package escaping the index. It now installs from the CUDA index with the rest, resolving to torchcodec 0.11.1+cu128. audio — 'No module named audiocraft'. audiocraft pins an old torch, so putting it in the image would drag torch backwards and undo the CUDA pin. transformers ships MusicGen itself, so a pod serves it through transformers instead; an install that has audiocraft still uses it. Melody conditioning stays audiocraft-only rather than silently doing something different. video — 'TextToVideoSDPipeline.__call__() got an unexpected keyword argument callback_on_step_end'. Older diffusers pipelines do not take it and raise inside the call, minutes in, with the model loaded and the GPU paid for. The kwarg is now added only when the pipeline's signature accepts it. The try/except that used to wrap it caught nothing: assigning to a dict never raises. This one bites local generations too, not just pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr names its engine in a multipart field called `engine`, not `model`. The gateway only ever looked for `model`, so an OCR request carried no model as far as it was concerned: per-model placement resolved correctly, the test reported 'target: its own pod', and then the gateway refused with 'nothing configures ocr to run remotely'. Two layers disagreeing about the same request. _MODEL_FIELD maps the endpoints whose field is named something else, and the extractor takes the field name for JSON, multipart and urlencoded alike. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Voice cloning runs on a TTS model (XTTS, F5) but is a different endpoint served by a different pod image, so section alone maps it to 'tts' and it could never be placed as 'voice'. That is why the voice image had never been tested: not a missing probe, a model that could not be pointed anywhere. The harness also gains --add-as, to create a temporary catalogue entry for a capability nobody has a model configured for (music generation, a small video model), instead of requiring the real catalogue to be edited by hand first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three capabilities were listed as un-probeable. Only one of the reasons held. Voice cloning 'needs a reference sample' — and espeak produces one, with a transcript we know exactly because we chose the sentence. Audio generation was 'minutes of GPU': asking for 2 seconds instead of the 10s default proves the model renders audio for a fifth of the cost. Video was 'real money': the smallest clip a model can make — 9 frames, 256x256, 4 steps — proves the path without paying for a film. 'Expensive' was an argument for asking for less, not for proving nothing. Generated media has no ground truth, so these check that media came back at all; a 200 carrying nothing is still a failure. Face swap stays reachability-only and now says why: a drawn face does not survive face detection, and a real person's photo is not ours to use. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
OCR engines are picked by name (paddle|doctr|surya) rather than registered like a model, so nothing could place one individually and the test for them failed with "not in models.json" before it ever reached a pod. The only way to send OCR to a pod was a capability remote — a production routing switch, and the mechanism that once sent live embedding traffic to a rented GPU. An ocr_models section maps to the ocr capability, so an engine gets the same per-model placement every other model type already has, and the harness creates a temporary entry for one when the catalogue has none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
--extra-index-url only ADDS an index; pip then picks the highest version across all of them, and PyPI's default torch is built for CUDA 13. So the pin never applied: every capability image shipped a CUDA-13 torch and failed on any machine with an older driver — 'The NVIDIA driver on your system is too old (found version 12080)'. RunPod's fleet is mixed, so the same GPU model passed or failed depending on which machine the pod landed on, which is exactly why this looked like flakiness for so long. torch, torchvision and torchaudio now install from the CUDA index alone, via --index-url, before anything else; installed later from PyPI they would pull a matching CUDA-13 torch back in. cu128 resolves to torch 2.11 — the version the local install already runs — and runs on the driver that failed. Both Dockerfiles print the resolved torch build at each layer, so a silent substitution cannot happen again. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The test run synthesises a spoken sentence to probe an STT model with real audio rather than a reachability check. espeak was never in the image, so the probe fell back to "configured, not verified" — correct behaviour (the gap is ours, not the model's) and therefore invisible: no error, no failing test. Found by inspecting a built image. Added to the full build and to the thin update layer, so it is present whichever path produced the running image. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod could not say WHICH build it was running, which is exactly the question a cached ':latest' raises — an images pod reported a dependency missing that had already been published under that tag. The build now bakes IMAGE_TAG in, so the boot record names the image. /boot also showed only 'lifespan start': the seed registration and the end of startup were plain prints. They are phases now, so one request covers the whole boot including how many seeded models registered and whether that step failed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
README gets the per-model test run and the pod boot record; docs/runpod.md and docs/remote-execution.md get the findings themselves, because each one cost a live failure to learn: - RunPod publishes no pod-log API. Checked against their OpenAPI spec: 23 routes, none for logs, and GraphQL refuses introspection. Code that "tried several routes" was guessing at URLs that never existed. - The port appears only after the image is pulled, so the boot budget covers the download: 133s warm vs repeated failures at 300s cold, each one renting a fresh machine to pull the same image again. - A machine may serve a cached ':latest', so a test against a floating tag cannot say which build it ran — pin a version when the answer has to mean something. - Setting remotes.endpoints[capability] to test ONE model moves every request of that kind; that is how production embeddings ended up on a rented pod. - Writing a config does not change routing until an idle engine takes the push, so a test must confirm the serving process has it before testing. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The port a pod exposes appears only AFTER its image is pulled, so the 300s budget had to cover the download — it was written assuming the container was already there. The same 7.3 GB image took 133s on a machine with the layers cached and failed repeatedly at 300s on cold ones, and each failure rented a fresh machine and paid for the whole pull again, three times. 900s, overridable per model; being wrong now costs six cents instead of three pods. RunPod publishes no pod-log API: the REST OpenAPI spec lists 23 routes and none serve logs, and GraphQL refuses introspection. Every attempt returning HTTP 400 was code guessing at URLs that never existed. So the pod reports on itself — boot.sh is the entrypoint and logs timestamped phases from the first instant the container exists (including the seed list, which is where "not available. Use one of: " would have been caught in one line), and /boot serves that record plus the app's own phases. A failed boot now asks the pod before falling back to a console link a human has to open. Nothing covers the pull window — no code of ours is running yet — so wait_ready reports status and uptime every 60s instead of going silent for minutes. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Testing an image model by setting remotes.endpoints["images"] moves every request of that kind. An embeddings remote left on from an earlier test did exactly that on a live box: production embedding traffic went to a rented pod while the local GPU sat idle, and the operator noticed because the card was quiet. tools/runpod_model_test.py writes a runpod block on the single model entry instead — per-model placement already beats the capability map, which is what lets two models of the same kind sit in different places. Writing the config is also not enough. The front pushes a reload and the engine takes it only when idle, so a harness that writes and immediately tests is testing the PREVIOUS configuration. That produced a whole pass of false "nothing configures this to run remotely" failures, and could just as easily have reported a remote pass for a request served locally. GET /v1/models/test/state answers for the process that serves the request; the harness polls it until the placement it wrote is visible and refuses to test if it never is. The new endpoint is in _SKIP_PATHS: forwarded, it would report a pod's view of a config the pod does not have. Co-Authored-By:
-
- 14 Sep, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
The first RunPod pass over every capability produced four failures, three of which were ours: The TTS probe sent voice "default", which no shipped model has — Kokoro loaded fine on a rented pod and then answered "Voice default not found in available voices". TTSRequest already defaults to a real voice, so the probe stops inventing one. OCR engines are chosen by name (paddle|doctr|surya) and are not models.json entries, so the test refused with "'surya' is not in the model list". _describe now recognises them. For a text model the refusal offered "a remote for its text capability" as a fix. It is not one: /v1/chat/completions is outside the gateway, so that config routes nothing and the test would then report a remote pass for a request served locally — the exact failure the forced-remote check exists to prevent. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A rented images pod downloaded SDXL for ten minutes and then failed with "No package metadata was found for bitsandbytes". models.json carries quantization settings for EVERY model type, not just LLMs, but bitsandbytes was only in the text profile. It moves to core, the shared layer under all ten images. stt and ocr were reachability-only, which reported ok:true while proving nothing beyond "a URL is configured" — a pass that would survive a completely broken pod. Both inputs can be synthesised honestly: espeak speaks a known sentence, PIL draws known words. The result is checked against what went in, so a model that returns fluent nonsense fails too. When the synthesis itself cannot run (no espeak, no fonts) the probe falls back to the reachability check rather than blaming the model for a gap on this side. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs, both found by the first end-to-end embeddings test on a rented pod. A capability image's entrypoint is `uvicorn codai.api.app:app` — it never runs codai/main.py, so set_config_manager() was never called and codai.admin.routes.config_manager stayed None. Both the CODERAI_SEED_MODELS list and the runtime catalogue write read through that global, so a pod seeded with one model answered "not available. Use one of: " with nothing after the colon. The lifespan now builds its own ConfigManager when none was set — loading it is what parses the seed list — and says how many seeds it registered. The bearer token a pool launches its pods with is generated in memory when none is configured. A pod that outlived the process that rented it was then adopted with a DIFFERENT token and answered 401 for the rest of its paid life (the test run reported serves: "(HTTP 401)"). The token is now recorded in the shared pod registry and adopted along with the pod. Verified live: bge-m3 on a rented RTX 2000 Ada returned a real 1024-dimension embedding, 67.9s cold. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Registration reaches the model manager, seeded models survive the engine assignment filter, siblings wait for a booting pod, and pod boot phases plus catalogues are reported for diagnosis. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The last three pod failures were each diagnosed the same way: find the pod URL in the engine log, ask the pod by hand what it serves, work backwards. That is a diagnosis loop measured in minutes of log archaeology per attempt. Boot is now reported in PHASES, because "it took 150s" is not a diagnosis: port open after Ns (image pull + container start), serving after N more. The two have different fixes — a bigger boot_timeout_s versus a bigger load_timeout_s. The moment a pod is up, its /v1/models is logged. An empty catalogue now says so outright — "pod serves NO models — it will refuse every request" — instead of surfacing minutes later as "Model 'x' is not available. Use one of: " with nothing after the colon. And a test run's answer carries the pod diagnostics: which pods are live, their GPU, uptime, spend so far, URL, and the models each one lists. The fact that mattered in every failure so far was the catalogue, and it is now in the response rather than three commands away. Co-Authored-By:
-