- 27 Aug, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authoredFollow-up to the same-path multi-config runtime fix. Two more path-keyed spots broke a same-file sibling addressed by its alias (e.g. lisa-32k): 1) GGUF dispatch: _resolve_local_gguf matched entries by path/basename only, so a sibling addressed by alias wasn't recognized as GGUF and fell to the HF loader ("lisa-32k is not a valid model identifier"). Now also match the entry's alias. 2) Measured footprint cross-contamination: persist_model_field matched entries by path, so one config's measured_vram_gb/n_gpu_layers overwrote the sibling's — the small-ctx variant inherited the large-ctx footprint (34 GB) and still offloaded to CPU. persist_model_field now targets by config_id when given; record_vram_delta passes cfg['config_id']; _model_cfg carries config_id/ config_name so the config knows its own entry identity. Net: lisa (178K) and lisa-32k (32K) are now fully independent — distinct config, GGUF load from the shared file, and separate measured footprints. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
-
Stefy Lanza (nextime / spora ) authored
The web UI stores sibling configs for one model file (distinct config_id, e.g. a smaller-context variant), and the front already routes by alias (_route_key: "two configs of the same model with distinct aliases can be assigned to different engines"). But engine startup keyed every text-model config by path (_model_id -> path), so a second same-path entry OVERWROTE the first — BOTH aliases then resolved to the last-registered config (verified: lisa and lisa-32k both got n_ctx=32768, silently changing the primary too). Fix: in main.py text-model registration, key the PRIMARY config by path (stays path-addressable) and each SIBLING (same path, already seen) by its ALIAS, so the two configs are distinct and each alias resolves to its own n_ctx. A sibling without an alias is skipped (can't be addressed). The GGUF loader already reads the file from cfg['path'], so an alias-keyed sibling still loads the right file. Verified: lisa->178000, lisa-32k->32768, same underlying .gguf. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The reranker is registered in embedding_models, so request_model(model,"rerank") was rejected ("is a embedding model and cannot be used for rerank generation"). Resolve with model_type="embedding" (its actual category) so validation passes; rerank.py still loads it as a cross-encoder (AutoModelForSequenceClassification), re-loading if a cached embedder object is found under the same key. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
bitsandbytes 0.49.2's Params4bit.__new__ has no **kwargs and rejects the _is_hf_initialized kwarg that transformers 5.12 passes during 4-bit weight init, so every load_in_4bit model crashed with: Params4bit.__new__() got an unexpected keyword argument '_is_hf_initialized' In production this turned a single default-model chat request (Qwen3.5-9B, load_in_4bit) into a ~2h internal 60-retry storm. bnb 0.50.2 adds **kwargs to Params4bit.__new__ (absorbs _is_hf_initialized); verified end-to-end: a transformers load_in_4bit load now succeeds on cuda:0 (torch 2.11/cu130), and a GPU 4-bit quantize/dequantize round-trips. Image rebuilt with bnb 0.50.2. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reranker (e.g. BAAI/bge-reranker-v2-m3): - new codai/api/rerank.py: /v1/rerank scores query
↔ document pairs with a native AutoModelForSequenceClassification cross-encoder (no new dep), cached + VRAM/thermal-managed via request_model like embeddings. Returns {index, relevance_score(sigmoid), document?} sorted desc, top_n honored. - router: /v1/rerank added to _INFERENCE_PATHS so the front routes it by model capability (transformers → nvidia engine), like /v1/embeddings. - app: include rerank_router. - capabilities: add `reranking` + `ocr` fields; detect 'rerank'/'cross-encoder' names as reranking. /v1/models discoverability: - list_models() now appends enabled OCR engines (paddle/doctr/surya) as type='ocr' entries (a node subsystem invoked at /v1/ocr?engine=<id>), so clients can discover them. - a reranker registered in models.json (embedding_models, capabilities ['reranking']) surfaces automatically via list_models, like bge-m3. Co-Authored-By:
-
- 26 Aug, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
The per-token Python loops that built the sparse {token_id: weight} dict and the colbert vector lists held the GIL between forward passes, starving the GPU (steady ~10-36% util with brief bursts). Replace them with GPU tensor ops: - sparse: relu(sparse_linear·h) with invalid positions (specials/padding) zeroed, then max-pooled per token id via a single scatter_reduce(amax) into a (B, vocab) matrix; per-row nonzero extraction is O(nnz), not O(L). - colbert: one C-level tolist per row over its valid tokens (vs per-token). - valid mask computed on-device via torch.isin. Output byte-matches the previous per-token logic (dense/sparse/colbert within 1e-4). Benefit grows with batch size, so larger client batches translate to GPU work instead of Python-loop time. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
hf_hub_download(local_files_only=True) intermittently failed to locate the cached sparse_linear.pt inside the engine process, so _is_bge_m3 returned False and bge-m3 silently degraded to the dense-only sentence-transformers path. _resolve_repo_file now falls back through try_to_load_from_cache and a direct filesystem glob of the HF cache (models--org--name/snapshots/*/file), which resolves reliably regardless of process context. Detection keys off the sparse head file rather than the model_type string. (Debug scaffolding removed.) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- import build_from_pretrained_kwargs inside _load_bge_m3 (it's a local import of _load_embedding_model, not module-level) — the NameError made every bge-m3 load fall over so the model silently degraded to the dense-only path. - cast the sparse/colbert heads to the base model's dtype (bf16) so the head matmul over the last hidden state doesn't dtype-mismatch. - requires_grad_(False) on the heads (inference only) — no autograd graph per request. - log the bge-m3 backend selection at load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add BAAI/bge-m3 hybrid embedding (dense + sparse lexical, optional colbert) without the FlagEmbedding dependency (which pins older transformers/peft and would conflict with the transformers 5.x stack): the sparse/colbert heads are plain Linear layers over XLM-RoBERTa's last hidden state, reproduced natively with transformers + torch already in the venv. - new 'bge-m3' embedding backend: detected by xlm-roberta model_type + a sparse_linear.pt head (so plain bge-large etc. stay on the dense ST path); loads the base model + sparse_linear.pt (+ colbert_linear.pt if present). - _bge_m3_encode: dense = L2-normalized CLS; sparse = max-pooled relu(sparse_linear·h) per token id (specials dropped) as {token_id: weight}; colbert = L2-normalized colbert_linear·h per content token. - /v1/embeddings extended: request.embedding_types selects any of dense|sparse|colbert (overrides the model-config default); response items gain sparse_embedding {indices, values} and colbert_embedding. Dense stays in the standard `embedding` field (base64 honored). Configurable default via the model's embedding_types config. - cleanup()/eviction branch moves the base model + heads off GPU. Co-Authored-By:
-
- 22 Aug, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
The Surya-2 engine serves its VLM via a managed vLLM subprocess (vllm_worker.ensure_service). That subprocess is not a manager-tracked model, and the OCR VRAM releaser only tore down the lightweight worker pools (killing the HTTP workers) — it never stopped the vLLM holding the bulk VRAM (gpu_memory_utilization × card). So on-request eviction saw the VRAM as used, found nothing it could free, and (with no CPU fallback) a docTR/Paddle load on the same card could OOM. - vllm_worker: add stop_service_for(cfg, model_path, served_name) (stop ONE service, return estimated GB freed) + is_running() helper. - OcrManager._release_vram now also stops the managed Surya-2 vLLM (_stop_surya_vllm) when surya_serve == "vllm", so the registered external releaser actually reclaims its VRAM. Next OCR request re-boots it via ensure_service. Targeted by (model_path, served_name) so an LLM vLLM instance is left alone. Co-Authored-By:
-
- 21 Aug, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
OCR ran outside the shared GPU governors: /v1/ocr(/batch) and run_ocr_step call OcrManager.ocr_document directly, bypassing request_model, so OCR neither waited on thermal cooldown nor evicted/released VRAM against LLM/diffusion models. - thermal: ocr_document now awaits thermal.wait_until_safe (via to_thread) before OCR-ing pages, so OCR honours the same cooldown/front-pause gate as every other GPU workload. - VRAM (both directions): * evict-before-build: when a pool loads its instances, ask the model manager to free size*per-instance-VRAM first (_evict_models_for_vram), so OCR contends on equal footing instead of OOMing. * releaser: OcrManager registers _release_vram(needed_gb) via register_external_vram_releaser, tearing down built pools so loading an LLM/diffusion model can reclaim OCR VRAM. Pools rebuild lazily next request. - engine cleanup() (SYNC, eviction-thread safe) + vram_gb() footprint per engine: docTR ~0.7 GB (frees model + empty_cache), subprocess ~2.5 GB (kills worker). _Pool tracks all instances and release_sync() tears them down. - configure() now tears down stale pools on reconfigure (prompt VRAM free instead of GC-deferred). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Each engine's status payload reported only running/queued/paused tasks, so finished generations on a non-primary engine (e.g. radeon, reached via the API) never got to the front and vanished from the Tasks page. Now every engine also reports its most recent terminal tasks (capped at 10, newest-first); the front's _merge_engine_tasks already merges non-primary e.tasks (tagged by engine), so the page shows the last-N per engine. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
When vllm.enabled, the front-proxy spawns a dedicated 'vllm' engine on the NVIDIA card(s) with capability {vllm} only (mirrors the nvidia-gguf split) — a real subprocess, so it appears in the engine/task page and routing/VRAM/thermal apply. Models pinned backend:vllm route to it; vllm removed from the shared _DEFAULT_CAPS so no other node claims them. vllm.gpu -> that node's CUDA_VISIBLE_DEVICES/CODERAI_ENGINE_GPUS. CUDA-only. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
_resolve_model_dir now returns a model-list entry's path even when it's a HF repo id (e.g. Qwen/Qwen2.5-0.5B-Instruct), and treats an org/model model_name as a repo id. vLLM's --model accepts both; the previous isdir-only check returned None -> 'no model resolved'. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM now works like the in-process engines: tag a model entry with backend:vllm and it's served by its own path, under its own name (VllmBackend uses the requested model name as --served-model-name, resolved from models.json). vllm.model_id/model_path are optional (single-model convenience only); default blank -> no synthetic /v1/models entry, no alias auto-routing, pin-only. Docs + admin card updated to reflect the model-list flow. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- _get_model_used_vram_gb reports a vLLM model's real footprint (gpu_memory_utilization x total VRAM via new _total_vram_gb), so the standard evict-before-load path frees the right room before starting vLLM (gmu~0.9 => effectively exclusive; low gmu co-tenants). Eviction OF vLLM to free room for others is already handled by normal LRU (backend .cleanup -> stop_service); thermal throttling is inherited via request_model/wait_until_safe. - vllm.gpu (CUDA_VISIBLE_DEVICES) to pin the instance to specific NVIDIA GPU(s); wired in vllm_worker env + config + admin card + routes. vLLM is CUDA-only (capability on nvidia/ cuda nodes); AMD would need a ROCm vLLM build. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Surya2 returns PageOCRResult.blocks[] (html/polygon/confidence/label/reading_order), not .text_lines like classic surya. Worker now extracts text from block HTML (tag-stripped) ordered by reading_order, with polygon->bbox. Verified end-to-end: latest Surya2 served via the vLLM backend transcribes a scanned IT legal page with all codici fiscali correct. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
flashinfer JIT-compiles kernels with ninja at startup, which fails on hosts without a build toolchain wired (exit 127). Default it off so vLLM uses FLASH_ATTN + native sampler out of the box; overridable via vllm.extra_env. Verified end-to-end: VllmBackend served Qwen2.5-0.5B and answered a chat with token usage. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM integrated exactly like ds4/ktransformers — a managed external engine in an ISOLATED venv (it pins torch 2.13/cu13, conflicting with the main venv), proxied over its OpenAI HTTP API, selected per-model via a `backend: vllm` pin or the vllm.model_id alias (never auto-claimed). Continuous batching for high aggregate throughput. - config.py: VllmConfig (+ Config field, from_dict, to_dict) - codai/api/vllm_worker.py: managed vllm.entrypoints.openai.api_server subprocess in the isolated venv (venv resolver: config>/opt/coderai/vllm_venv>/cache>~/.coderai), /v1/models health gate, auto_build - codai/backends/vllm.py: VllmBackend OpenAI proxy (mirrors ktransformers) - manager: get_active_vllm_config, "vllm" in _ENGINE_BACKENDS, _vllm_name_claims=False, vllm_should_handle, load branch, text-accept, /v1/models surfacing - front-proxy: required_capability (vllm pin+alias), _DEFAULT_CAPS vllm on GPU nodes, assignment/engine_supervisor/app threading + reload list - admin: routes get/set, settings.html vLLM card, models.html dropdown option - requirements-vllm.txt (vllm==0.27.1); docs/vllm.md marked implemented; __version__ 0.1.86 OCR Surya2-via-vLLM: ocr.surya_serve = local|vllm|llamacpp. In vllm mode the Surya engine serves surya_model (datalab-to/surya-ocr-2) through the vLLM backend and attaches via SURYA_INFERENCE_URL — the correct path for the latest "Surya2" VLM (llama-cpp-python's server hit a recurrent/hybrid KV-slot bug on surya-2). Co-Authored-By:
-
- 20 Aug, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
- PaddleOCR 3.7 dropped use_gpu/use_angle_cls/show_log and raises ValueError for unknown ctor args; worker now tries 3.x (device=) then 2.x kwargs and prefers .predict() over .ocr(). Verified on a real scanned IT legal page. - surya-ocr >=~0.15 ('Surya2') rearchitected to a VLM needing an external vLLM-in-Docker or llama-server backend (SpawnError in an isolated venv). Pin to 0.6.4 which runs det+recognition locally on torch, matching the worker's API. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Unpinned paddlepaddle-gpu>=3.0.0 made pip backtrack for ~an hour across the 2 GB package while satisfying paddleocr/paddlex constraints, so the isolated venv never finished. Pinning the exact version (which ships a cp313 GPU wheel) makes it install cleanly in one pass. Co-Authored-By:
-
- 16 Aug, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
First OCR request for an unbuilt paddle/surya engine no longer blocks for minutes; with auto_build it kicks off the background builder (same one the web UI uses) and returns 503 'building — retry shortly'. Without auto_build it points at Settings. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
OCR (v0.1.85): new codai/ocr/ subsystem — dedicated OCR engines (NOT a VLM): - docTR (in-process, GPU) + PaddleOCR & Surya (isolated-venv subprocess workers via codai/ocr/workers/ocr_worker.py to avoid opencv-contrib/pillow<11 clashes; paddle-gpu bundles its own CUDA so a cu12x wheel runs on newer-CUDA hosts). - /v1/ocr, /v1/ocr/batch, /v1/ocr/schemas CRUD; `ocr` pipeline step. - Data-driven, user-extensible extraction schemas (named files + inline + auto), built-in seeds (italian_sentenza/generic_document/invoice); optional JSON Schema validation. Structured extraction via an existing coderai text model. - Configurable stamp/signature detection (off|layout|detector|both); bundled signature YOLO default; layout markers EN+IT (timbro/sigillo/firma). - Admin OCR card: engines/instances/detect/extraction + schema picker/raw-JSON/ field-builder + isolated-venv build trigger (background build + status). - Isolated venvs resolve baked(/opt) > /cache mount (persistent) > ~/.coderai. - Deps: pypdfium2 in base requirements; requirements-ocr(.txt/-paddle.txt); requirements-surya.txt; build.sh --ocr. Also lands the multi-engine backend work: colibri multi-family (GLM/DeepSeek/Kimi), kimi-k3-in-c (k3), ktransformers (kt) — per-model selectable via resolve_engine_backend; plus a deferred vLLM backend design note (docs/vllm.md). Co-Authored-By:
-
- 31 Jul, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
Embedding admission gate: - default min-interval 200ms -> 300ms (paces GPU starts harder) - expose embed_max_concurrency / embed_max_backlog / embed_min_interval_ms per-model in the admin model editor (embedding-gated section), round-tripped through /admin/api/model-configure into models.json Engine supervisor circuit breaker: - quarantine an engine that exits crashloop_max (5) times within crashloop_window (120s) instead of respawning it ~1/s forever. A GPU that has fallen off the bus (Polaris secondary-bus-reset bug) makes its engine die on every launch; the breaker takes radeon cleanly out of routing (embeddings fail fast) instead of hammering a dead card. Manual restart_engine() clears it. Pairs with host-side RX580 mitigations (dpm=high lock, lockup_timeout=25s, conc=1/300ms on the 3 vulkan embedding models, reset watchdog + thermal sampler).
-
- 30 Jul, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
The RX 580 (Polaris/Vulkan) keeps timing out its GPU ring under relentless back-to-back embedding load — a global throttle can't distinguish a fragile card from a robust one. Make the admission gate PER-MODEL: each model id gets its own semaphore + backlog counter + pacing, resolved from the models.json entry (embed_max_concurrency / embed_max_backlog / embed_min_interval_ms), falling back to the CODERAI_EMBED_* env vars, then defaults. Default min interval is now 200ms (was 0) so GPU starts are paced out of the box, giving the ring breathing room; a robust CUDA model can set it to 0 per-model. Bump 0.1.82. Co-Authored-By:
-
- 29 Jul, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP speculative-decode head ("speculation is not ragged-safe across KV slots") and its batched decode loop has no MTP path at all — MTP lives only in run_serve's single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes = fewer expert-streaming rounds, a real decode win on this IO-bound engine. MuxEngine now selects single-client mode when kv_slots==1: launches run_serve, drains its one-shot startup telemetry, and runs turns synchronously over the run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 + STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path. pause/resume are no-ops in single-client mode (run_serve would misread the frame; thermal falls back to the front's SIGSTOP). Bump 0.1.81. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
allow_cross ("use both cards for one model") was applied to EVERY engine the moment any model enabled gpu_split — which un-isolated the radeon engine's Vulkan ICD too. Combined with the embedding loader not pinning non-split models to their native card, the radeon engine's embeddings (Qwen3-Embedding, gme-Qwen2-VL) drifted onto the NVIDIA 3090 after a Vulkan device-order change. Scope it: each engine is isolated to its own backend by default; a per-model split names its lead engine (the model's `engine` field), and only that lead's card gets cross-backend visibility. A global offload.gpu_split still crosses all engines (explicit opt-in). Preserves nvidia-led splits onto the RX 580 while keeping radeon/intel Vulkan engines pinned to their own card. Bump 0.1.80. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Embeddings bypassed the front request queue (relayed unqueued) and the per-model lease scheduler, so the only guard was ratelimit.py's per-IP fixed window. A single authenticated client could pipeline unlimited embedding requests straight at the GPU — which on the Polaris/RX580 Vulkan backend wedged the SDMA ring (ring timeout -> GPU reset -> device lost from bus). Add _embed_admission() around create_embeddings: bounded concurrency, backlog shedding (429 when in-flight >= concurrency + backlog), and optional pacing between GPU starts. Tunable via CODERAI_EMBED_MAX_CONCURRENCY (2), CODERAI_EMBED_MAX_BACKLOG (32), CODERAI_EMBED_MIN_INTERVAL_MS (0). Bump 0.1.79. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
GLM output was garbage: colibri keeps only EOS as a hard stop in serve mode (it filters the non-EOS special-token stops for tool-call safety, #401), so the model ran past its turn and emitted <|user|>/<think>/</think> and repeated the prompt. - backend: clean_glm_output cuts the reply at the first turn marker (<|user|>, <|observation|>, <|assistant|>, <|system|>, <|endoftext|>) and strips control tokens (<think>,</think>,[gMASK],<sop>). Applied to both non-stream and stream paths; the stream holds back a short tail so a marker split across chunks never leaks. - Early stop: generate paths pass a `cancelled` hook that fires the moment a turn marker appears, so colibri CANCELs instead of decoding to max_tokens — a big saving at streaming-bound decode speeds. MuxEngine.run now treats our own CANCELLED ack as a clean early stop (returns partial stats) instead of raising. - alias: _resolve_container now resolves the configured colibri model_id (e.g. glm-5.2-colibri) and colibri.model_path to the container, not only the full repo path — so `{"model":"glm-5.2-colibri"}` loads instead of "no container resolved". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per design: a hot CPU is a global hardware event, so every engine pauses — not just the CPU-heat source. Reverts the per-engine CPU exemption from 0.1.75 (front _thermal_loop) and 0.1.76 (engine-level wait_until_safe): the shared-CPU term again applies to all engines. The original "radeon stuck while colibri ran free" bug is addressed the right way — by making colibri ACTUALLY honour the pause (targeted SIGSTOP that keeps the engine's HTTP alive, and colibri's cooperative PAUSE/RESUME mux frame), both retained — so colibri stops, the CPU cools, and all engines resume together instead of the innocent engine being stranded. Removed the now-unused _scan_pgroup_cpu/_engine_cpu_relevant helpers and _self_cpu_relevant. Co-Authored-By:
-
- 28 Jul, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
Companion to the front-side fix (0.1.75): the ENGINE-level guard (wait_until_safe in codai/models/thermal.py) still applied the global CPU term to every engine, so a GPU-bound engine (radeon embeddings, ~0 CPU) kept "Cooling … CPU>78 — waiting" while colibri (596% CPU, dense-on-CPU) held the shared CPU above resume. It never cooled because the engine that was cooking it wasn't the one being made to wait. wait_until_safe now honours the CPU term (pause / join-cooldown / stay-in-cooldown / soft-throttle) only when THIS engine's own process tree is a real CPU-heat source (_self_cpu_relevant: rolling-max of read_process_tree_cpu ≥ 1.5 cores). The radeon engine no longer waits on colibri's CPU heat; the nvidia engine (colibri in its tree) still throttles itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Fixes the "only radeon was stopped while everything under throttle" case and gives colibri a clean pause instead of SIGSTOP. Thermal (engine_supervisor): - Per-engine CPU gating: measure each engine's process-group CPU (rolling max) and apply the GLOBAL cpu_hot/cpu_warm term ONLY to real CPU-heat sources. A GPU-bound engine (embeddings) is never paused/stranded in the resume<CPU<high dead-band that a CPU-heavy engine (colibri) keeps the shared CPU parked in. - Targeted SIGSTOP: _thermal_signal now freezes the engine's native compute CHILDREN (colibri/ds4/whisper-server), leaving the Python HTTP server alive to ack pause/resume — killing the resume-fail / SIGSTOP
↔ SIGCONT thrash. Falls back to killpg for in-process (torch) engines. colibri graceful PAUSE/RESUME: - packaging/patch-colibri.py: idempotent serve-mux patch adding PAUSE/RESUME control frames — colibri idles the decode loop between tokens (keeping KV) instead of being frozen; build.sh applies it after clone. - MuxEngine.pause()/resume() + colibri_worker.pause_all()/resume_all(); the engine's /internal/thermal-pause|-resume now drives them, so the cooperative thermal throttle can cool the box without SIGSTOP. Credits (requested): clear acknowledgements for the brilliant engines coderai builds on — colibri (JustVugg), ds4/DwarfStar (antirez), llama.cpp/whisper.cpp (Gerganov), and VPR/geo research — in the admin settings UI, README, and docs (new docs/glm-colibri.md, ds4 doc footer). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
colibri's context is the CTX env (default 4096), DISTINCT from NGEN (max new tokens). We only set NGEN, so any prompt over ~4095 tokens was rejected ("prompt does not fit … CTX=4096") — coding clients with big system prompts always tripped it. MuxEngine now sets CTX from the model's configured n_ctx (a caller/extra_env CTX still wins). GLM-5.2 (glm_moe_dsa) uses compressed MLA KV held in host RAM, so large contexts cost RAM, not VRAM — the model itself supports up to 1,048,576 positions. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
colibri's GPU tiering is OFF unless COLI_CUDA=1 (CUDA/HIP) / COLI_METAL=1 is set — it is NOT implied by shipping a CUDA binary. We passed CUDA_EXPERT_GB without it, so the engine aborted at startup: "CUDA_EXPERT_GB requires COLI_CUDA=1" -> exited before READY. _build_env now defaults the backend on for cuda/hip (COLI_CUDA=1) and metal (COLI_METAL=1) targets — coderai ships a CUDA build and routes colibri to the CUDA engine — and only sets CUDA_EXPERT_GB when the CUDA backend is enabled. extra_env is applied last so COLI_CUDA=0 can force a CPU run. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engine is a pre-built binary, bundled in the image and built on a host with the CUDA toolkit — never compiled per-request in the CUDA-*runtime* container (which has no nvcc). Build it with the host CUDA 13.x toolkit like ds4: the binary links libcudart.so.13, resolved in-container from /opt/coderai/local-libs (the CUDA-13 runtime coderai already ships for PyTorch), so it adds no portability constraint. - worker: _detect_build_target now needs a REAL nvcc (a runtime container has /usr/local/cuda but no compiler); ensure_built fails early with a clear "pre-compile the binary" message instead of cloning and dying deep in `make` with the confusing "nvcc not found … backend_cuda.o Error". _make_args defaults CUDA to a PORTABLE arch (sm_80..120 + PTX) and points at the resolved nvcc/CUDA_HOME. - build.sh --colibri: require a real nvcc and default CUDA_ARCH=portable (like ds4's cuda-generic), so the bundled binary isn't locked to the build host's GPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
hf leaves a stale <sha>.<etag>.incomplete orphan in blobs/ after a resumed or retried chunk even though the real file finished — so "download complete" and the model page disagreed (settings said done, the page showed
⚠ incomplete). - model list: an .incomplete file now counts as an unfinished download ONLY when its final blob (<sha>) is absent; an orphan whose final blob exists is ignored. Fixes the false badge for ALL HF models, not just colibri. - colibri catalog: _colibri_repo_complete now tests for dangling snapshot symlinks (the true "file missing" signal) instead of .incomplete presence, so the settings "present" indicator matches reality. - colibri catalog label → the real size (~429 GB / 400 GiB). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The GLM-5.2 model is a DIRECTORY (an HF repo of int4 .safetensors shards + config + tokenizer/MTP head), not a single file/GGUF — so it downloads via the normal whole-repo snapshot and lives in the HF cache snapshot dir. - routes: _COLIBRI_DEFAULT_MODELS catalog (mastouri/GLM-5.2-colibri-int4-g64-with- int8-mtp) + GET /admin/api/colibri/default-models; extend /admin/api/model-add to accept `backend` and `alias` so a downloaded repo registers as a colibri-backed model in one step. - settings.html: a "Download the GLM-5.2 container" picker + button in the colibri card — full-repo snapshot (no file_pattern → a directory), then registers it as a colibri model addressable by the configured model id. - backends/colibri: _resolve_container resolves an HF repo id → its local snapshot DIRECTORY (snapshot_download local_files_only), so a normally-downloaded model serves without any manual path wiring. - models.html: add "colibri" to the per-model backend dropdown (manual add path). - app: allow the new catalog endpoint through the no-auth admin allowlist. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- parser: add GLMParser + parse_glm_tool_calls/strip_glm_tool_calls for GLM-5.2's <tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value>...</tool_call> format (byte-compatible with colibri's own parser). Gated on the <arg_key> marker so a generic <tool_call>{json} from other families is never hijacked. Declared-type coercion keeps string args verbatim (no "12345"->int). Unclosed-box recovery for budget-truncated calls. Wired into family selection ('glm'/'colibri'), the model-agnostic ToolCallParser path, and both strip_tool_calls_from_content paths. - manager: colibri no longer seizes the whole GPU like ds4. It pins only a configurable expert tier (CUDA_EXPERT_GB) and streams the rest, so it coexists and is evicted like any other model. VRAM footprint estimated from cuda_expert_gb (+overhead) until measured. Verified: typed/untyped parse, strip, unclosed recovery, non-GLM gating, and the by-name dispatcher all correct. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Add JustVugg/colibri as a managed engine, mirroring the ds4 integration but driving the pure-C `colibri` binary DIRECTLY over its stdin/stdout mux protocol (docs/serve_protocol.md) instead of proxying to a Python gateway — coderai reproduces openai_server.py's engine client and GLM-5.2 chat template itself. - config: ColibriConfig (disabled by default; model is a directory container, not a GGUF — routes by model_id/alias/name, no arch sniff) - codai/api/colibri_worker.py: clone+build (make colibri CUDA=1), MuxEngine protocol client (READY handshake, SUBMIT/DATA/DONE, KV-slot pool, CANCEL, stderr log pump), per-container engine registry - codai/backends/colibri.py: ColibriBackend + render_chat (byte-exact port of colibri's GLM-5.2 template; the engine tokenizes what we send) - front routing: `colibri` capability on nvidia/cuda/auto; router/assignment/ app/engine_supervisor thread config.colibri - manager: colibri_should_handle, backend selection, /v1/models surfacing, exclusive-VRAM eviction (wants the whole GPU like ds4) - admin: config get/set + per-model overrides; settings.html card + models.html row - packaging: build.sh --colibri, OCI bundle (repo+binary, no 372GB model), entrypoint seed, CODERAI_COLIBRI_DIR, smoke test Ships OFF: no routing changes until colibri.enabled + colibri.model_path are set. Verified offline: config round-trip, routing predicate, render_chat byte-match vs colibri's own, MuxEngine end-to-end against a fake engine, CPU engine builds. Co-Authored-By:
-