embeddings 300ms default + admin GUI throttle fields; radeon crash-loop breaker

Embedding admission gate:
- default min-interval 200ms -> 300ms (paces GPU starts harder)
- expose embed_max_concurrency / embed_max_backlog / embed_min_interval_ms
  per-model in the admin model editor (embedding-gated section), round-tripped
  through /admin/api/model-configure into models.json

Engine supervisor circuit breaker:
- quarantine an engine that exits crashloop_max (5) times within
  crashloop_window (120s) instead of respawning it ~1/s forever. A GPU that has
  fallen off the bus (Polaris secondary-bus-reset bug) makes its engine die on
  every launch; the breaker takes radeon cleanly out of routing (embeddings fail
  fast) instead of hammering a dead card. Manual restart_engine() clears it.

Pairs with host-side RX580 mitigations (dpm=high lock, lockup_timeout=25s,
conc=1/300ms on the 3 vulkan embedding models, reset watchdog + thermal sampler).

REQUEST-geoclip-embeddings.md

+# Request: serve GeoCLIP through the OpenAI embeddings API
+
+**From:** HomeHunter (property search, `192.168.42.44`)
+**To:** whoever maintains CoderAI (`192.168.42.3:8000`)
+**Status:** request — nothing is built on the HomeHunter side yet that depends on it
+
+## What we need in one line
+
+Two new model ids on the existing `POST /v1/embeddings` endpoint — `geoclip` for
+images and `geoclip-location` for coordinates — both returning 512-dimension
+L2-normalised vectors in the **same** space.
+
+No new endpoint, no custom protocol. If you serve those two ids, HomeHunter can
+use them with the client it already has.
+
+## Why
+
+HomeHunter locates listings whose source published no coordinate. Today it asks a
+vision model to *describe* each photo, then geocodes the words. That loses almost
+all the signal: only photos containing something nameable (a street sign, a
+famous landmark) produce anything, and a name geocodes to a coarse or ambiguous
+point. Measured against ground truth, the text pipeline places **1.7%** of
+listings.
+
+GeoCLIP removes the text hop. It embeds images and GPS coordinates into one
+shared space, so a photo can be scored *directly* against candidate coordinates —
+the match itself is the answer. We would build a grid of candidate points over
+the listing's suburb, embed the photo once, and take the highest-scoring cell.
+
+We already tried the cheap alternative and it failed: nearest-neighbour retrieval
+over DINOv2 vectors fired on 2.5% of listings and missed by up to 288 km, because
+a *similarity* model is not a *location* model. That is why the ask is
+specifically GeoCLIP and not "some image embedding".
+
+## The contract
+
+### 1. Image model — `geoclip`
+
+Identical in shape to how `dinov2-large` is already served.
+
+```http
+POST /v1/embeddings
+Authorization: Bearer <token>
+Content-Type: application/json
+
+{"model": "geoclip", "input": "data:image/jpeg;base64,/9j/4AAQ...", "dimensions": 512}
+```
+
+```json
+{"data": [{"embedding": [0.013, -0.044, ...], "index": 0}], "model": "geoclip"}
+```
+
+### 2. Location model — `geoclip-location`
+
+The only unusual part: the "text" being embedded is a coordinate pair. The wire
+format stays completely standard.
+
+```http
+POST /v1/embeddings
+
+{"model": "geoclip-location",
+ "input": ["-33.9249,18.4241", "-33.9270,18.4300", "-33.9310,18.4185"],
+ "dimensions": 512}
+```
+
+```json
+{"data": [{"embedding": [...], "index": 0},
+          {"embedding": [...], "index": 1},
+          {"embedding": [...], "index": 2}], "model": "geoclip-location"}
+```
+
+Input format: `"<latitude>,<longitude>"` in decimal degrees, WGS84. Please accept
+optional surrounding whitespace. A single string (not an array) should also work.
+
+## Requirements that actually matter
+
+1. **Same space.** The two models must be the image and location encoders of the
+   *same* GeoCLIP checkpoint. Vectors from different checkpoints are not
+   comparable and the scores become meaningless without erroring — the worst
+   kind of failure.
+2. **L2-normalise both.** We compare with a plain dot product. If you do not
+   normalise, say so and we will normalise our side instead — but please pick one
+   and document it.
+3. **Batching on the location model.** This is the one that matters for cost: we
+   score a few hundred candidate cells per listing and want that in **one**
+   request, not one per cell. `input` as an array must return one vector per
+   element, in input order.
+4. **`index` must reflect input order** (or at least be present so we can sort by
+   it). We map results back to grid cells positionally.
+5. **Deterministic.** Same input, same vector. We cache by coordinate.
+
+Nice to have, not required: report the true width in `usage`/`dimensions`, and
+list both ids in `GET /v1/models` so we can detect availability.
+
+## Reference implementation
+
+The `geoclip` pip package ships both encoders and the weights (~1.7 GB, first
+load downloads them).
+
+```python
+import torch, torch.nn.functional as F
+from geoclip import GeoCLIP
+
+model = GeoCLIP().to(device).eval()
+
+@torch.inference_mode()
+def embed_image(pil_image):
+    batch = model.image_encoder.preprocess_image(pil_image).to(device)
+    return F.normalize(model.image_encoder(batch), dim=-1)[0].tolist()   # 512 floats
+
+@torch.inference_mode()
+def embed_locations(pairs):            # [(lat, lon), ...]
+    gps = torch.tensor(pairs, dtype=torch.float32, device=device)
+    return F.normalize(model.location_encoder(gps), dim=-1).tolist()     # [[512 floats], ...]
+```
+
+The location encoder is tiny (an MLP over random Fourier features) — it is the
+image encoder that carries the weight and wants the GPU. If it helps, a fuller
+sketch including a FastAPI wrapper is in the HomeHunter repo at
+`deploy/geoclip/app.py`; it was written before we settled on serving this through
+CoderAI, so treat it as reference for the two torch calls only.
+
+## How we will verify it
+
+```sh
+# 1. image side returns 512 dims
+curl -s $GW/v1/embeddings -H "Authorization: Bearer $TOK" \
+  -d '{"model":"geoclip","input":"data:image/jpeg;base64,'"$(base64 -w0 photo.jpg)"'"}' \
+  | jq '.data[0].embedding | length'
+
+# 2. location side batches, in order
+curl -s $GW/v1/embeddings -H "Authorization: Bearer $TOK" \
+  -d '{"model":"geoclip-location","input":["-33.92,18.42","-33.90,18.60"]}' \
+  | jq '[.data[] | {index, n: (.embedding|length)}]'
+```
+
+Then the real test, which is ours to run: a photo taken in a known place should
+score its true location higher than a location 20 km away. If that ordering does
+not hold, the two encoders are not in the same space.
+
+After that we measure it against ground truth (listings whose source published a
+coordinate, re-located with that coordinate hidden). The bar is the
+suburb-centroid baseline, currently **~1.1 km median error**. If GeoCLIP cannot
+beat that, we will not ship it — the same way we rejected the DINOv2 attempt.
+
+## What we do not need
+
+- No `/locate` or top-k prediction endpoint. GeoCLIP's built-in worldwide gallery
+  answers "where on Earth", but our listings already claim a suburb; the useful
+  question is *where inside that area*, which we answer ourselves by scoring our
+  own grid.
+- No new auth, no streaming, no changes to any existing model.
+
+## Contact
+
+Questions about the HomeHunter side, the grid, or the evaluation: ask in the
+HomeHunter repo (`/working/homehunter`, see
+`docs/geolocation-visual-matching-design.md` for the full design and
+`app/services/geo_eval.py` for how accuracy is measured).

REQUEST-vpr-embeddings.md

+# Request: serve a visual place recognition (VPR) model
+
+**From:** HomeHunter (property search, `192.168.42.44`)
+**To:** whoever maintains CoderAI (`192.168.42.3:8000`)
+**Status:** request — the HomeHunter side is already built and waiting on the model
+**Follows:** `REQUEST-geoclip-embeddings.md` (delivered — thank you, it works)
+
+## What we need in one line
+
+One model id on the existing `POST /v1/embeddings` endpoint — **`mixvpr`** (or
+SALAD/EigenPlaces, your choice) — that turns an image into a single vector
+trained so that **two photos of the same place land close together**.
+
+Same request shape as `dinov2-large` already uses. Nothing else changes.
+
+## Why, and why not the models you already serve
+
+We are trying to find a listing's **actual street address** by matching its
+exterior photo against Google Street View panoramas: the matching panorama's own
+coordinate *is* the address. The matching code, the panorama cache and the spend
+caps are already built and deployed. The only missing piece is a model that can
+reliably answer "is this the same building?"
+
+Nothing currently on the gateway answers that question:
+
+| Model | What it actually does | Why it fails here |
+|---|---|---|
+| **dinov2-large** | generic self-supervised features | Rates *similar-looking* houses as matches. We measured this: nearest-neighbour retrieval over DINOv2 vectors placed one listing **288 km** from truth. |
+| **gme-Qwen2-VL** | text↔image semantic retrieval | Matches content categories ("a house with a pool"), not building identity. |
+| **geoclip** (just added) | image → global GPS prior | Separates continents, not streets. Measured: a Cape Town photo scores Cape Town **0.190**, London **-0.064**, but a point **16 km away in the same metro still 0.185**. Median error 1.98 km against a 1.17 km baseline — worse than doing nothing. Now disabled. |
+
+VPR models are trained specifically for the discrimination the others lack:
+same place under different viewpoint, lighting, weather and season.
+
+## The contract
+
+```http
+POST /v1/embeddings
+Authorization: Bearer <token>
+Content-Type: application/json
+
+{"model": "mixvpr", "input": "data:image/jpeg;base64,/9j/4AAQ...", "dimensions": 4096}
+```
+
+```json
+{"data": [{"embedding": [0.021, -0.008, ...], "index": 0}], "model": "mixvpr"}
+```
+
+Exactly the shape `dinov2-large` already answers in. We send a JPEG data URI
+(downscaled to 1024 px on our side) and expect one vector back.
+
+## Requirements that matter
+
+1. **L2-normalise the output.** We compare with a plain dot product.
+2. **Tell us the true width** and keep it stable — MixVPR is typically 4096,
+   SALAD 8448, EigenPlaces 2048. We set it in config and validate the response
+   length, so a mismatch is rejected rather than silently stored.
+3. **Deterministic.** Same image, same vector. We cache panorama embeddings by
+   location and reuse them across listings.
+4. **One vector per image** (a pooled/aggregated descriptor, not patch tokens).
+
+Any of these are fine, in rough order of preference:
+
+| Model | Dim | Note |
+|---|---|---|
+| **MixVPR** | 4096 | Strong, small, single forward pass. Good default. |
+| **SALAD** (DINOv2-SALAD) | 8448 | Currently near state of the art; heavier. |
+| **EigenPlaces / CosPlace** | 512–2048 | Lighter, still far better than raw DINOv2. |
+| AnyLoc | — | Only if patch tokens are easy for you: it is DINOv2 features + VLAD, so it could reuse the DINOv2 you already serve. |
+
+All are open weights on GitHub/HuggingFace and much smaller than GeoCLIP.
+
+## How we will verify it
+
+The decisive test is **ordering**, not absolute scores. Given a listing's
+exterior photo:
+
+```
+similarity(listing photo, Street View AT that address)
+    >  similarity(listing photo, Street View 200 m down the same street)
+    >> similarity(listing photo, Street View in a different suburb)
+```
+
+DINOv2 fails this — it ranks any pleasant suburban facade about equally. If the
+model you pick reproduces that ordering, it is the right one.
+
+Then the real test, which is ours to run: Street View matching is measured
+against ground truth (listings whose source published a coordinate, re-located
+with that coordinate hidden). The bar is the **suburb-centroid baseline, 1.17 km
+median**. GeoCLIP was wired in, measured, and left disabled for failing it. VPR
+gets the same treatment — no model ships on plausibility.
+
+## What we do not need
+
+- No new endpoint, auth, or streaming.
+- No changes to any existing model.
+- Not a VLM or a captioner — we are not asking it to describe anything.
+
+## What HomeHunter is already configured for
+
+The `vpr` provider is set up and pointing at the **same gateway DINOv2 uses**
+(`http://192.168.42.3:8000/v1`, plain model ids like `dinov2-large-Q8_0.gguf` and
+`geoclip`), currently expecting:
+
+    model_id: salad        dimensions: 8448     <- active
+    model_id: eigenplaces  dimensions: 2048     <- alternative, swap both fields
+
+Serving either id is enough; the model id and width are editable in the admin, so
+any other place-recognition model works too as long as the pair matches. If the
+model is missing the matcher logs it and falls back to DINOv2 rather than
+disabling Street View matching, so configuring ahead of you is harmless.
+
+## Contact
+
+HomeHunter side: `/working/homehunter`. The matcher is
+`app/services/streetview.py` (already switches to a `vpr` provider the moment one
+is configured, falling back to DINOv2 otherwise), the design rationale is
+`docs/geolocation-visual-matching-design.md`, and accuracy is measured by
+`scripts/eval_locations.py`.

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.1.82"
+__version__ = "0.1.83"
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
 # expandable_segments lets the allocator return freed pages to the driver even

codai/admin/routes.py

@@ -2622,6 +2622,7 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                 "component_quantization", "output_crf", "force_vram_update",
                 "balanced_gpu_percent", "acceleration",
                 "cache_type_k", "cache_type_v", "kv_offload", "n_batch", "n_ubatch", "n_seq_max",
+                "embed_max_concurrency", "embed_max_backlog", "embed_min_interval_ms",
                 "gpu_split", "tensor_split", "split_strategy", "split_secondary_cap_gb",
                 "turboquant", "engine", "engine_fallback",
                 "quant_backend", "kv_cache_budget_mb", "kv_cache_slots", "mmproj",

codai/admin/templates/models.html

@@ -961,6 +961,28 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
         </div>
       </div>
 
+      <!-- Embedding throttle (embedding models) — protect fragile GPUs from request floods -->
+      <div id="cfg-embed-throttle-section" style="display:none">
+        <div class="card-title" style="margin-top:1.25rem">Embedding throttle <span class="muted" style="font-weight:normal">(admission gate — bounds concurrency, sheds bursts, paces GPU starts)</span></div>
+        <div style="display:flex;gap:1rem;flex-wrap:wrap">
+          <div class="form-row" style="max-width:200px">
+            <label class="form-label">Max concurrency</label>
+            <input type="number" id="cfg-embed-max-concurrency" class="form-input" min="1" step="1" placeholder="auto (2)">
+            <span class="form-hint">simultaneous embeds on the GPU (1 = strict serialise; use for fragile cards like Polaris/RX580)</span>
+          </div>
+          <div class="form-row" style="max-width:200px">
+            <label class="form-label">Max backlog</label>
+            <input type="number" id="cfg-embed-max-backlog" class="form-input" min="0" step="1" placeholder="auto (32)">
+            <span class="form-hint">requests queued beyond concurrency before returning 429 (0 = never shed)</span>
+          </div>
+          <div class="form-row" style="max-width:200px">
+            <label class="form-label">Min interval (ms)</label>
+            <input type="number" id="cfg-embed-min-interval-ms" class="form-input" min="0" step="10" placeholder="auto (300)">
+            <span class="form-hint">minimum gap between GPU starts; paces a flooding client (0 = no pacing)</span>
+          </div>
+        </div>
+      </div>
+
       <!-- components -->
       <div class="card-title" style="margin-top:1.25rem">Components</div>
       <div class="form-row">
@@ -3350,6 +3372,9 @@ function openCfgModal(idx, cfgIdx){
   document.getElementById('cfg-n-ctx').value = nCtxForEst;
   document.getElementById('cfg-n-batch').value = s.n_batch != null ? s.n_batch : '';
   document.getElementById('cfg-n-seq-max').value = s.n_seq_max != null ? s.n_seq_max : '';
+  document.getElementById('cfg-embed-max-concurrency').value = s.embed_max_concurrency != null ? s.embed_max_concurrency : '';
+  document.getElementById('cfg-embed-max-backlog').value = s.embed_max_backlog != null ? s.embed_max_backlog : '';
+  document.getElementById('cfg-embed-min-interval-ms').value = s.embed_min_interval_ms != null ? s.embed_min_interval_ms : '';
   document.getElementById('cfg-cache-type-k').value = s.cache_type_k || '';
   document.getElementById('cfg-cache-type-v').value = s.cache_type_v || '';
   _populateMmprojSelect(m, s);
@@ -3616,8 +3641,11 @@ function _turboQuantApplies(){
     .some(cb => cb.value === 'embedding_models');
 }
 function _refreshTurboQuantVisibility(){
+  const applies = _turboQuantApplies();
   const section = document.getElementById('cfg-turboquant-section');
-  if (section) section.style.display = _turboQuantApplies() ? '' : 'none';
+  if (section) section.style.display = applies ? '' : 'none';
+  const thr = document.getElementById('cfg-embed-throttle-section');
+  if (thr) thr.style.display = applies ? '' : 'none';
 }
 function onTurboQuantToggle(){
   const on = document.getElementById('cfg-tq-enabled').checked;
@@ -3816,6 +3844,9 @@ async function saveModelConfig(){
     n_ctx:             parseInt(document.getElementById('cfg-n-ctx').value) || 2048,
     n_batch:           parseInt(document.getElementById('cfg-n-batch').value) || null,
     n_seq_max:         parseInt(document.getElementById('cfg-n-seq-max').value) || null,
+    embed_max_concurrency: document.getElementById('cfg-embed-max-concurrency').value === '' ? null : parseInt(document.getElementById('cfg-embed-max-concurrency').value),
+    embed_max_backlog:     document.getElementById('cfg-embed-max-backlog').value === '' ? null : parseInt(document.getElementById('cfg-embed-max-backlog').value),
+    embed_min_interval_ms: document.getElementById('cfg-embed-min-interval-ms').value === '' ? null : parseInt(document.getElementById('cfg-embed-min-interval-ms').value),
     cache_type_k:      document.getElementById('cfg-cache-type-k').value || null,
     cache_type_v:      document.getElementById('cfg-cache-type-v').value || null,
     mmproj:            document.getElementById('cfg-mmproj').value || null,

codai/api/embeddings.py

@@ -61,12 +61,12 @@ def set_global_args(args):
 # runs wide open. Keys / envs / defaults:
 #   embed_max_concurrency  CODERAI_EMBED_MAX_CONCURRENCY  (default 2)
 #   embed_max_backlog      CODERAI_EMBED_MAX_BACKLOG      (default 32; 0 = no shed)
-#   embed_min_interval_ms  CODERAI_EMBED_MIN_INTERVAL_MS  (default 200 — paces GPU
+#   embed_min_interval_ms  CODERAI_EMBED_MIN_INTERVAL_MS  (default 300 — paces GPU
 #       starts so the ring gets breathing room between ops)
 # Each distinct model id gets its OWN gate (semaphore + backlog counter + pacing).
 _EMBED_CONC_DEFAULT = 2
 _EMBED_BACKLOG_DEFAULT = 32
-_EMBED_INTERVAL_MS_DEFAULT = 200
+_EMBED_INTERVAL_MS_DEFAULT = 300
 _embed_gates: dict = {}          # model key -> {sem, conc, inflight, last_start}
 _embed_gate_lock = asyncio.Lock()
 

codai/frontproxy/engine_supervisor.py

@@ -136,6 +136,8 @@ class EngineSupervisor:
         self._poll_thread = None
         self._logs = {}   # engine_id -> deque tail
         self._restart_lock = threading.RLock()
+        self._restart_times = {}         # engine_id -> [exit epoch,...] (crash-loop window)
+        self._quarantined = set()        # engine_ids taken out of respawn (GPU lost, etc.)
         # Serialise terminal writes across engine pump threads + track whether the
         # last thing we printed was an in-place tqdm progress line (so the next
         # normal line finalises it with a newline).
@@ -910,6 +912,10 @@ class EngineSupervisor:
             self._push_assignment_if_changed(client)
             self._flush_pending_reloads(client)   # deliver queued reloads to idle engines
             for engine in self.registry.all():
+                # A quarantined engine (repeated crash-loop; e.g. GPU off-bus) stays
+                # down until a manual restart_engine() revives it — don't poll/respawn.
+                if engine.id in self._quarantined:
+                    continue
                 # Respawn engines whose process has exited.
                 if engine.proc is not None and engine.proc.poll() is not None:
                     self._maybe_restart(engine)
@@ -973,10 +979,32 @@ class EngineSupervisor:
         with self._restart_lock:
             if self._stopped.is_set():
                 return
+            if engine.id in self._quarantined:
+                return
             code = engine.proc.poll() if engine.proc else None
             tail = " | ".join(list(self._logs.get(engine.id, []))[-3:])
-            print(f"[front] engine#{engine.id} exited (code {code}); respawning. {tail}",
-                  flush=True)
+            # Circuit breaker: a GPU that has fallen off the bus (e.g. the Polaris
+            # secondary-bus-reset bug) makes its engine exit on every launch. Without
+            # a breaker the supervisor respawns it ~1/s forever — hammering a dead
+            # card and flooding the log. Quarantine after `crashloop_max` exits within
+            # `crashloop_window` seconds; a manual restart_engine() clears it.
+            now = time.time()
+            window = float(getattr(self.config.server, "engine_crashloop_window", 120.0) or 0.0)
+            maxn = int(getattr(self.config.server, "engine_crashloop_max", 5) or 0)
+            hist = self._restart_times.setdefault(engine.id, [])
+            if window > 0:
+                hist[:] = [t for t in hist if now - t < window]
+            hist.append(now)
+            if maxn > 0 and len(hist) >= maxn:
+                self._quarantined.add(engine.id)
+                self.registry.update_state(engine.id, healthy=False)
+                print(f"[front] engine#{engine.id} ({engine.name}) exited {len(hist)}x "
+                      f"in {window:.0f}s (code {code}) — QUARANTINED, not respawning. "
+                      f"GPU likely lost (reset/off-bus); reboot or manual restart "
+                      f"needed. {tail}", flush=True)
+                return
+            print(f"[front] engine#{engine.id} exited (code {code}); respawning "
+                  f"({len(hist)}/{maxn} in {window:.0f}s). {tail}", flush=True)
             self.registry.update_state(engine.id, healthy=False)
             time.sleep(1.0)   # avoid a tight crash loop
             self._spawn(engine)
@@ -997,6 +1025,11 @@ class EngineSupervisor:
             drain_grace = float(getattr(self.config.server,
                                         "engine_restart_drain_grace", 30.0) or 0.0)
         with self._restart_lock:
+            # A manual restart is an explicit "try again" — clear any crash-loop
+            # quarantine and history so the engine is respawned below (e.g. after the
+            # GPU has been power-cycled / recovered).
+            self._quarantined.discard(engine_id)
+            self._restart_times.pop(engine_id, None)
             # If we'd thermally frozen it, wake it so it can drain in-flight work and
             # honour SIGTERM (a stopped process ignores both until continued).
             self._thermal_resume_if_frozen(engine)