These tests lagged behind two refactors:

- build_studio_catalog is now an async coroutine. Update the studio tests to
  await it: direct-call tests wrap it in asyncio.run(), and the HTTP tests'
  monkeypatched stubs are now async (the route handler awaits the result, so a
  sync stub returning a dict raised "object dict can't be used in 'await'
  expression").
- The providers page bootstraps provider data from a dedicated JSON <script>
  element (providers_bootstrap_json, escaped by the route's _json_parse_bootstrap
  helper) parsed via JSON.parse, instead of an inline `let providersData = {...}`
  literal. Update the admin bootstrap test to feed providers_bootstrap_json and
  assert on the <script id="providers-bootstrap"> element (escaping + double
  JSON decode) and the JSON.parse usage.
- Add the missing record_dashboard_event method to MarketReferenceImportDbStub,
  which the market import route now calls.

tests/routes now passes 86/86 (was 18 failing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

User-scoped dashboard route tests set a session cookie for a synthetic user but
only stubbed the database in the route-handler modules — not the one the auth
middleware consults. The middleware's "account deleted" guard
(get_user_by_id(user_id) is falsy -> redirect to /dashboard/login) therefore
bounced every request until httpx raised TooManyRedirects, failing ~32 tests.

Add tests/routes/conftest.py with an autouse fixture that wraps
DatabaseRegistry.get_config_database so get_user_by_id returns a present user
ONLY when the lookup comes from the auth middleware (detected via call stack).
Direct database assertions in other tests (e.g. signup cleanup, which expects
deleted users to read back as None) keep observing the real database, and tests
that install their own get_config_database stub still win.

Route suite: TooManyRedirects failures 32 -> 0. Remaining route failures are
unrelated pre-existing issues (studio tests await-ing coroutines, a handler
awaiting a dict, a stub missing record_dashboard_event, template bootstrap
content drift) and are out of scope for this fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The broker was refactored (file-based session persistence -> shared-cache
persistence; WebSocket handshake -> client-speaks-first), leaving 5 tests
failing against APIs that no longer exist. Update them to the current design:

- WebSocket tests now send op=register first, then expect event=registered
  (the server no longer greets before the client registers).
- Remote-node queue test simulates a session owned by another cluster node via
  a direct shared-cache entry (foreign broker_node_id, no local WebSocket), so
  send_request() correctly takes the queue slow path instead of a websocket
  send. The old approach (touch broker_node_id) no longer works because the
  cache always records the owning node.
- Replace the obsolete file-based persistence tests (_persist_sessions_locked /
  _load_persisted_sessions / _state_path, all removed) with cache-based
  persistence tests: cross-node session visibility and the offline tombstone.

Full file now passes 11/11 (was 6 passed / 5 failed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Move the CoderAI broker warm-up handling into the provider so it applies on
every request path (rotation, autoselect and direct), and only for the
broker-session "cooling down" error:

- coderai.py: handle_request now wraps the broker decision in a warm-up retry
  loop — on "No active CoderAI broker session" it waits CODERAI_WARMUP_WAIT_
  SECONDS (10s) and retries the same provider up to CODERAI_MAX_WARMUP_WAITS
  (3) times, then surfaces the error WITHOUT recording a failure. Streaming
  and native-proxy paths get the same treatment via _broker_request_with_warmup
  (lazy generators retry on first broker contact). Other errors still record a
  failure as before.
- handlers.py: every place that could disable a provider on a caught error now
  skips record_failure() for the CoderAI warm-up condition (direct chat, direct
  streaming, rotation streaming, audio/TTS/image/embeddings, and the rotation
  retry loop), via the shared _is_coderai_warmup_error() helper. The rotation
  loop no longer sleeps itself (the provider already waited) — it just fails
  over without recording a failure.
- Default CoderAI request timeout raised from 5 min to 3 hours
  (CODERAI_DEFAULT_REQUEST_TIMEOUT, overridable via coderai_config.request_
  timeout); the timeout is also applied to the direct OpenAI client.

Bump version to 0.99.77.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CoderAI workers often run on small/edge hardware that drops its broker
session while cooling down to avoid overheating. A missing broker session
("No active CoderAI broker session...") was treated as a hard error:
record_failure() was called in both the provider handler and the rotation
loop, so three of them in one request disabled the provider for a 300s
cooldown — exactly when it just needed a moment to warm back up.

Now this transient condition is handled gracefully:
- coderai.py no longer records a failure when the error is a missing broker
  session (in any request path), so it never counts toward the disable
  threshold.
- The rotation handler detects the CoderAI warm-up condition and waits
  CODERAI_WARMUP_WAIT_SECONDS (10s) before retrying the same provider, up to
  CODERAI_MAX_WARMUP_WAITS (3) times, fully transparent to the client. Only
  after that does it fail over to the next provider — still without recording
  a failure.

Bump version to 0.99.76.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The provider, rotation and autoselect detail forms were single huge forms.
Split them into collapsible sub-panels so only basic settings stay visible:

- Shared subPanel()/toggleSubPanel() helper + CSS in base.html, defined
  before the content block so page scripts can call it during first render.
- Providers: basic fields visible; Authentication, Pricing & Tiers, Rate
  limits & defaults, Feature Overrides, Native Caching, Models as panels;
  each model has basic config visible + an Advanced sub-panel.
- Rotations: basic visible; Advanced configuration panel; one panel per
  provider (model-count + usage badge); one panel per model with an
  Advanced sub-panel.
- Autoselect: basic visible; Classification, Feature Overrides, Default
  settings panels; one panel per available model.
- Panel open/close state and top-level item expansion persist via
  sessionStorage, so they survive navigation within a session.

Bump version to 0.99.75.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A 402 (e.g. Kilo "usage_limit_exceeded"/out of credits) is deterministic:
retrying the same provider only burns the failure budget and needlessly
disables it within a single request. Add 402 to the non-retryable set so
the rotation fails over to the next provider immediately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- /api/v1/models/{model_id} and /api/u/{username}/models/{model_id} used a
  single-segment param, so model IDs containing '/' (e.g. rotations/lisa)
  returned 404. Use {model_id:path}.
- Add capability-probe endpoints so OpenAI/Ollama/llama.cpp-style clients stop
  getting 404/405 on connect: GET /api/version, /v1/version, /version;
  GET /api/tags (Ollama model list); GET /props, /api/props, /v1/props.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- anthropic: previously ignored the stream flag entirely (always non-stream).
  Add _handle_streaming_request converting Anthropic SSE events into
  OpenAI-compatible chunks (text deltas, tool_use/input_json deltas, usage).
- ollama: previously hard-coded stream=False. Add incremental streaming over
  /api/generate yielding OpenAI chunks.
- google: was buffering the entire response before yielding. Stream chunks
  incrementally as they arrive when no tools are requested (the common case);
  keep the buffer-and-parse path only when tools are present, since the
  text-encoded tool-call detection needs the complete response.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

For streaming requests providers returned a lazy generator and immediately
called record_success(), resetting the failure counter to 0 BEFORE the
rotation handler primed the stream and the upstream returned 400/429. Every
attempt did success(reset->0) then failure(->1), so the disable threshold (3)
was never reached and a consistently-failing provider was never disabled.

Guard the premature record_success() so it does not fire for streaming
(codex, openai, runpod, coderai). The caller already records success after
the stream is primed/consumed, so the failure counter now accumulates and a
failing provider is correctly disabled/cooled-down. Bump to 0.99.74.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Adds a per-provider Disable/Enable button to the rotations page for both admin
and users, exposing a manual disable that — unlike the failure cooldown — never
auto-expires and persists across reboots.

- providers/base.py: manual_disable()/manual_enable()/is_manually_disabled();
  manual disable takes precedence in is_rate_limited(); manual_enable() also
  clears any cooldown and resets the failure budget. Source of truth is the
  database (mirrored to cache as a fast path) so state survives restarts and
  cache flushes. Also: when a failure cooldown elapses, the provider is
  reactivated with a fresh failure budget instead of a hair-trigger.
- database.py: dedicated provider_manual_disabled table (+migration) and
  get/set/clear helpers, user-scoped so a user's toggle never affects others.
- dashboard: manual-disable / manual-enable / bulk manual-status endpoints; the
  rotations pages render and toggle the button state.

Bump to 0.99.73.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Each internal-model functionality can now point at a rotation or autoselect
instead of the local model, falling back to the local model when the override is
empty or fails. Per-functionality fields in internal_model (default empty):
condensation_override, autoselect_override, nsfw_classifier_override,
privacy_classifier_override.

- handlers.py: run_meta_target() runs a global rotation/autoselect and returns
  the assistant text (or None -> fallback); classification overrides prompt the
  chat model for a strict YES/NO and parse it, falling back to the local
  classifier; autoselect selection can run through an override.
- context.py: condensation prefers the override, falling back to the local model.
- settings page: per-field override inputs backed by a datalist of available
  rotations/autoselects; saved to internal_model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A rotation provider entry whose provider_id is "rotations" or "autoselect" is a
meta-provider: each model "name" is the id of a target rotation/autoselect, and
the request is forwarded there (weighted like any other entry, with failover to
the next entry on total failure).

- handlers.py: detect meta-providers in model-building, delegate in the retry
  loop, and resolve rotation vs autoselect targets.
- Loop protection: a delegation chain rides on request_data and is registered by
  both RotationHandler and AutoselectHandler; re-entering an id already in the
  chain (A->B->A) or exceeding depth 8 raises HTTP 508 (caught upstream -> fail
  over). Loops are also pre-filtered at model-build time.
- Dashboard: the provider select in rotations.html / user_rotations.html now
  offers "rotations"/"autoselect"; the model row becomes a target picker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

When a rotation fails over to a different provider/model than the preferred
(highest-weight) pick, optionally notify the client so the provider/model
change isn't silent. Opt-in via a new notify_on_failover flag, settable
globally (rotations.json top-level) or per-rotation, mirroring notifyerrors.

On a real switch (served provider/model differs from preferred), three
mechanisms carry the notice:
- Visible banner: prepended to message content (non-streaming) or a leading
  delta chunk (streaming).
- JSON metadata: an aisbf_failover object on the body / leading stream chunk.
- HTTP headers: X-AISBF-Failover/Provider/Model/Preferred-* on the
  StreamingResponse and (for non-streaming) at the rotation route.

Applied to both the normal and chunked rotation paths. The banner is injected
after caching so cache hits don't replay a stale notice; no switch means zero
overhead and nothing emitted; default off preserves existing behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Streaming rotation requests returned the provider's lazy async generator to
the client before the upstream HTTP call was made, so an immediate provider
error (e.g. HTTP 400 "model not supported") surfaced mid-stream after the
200 OK headers were already sent — leaking the error to the client instead
of failing over.

- Prime streaming responses (_prime_stream) inside the rotation retry loop so
  immediate provider errors are raised within the try/except and trigger
  failover to the next provider by weight, transparently to the client.
- Add _is_non_retryable_error to skip pointless same-provider retries for
  client errors (400/401/403/404/405/422) and move straight to the next
  provider; 5xx/timeouts/network remain retryable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add _record_dashboard_proxy_event, _settle_market_result,
_get_market_source_details, _market_request_id and
_extract_usage_from_sse_chunk to RotationHandler, and the market
settlement cluster to AutoselectHandler. These were defined only on
RequestHandler, so rotation/autoselect requests raised AttributeError
(crash on the dashboard event path, silently swallowed elsewhere),
dropping market settlement and streaming usage capture.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The collapsed provider-list status chip only rendered gpus[0], so a
coderai client connecting with multiple GPUs displayed just the first.
Render one chip per reported GPU (name + per-GPU VRAM), falling back to
an aggregate VRAM chip when no per-GPU detail is sent. GPU names are
HTML-escaped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>