When a coderai broker provider's worker is offline, requests are held and
retried while it warms up. The hold window was hardcoded (10s x 3 = 30s).
Expose it per provider via coderai_config:
- broker_queue_timeout_seconds: total hold time (0 disables holding)
- broker_warmup_wait_seconds: poll interval between retries
- broker_max_warmup_waits: explicit retry count (when no queue timeout)

Defaults preserve prior behavior. Applied to both warm-up retry loops.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Extend max output token resolution so per-model and default values are
honored at every layer when the client omits max_tokens.

Effective priority (client value always wins if present):
  autoselect per-model -> rotation per-model -> provider per-model
  -> provider default -> rotation default -> autoselect default

- rotation path: consult the selected provider's config (new
  RotationHandler._get_provider_config) so a provider default_max_tokens
  applies even though rotations can't configure max_tokens themselves
- AutoselectModelInfo.max_tokens: new per-model override field
- AutoselectHandler._apply_autoselect_max_tokens applied in both the
  streaming and non-streaming dispatch: per-model override set directly
  (highest), autoselect default threaded via _autoselect_default_max_tokens
  as a lowest-priority fallback
- rotation and both RequestHandler injections consume the threaded
  autoselect default, covering autoselect->rotation and
  autoselect->provider/model dispatch routes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Provider-level defaults were ignored when a provider had no per-model
configs, so coderai/broker providers reported a fixed 4096 context window
and clients fell back to their own defaults.

- model list (handlers.py): fall back to provider_config.default_context_size
  for context_window/context_length before inferring
- rotation/autoselect model list (routes/api.py): attach context_window/
  context_length resolved from rotation/autoselect/provider defaults (was
  absent entirely, causing clients to default to 4096)
- handle_rotation_model_list: honor provider default_context_size before
  heuristic auto-derivation
- max output tokens: add ProviderModelConfig.max_tokens and
  default_max_tokens on Provider/Rotation/Autoselect configs, plus
  get_max_completion_tokens_for_model resolver, applied as a fallback in
  all request paths when the client omits max_tokens
- get_max_request_tokens_for_model: fall back to
  provider_config.default_max_request_tokens; drop duplicated dead block
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Streaming was scaffolded (stream_queue, _publish_stream_response,
wait_for_stream_event, _iter_broker_stream_chunks) but send_request always awaited
the terminal future and then popped the pending entry in finally — so by the time
_broker_stream began consuming, the stream was over and its queue was gone. Result:
the whole reply arrived at once.

Fixes:
- send_request: for streaming requests, return a {event:"stream_start"} immediately
  and do NOT pop _pending in finally — the consumer drives it. Pop on error.
- _publish_stream_response: queue EVERY event (chunks AND the terminal done) so the
  consumer's loop receives the end marker; still resolves the future for safety.
- finish_stream(): new — removes the streaming request's _pending once drained.
- wait_for_stream_event: simplified to a queue.get (dropped the event_log path).
- providers/coderai._iter_broker_stream_chunks: wrap in try/finally that calls
  finish_stream(request_id) on completion/error; treat stream_start as a no-data
  continue.

Pairs with the coderai-side change that emits chunk/done envelopes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Real bugs surfaced by the retry tests (code referenced schema that did not
exist / wrong table):
- subscriptions was missing cancelled_at (written when a renewal exhausts its
  retries and cancels the subscription).
- payment_retry_queue was missing last_attempt_at (written when scheduling the
  next retry).
- retry.py queried a non-existent `tiers` table for the free tier; corrected to
  account_tiers (and is_default = 1 for cross-DB boolean compatibility).

Migration changes add the two columns to the CREATE statements and provide
idempotent, cross-DB (SQLite PRAGMA / MySQL INFORMATION_SCHEMA) ALTERs for
existing databases.

test_retry: crypto retries are intentionally skipped without incrementing while
the wallet is unfunded, so the increment/downgrade tests now drive a failing
gateway charge instead, and expect the same free tier the code selects.

Verified all payment migration DDL (CREATE + new ALTERs + INFORMATION_SCHEMA
existence checks) executes cleanly on MariaDB/MySQL as well as SQLite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Rewrite the integration tests against the current schema/behaviour:
- crypto flow: process_transaction records the tx itself (drop the manual
  crypto_transactions insert with non-existent from_address/to_address columns;
  ensure a crypto wallet row exists; pass a float amount for SQLite binding).
- subscriptions: use account_tiers + subscriptions (not subscription_tiers/
  user_subscriptions), create the required card payment method, and make the
  renewal subscription actually due (current_period_end in the past); assert via
  process_renewals' processed count.
- consolidation: enable crypto_consolidation_settings so candidates are queued.
- notifications: enable email_notification_settings and add an email_templates
  row so the notification is actually sent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- test_auto_topup: use MagicMock (not Mock) so the DB context manager works in
  auto_charge; mock _get_or_create_customer and return a Stripe-object-shaped
  PaymentIntent (attribute access, not dict); replace the retry-logic test's
  bare AsyncMock with a small stateful fake session so record_auto_topup_attempt
  actually transitions auto_topup_enabled to False after 3 failures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

PaymentService construction now initializes crypto master keys, so the tests
need a real migrated SQLite DB rather than a Mock. Also align with the current
code: initiate_topup delegates to create_topup_intent/create_topup_order; the
Stripe webhook path requires amount in metadata, passes a metadata dict, and is
exercised by patching stripe.Webhook.construct_event; crypto deposits credit the
fiat wallet directly via SQL (assert the resulting balance) using the monitor's
own price service (returning a float for SQLite binding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Several payment code paths referenced database schema that no migration ever
created, so they failed at runtime:

- distributed_locks: created the table used by PaymentScheduler._acquire_lock /
  _release_lock (it was referenced but never defined anywhere).
- email_notification_queue: the queue code uses context_json + retry_count, but
  the table was defined with recipient_email/subject/body. Redefined the table
  to match the code.
- crypto_consolidation_queue: WalletConsolidator writes user_id/from_address/
  to_address/amount, but the table had total_balance/address_count. Redefined to
  match the code.
- payment_methods.gateway: the base DatabaseManager schema creates
  payment_methods without a gateway column, so the migration's CREATE TABLE IF
  NOT EXISTS was a no-op and the column (read by the renewal processor) was
  missing. Added an idempotent ALTER.

Added cross-DB helpers (_table_exists/_column_exists work on both SQLite via
PRAGMA and MySQL via INFORMATION_SCHEMA). The two transient queue tables hold
only regenerable rows, so a legacy-shape table is dropped and recreated rather
than ALTERed column-by-column (avoids fragile cross-dialect constraint changes);
the gateway add is a plain additive ALTER guarded by _column_exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Bring several payments tests back in line with the current implementation:

- test_wallet.py: mock self.db.begin() as an async context manager (was an
  AsyncMock coroutine) for credit/debit; update get_or_create_user_address to
  the new contract (legacy helper now allocates a fresh address per call).
- test_wallet_renewal.py: SubscriptionRenewalProcessor now takes its gateway/
  price collaborators explicitly (pass mocks); patch trigger_auto_topup at its
  real source aisbf.payments.scheduler; renewal DB cursor access is synchronous
  (MagicMock, not AsyncMock).
- test_integration.py: use a valid Fernet encryption key; DatabaseManager takes
  a config dict (not db_type/db_path kwargs); decorate the async payment_service
  fixture with @pytest_asyncio.fixture (strict asyncio mode).
- test_topup.py: use a valid Fernet encryption key.

Payments suite: 31 not-passing (17 failed + 14 errors) -> 22 failed, 52 passed;
remaining failures now surface real causes instead of setup errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

These tests lagged behind two refactors:

- build_studio_catalog is now an async coroutine. Update the studio tests to
  await it: direct-call tests wrap it in asyncio.run(), and the HTTP tests'
  monkeypatched stubs are now async (the route handler awaits the result, so a
  sync stub returning a dict raised "object dict can't be used in 'await'
  expression").
- The providers page bootstraps provider data from a dedicated JSON <script>
  element (providers_bootstrap_json, escaped by the route's _json_parse_bootstrap
  helper) parsed via JSON.parse, instead of an inline `let providersData = {...}`
  literal. Update the admin bootstrap test to feed providers_bootstrap_json and
  assert on the <script id="providers-bootstrap"> element (escaping + double
  JSON decode) and the JSON.parse usage.
- Add the missing record_dashboard_event method to MarketReferenceImportDbStub,
  which the market import route now calls.

tests/routes now passes 86/86 (was 18 failing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

User-scoped dashboard route tests set a session cookie for a synthetic user but
only stubbed the database in the route-handler modules — not the one the auth
middleware consults. The middleware's "account deleted" guard
(get_user_by_id(user_id) is falsy -> redirect to /dashboard/login) therefore
bounced every request until httpx raised TooManyRedirects, failing ~32 tests.

Add tests/routes/conftest.py with an autouse fixture that wraps
DatabaseRegistry.get_config_database so get_user_by_id returns a present user
ONLY when the lookup comes from the auth middleware (detected via call stack).
Direct database assertions in other tests (e.g. signup cleanup, which expects
deleted users to read back as None) keep observing the real database, and tests
that install their own get_config_database stub still win.

Route suite: TooManyRedirects failures 32 -> 0. Remaining route failures are
unrelated pre-existing issues (studio tests await-ing coroutines, a handler
awaiting a dict, a stub missing record_dashboard_event, template bootstrap
content drift) and are out of scope for this fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The broker was refactored (file-based session persistence -> shared-cache
persistence; WebSocket handshake -> client-speaks-first), leaving 5 tests
failing against APIs that no longer exist. Update them to the current design:

- WebSocket tests now send op=register first, then expect event=registered
  (the server no longer greets before the client registers).
- Remote-node queue test simulates a session owned by another cluster node via
  a direct shared-cache entry (foreign broker_node_id, no local WebSocket), so
  send_request() correctly takes the queue slow path instead of a websocket
  send. The old approach (touch broker_node_id) no longer works because the
  cache always records the owning node.
- Replace the obsolete file-based persistence tests (_persist_sessions_locked /
  _load_persisted_sessions / _state_path, all removed) with cache-based
  persistence tests: cross-node session visibility and the offline tombstone.

Full file now passes 11/11 (was 6 passed / 5 failed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Move the CoderAI broker warm-up handling into the provider so it applies on
every request path (rotation, autoselect and direct), and only for the
broker-session "cooling down" error:

- coderai.py: handle_request now wraps the broker decision in a warm-up retry
  loop — on "No active CoderAI broker session" it waits CODERAI_WARMUP_WAIT_
  SECONDS (10s) and retries the same provider up to CODERAI_MAX_WARMUP_WAITS
  (3) times, then surfaces the error WITHOUT recording a failure. Streaming
  and native-proxy paths get the same treatment via _broker_request_with_warmup
  (lazy generators retry on first broker contact). Other errors still record a
  failure as before.
- handlers.py: every place that could disable a provider on a caught error now
  skips record_failure() for the CoderAI warm-up condition (direct chat, direct
  streaming, rotation streaming, audio/TTS/image/embeddings, and the rotation
  retry loop), via the shared _is_coderai_warmup_error() helper. The rotation
  loop no longer sleeps itself (the provider already waited) — it just fails
  over without recording a failure.
- Default CoderAI request timeout raised from 5 min to 3 hours
  (CODERAI_DEFAULT_REQUEST_TIMEOUT, overridable via coderai_config.request_
  timeout); the timeout is also applied to the direct OpenAI client.

Bump version to 0.99.77.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CoderAI workers often run on small/edge hardware that drops its broker
session while cooling down to avoid overheating. A missing broker session
("No active CoderAI broker session...") was treated as a hard error:
record_failure() was called in both the provider handler and the rotation
loop, so three of them in one request disabled the provider for a 300s
cooldown — exactly when it just needed a moment to warm back up.

Now this transient condition is handled gracefully:
- coderai.py no longer records a failure when the error is a missing broker
  session (in any request path), so it never counts toward the disable
  threshold.
- The rotation handler detects the CoderAI warm-up condition and waits
  CODERAI_WARMUP_WAIT_SECONDS (10s) before retrying the same provider, up to
  CODERAI_MAX_WARMUP_WAITS (3) times, fully transparent to the client. Only
  after that does it fail over to the next provider — still without recording
  a failure.

Bump version to 0.99.76.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The provider, rotation and autoselect detail forms were single huge forms.
Split them into collapsible sub-panels so only basic settings stay visible:

- Shared subPanel()/toggleSubPanel() helper + CSS in base.html, defined
  before the content block so page scripts can call it during first render.
- Providers: basic fields visible; Authentication, Pricing & Tiers, Rate
  limits & defaults, Feature Overrides, Native Caching, Models as panels;
  each model has basic config visible + an Advanced sub-panel.
- Rotations: basic visible; Advanced configuration panel; one panel per
  provider (model-count + usage badge); one panel per model with an
  Advanced sub-panel.
- Autoselect: basic visible; Classification, Feature Overrides, Default
  settings panels; one panel per available model.
- Panel open/close state and top-level item expansion persist via
  sessionStorage, so they survive navigation within a session.

Bump version to 0.99.75.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A 402 (e.g. Kilo "usage_limit_exceeded"/out of credits) is deterministic:
retrying the same provider only burns the failure budget and needlessly
disables it within a single request. Add 402 to the non-retryable set so
the rotation fails over to the next provider immediately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- /api/v1/models/{model_id} and /api/u/{username}/models/{model_id} used a
  single-segment param, so model IDs containing '/' (e.g. rotations/lisa)
  returned 404. Use {model_id:path}.
- Add capability-probe endpoints so OpenAI/Ollama/llama.cpp-style clients stop
  getting 404/405 on connect: GET /api/version, /v1/version, /version;
  GET /api/tags (Ollama model list); GET /props, /api/props, /v1/props.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- anthropic: previously ignored the stream flag entirely (always non-stream).
  Add _handle_streaming_request converting Anthropic SSE events into
  OpenAI-compatible chunks (text deltas, tool_use/input_json deltas, usage).
- ollama: previously hard-coded stream=False. Add incremental streaming over
  /api/generate yielding OpenAI chunks.
- google: was buffering the entire response before yielding. Stream chunks
  incrementally as they arrive when no tools are requested (the common case);
  keep the buffer-and-parse path only when tools are present, since the
  text-encoded tool-call detection needs the complete response.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

For streaming requests providers returned a lazy generator and immediately
called record_success(), resetting the failure counter to 0 BEFORE the
rotation handler primed the stream and the upstream returned 400/429. Every
attempt did success(reset->0) then failure(->1), so the disable threshold (3)
was never reached and a consistently-failing provider was never disabled.

Guard the premature record_success() so it does not fire for streaming
(codex, openai, runpod, coderai). The caller already records success after
the stream is primed/consumed, so the failure counter now accumulates and a
failing provider is correctly disabled/cooled-down. Bump to 0.99.74.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Adds a per-provider Disable/Enable button to the rotations page for both admin
and users, exposing a manual disable that — unlike the failure cooldown — never
auto-expires and persists across reboots.

- providers/base.py: manual_disable()/manual_enable()/is_manually_disabled();
  manual disable takes precedence in is_rate_limited(); manual_enable() also
  clears any cooldown and resets the failure budget. Source of truth is the
  database (mirrored to cache as a fast path) so state survives restarts and
  cache flushes. Also: when a failure cooldown elapses, the provider is
  reactivated with a fresh failure budget instead of a hair-trigger.
- database.py: dedicated provider_manual_disabled table (+migration) and
  get/set/clear helpers, user-scoped so a user's toggle never affects others.
- dashboard: manual-disable / manual-enable / bulk manual-status endpoints; the
  rotations pages render and toggle the button state.

Bump to 0.99.73.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Each internal-model functionality can now point at a rotation or autoselect
instead of the local model, falling back to the local model when the override is
empty or fails. Per-functionality fields in internal_model (default empty):
condensation_override, autoselect_override, nsfw_classifier_override,
privacy_classifier_override.

- handlers.py: run_meta_target() runs a global rotation/autoselect and returns
  the assistant text (or None -> fallback); classification overrides prompt the
  chat model for a strict YES/NO and parse it, falling back to the local
  classifier; autoselect selection can run through an override.
- context.py: condensation prefers the override, falling back to the local model.
- settings page: per-field override inputs backed by a datalist of available
  rotations/autoselects; saved to internal_model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A rotation provider entry whose provider_id is "rotations" or "autoselect" is a
meta-provider: each model "name" is the id of a target rotation/autoselect, and
the request is forwarded there (weighted like any other entry, with failover to
the next entry on total failure).

- handlers.py: detect meta-providers in model-building, delegate in the retry
  loop, and resolve rotation vs autoselect targets.
- Loop protection: a delegation chain rides on request_data and is registered by
  both RotationHandler and AutoselectHandler; re-entering an id already in the
  chain (A->B->A) or exceeding depth 8 raises HTTP 508 (caught upstream -> fail
  over). Loops are also pre-filtered at model-build time.
- Dashboard: the provider select in rotations.html / user_rotations.html now
  offers "rotations"/"autoselect"; the model row becomes a target picker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

When a rotation fails over to a different provider/model than the preferred
(highest-weight) pick, optionally notify the client so the provider/model
change isn't silent. Opt-in via a new notify_on_failover flag, settable
globally (rotations.json top-level) or per-rotation, mirroring notifyerrors.

On a real switch (served provider/model differs from preferred), three
mechanisms carry the notice:
- Visible banner: prepended to message content (non-streaming) or a leading
  delta chunk (streaming).
- JSON metadata: an aisbf_failover object on the body / leading stream chunk.
- HTTP headers: X-AISBF-Failover/Provider/Model/Preferred-* on the
  StreamingResponse and (for non-streaming) at the rotation route.

Applied to both the normal and chunked rotation paths. The banner is injected
after caching so cache hits don't replay a stale notice; no switch means zero
overhead and nothing emitted; default off preserves existing behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Streaming rotation requests returned the provider's lazy async generator to
the client before the upstream HTTP call was made, so an immediate provider
error (e.g. HTTP 400 "model not supported") surfaced mid-stream after the
200 OK headers were already sent — leaking the error to the client instead
of failing over.

- Prime streaming responses (_prime_stream) inside the rotation retry loop so
  immediate provider errors are raised within the try/except and trigger
  failover to the next provider by weight, transparently to the client.
- Add _is_non_retryable_error to skip pointless same-provider retries for
  client errors (400/401/403/404/405/422) and move straight to the next
  provider; 5xx/timeouts/network remain retryable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add _record_dashboard_proxy_event, _settle_market_result,
_get_market_source_details, _market_request_id and
_extract_usage_from_sse_chunk to RotationHandler, and the market
settlement cluster to AutoselectHandler. These were defined only on
RequestHandler, so rotation/autoselect requests raised AttributeError
(crash on the dashboard event path, silently swallowed elsewhere),
dropping market settlement and streaming usage capture.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The collapsed provider-list status chip only rendered gpus[0], so a
coderai client connecting with multiple GPUs displayed just the first.
Render one chip per reported GPU (name + per-GPU VRAM), falling back to
an aggregate VRAM chip when no per-GPU detail is sent. GPU names are
HTML-escaped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>