Skip to content
Commit 205f6d7f authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

Stop the manual-disable flag from evicting providers from rotations



A provider carrying the legacy global manual-disable flag was permanently
evicted from every rotation it belonged to, and clients saw an upstream 429.
Observed in production on rotation/lisa: codex had valid credentials and was
never once called upstream all day.

Two independent defects combined:

1. is_rate_limited() honoured the manual flag, so every provider's
   handle_request() raised Exception("Provider rate limited") before any
   upstream call. The rotation scan deliberately ignores that flag, so the
   provider was still selected, and the synthetic error was then counted
   against its health — three of them armed a real failure cooldown. This
   contradicted the behaviour already documented in handlers.py: the flag
   gates nothing, neither direct calls nor rotation. Rotation membership is
   expressed per entry via `enabled: false`.

2. The rotation retry loop fed a provider's own availability-gate error into
   record_failure(), pushing disabled_until another full cooldown into the
   future on every attempt. Under steady traffic the cooldown could never
   elapse: the failure counter reached 447 against a threshold of 3, with no
   reactivation in any retained log.

is_rate_limited() is now cooldown-only; use is_manually_disabled() when the
manual flag is what you actually want. The dashboard already reports the two
states separately, so the toggle is unaffected.

Co-Authored-By: default avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
parent 029c6986
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment