• Stefy Lanza (nextime / spora )'s avatar
    Don't let request-scoped errors trip the provider failure cooldown · fae38ea0
    Stefy Lanza (nextime / spora ) authored
    A model that intermittently lacks an image endpoint on kilo returns 404
    "No endpoints found that support image input". The kilo handler counted every
    exception (that 404, a 400 for an undecodable image, a 429) toward the
    three-strike cooldown, so a few request-scoped errors disabled kilo-spora for
    ALL traffic for 5 minutes — during which a high-rate client piled up thousands
    of 503 "in failure cooldown" rejections.
    
    Add a shared classifier in providers/base.py: is_request_scoped_error /
    is_request_scoped_status / extract_http_status treat 400/404/413/415/422/429
    (and the wrapped streaming-error message form) as request-scoped — about the
    specific request, not provider health. Genuine faults (5xx, timeouts, network)
    still count.
    
    Wire it into both places that record failures:
    - KiloProviderHandler.handle_request and _handle_streaming_request skip
      record_failure() for request-scoped errors (429 still routes through
      handle_429_error for adaptive backoff).
    - handlers._should_record_failure excludes them for every provider.
    
    The error still propagates to the caller; it just no longer disables the whole
    provider. Only a genuinely unhealthy provider now enters the cooldown.
    Co-Authored-By: 's avatarClaude Opus 4.8 (1M context) <noreply@anthropic.com>
    fae38ea0
base.py 85.1 KB