Operational: No circuit breaker pattern for failing providers #92

New issue

Open

opened 2026-05-11 13:48:53 +00:00 by thabeta · 0 comments

thabeta commented

2026-05-11 13:48:53 +00:00

Owner

Severity: Medium

Location

crates/hero_aibroker_lib/src/providers/keypool.rs

Finding

The KeyPool has health tracking per key but no circuit breaker pattern:

Keys are marked with FailureKind and backoff
No tripping threshold that removes a provider entirely
No half-open state for recovery testing
Failed providers continue receiving traffic (on less-preferred keys)

Impact

A provider experiencing a partial outage continues receiving requests
Requests fail slowly (waiting for timeout) instead of failing fast
No automatic isolation of unhealthy providers

Recommendation

Implement circuit breaker states: Closed → Open → Half-Open
Trip the circuit after N consecutive failures
Periodically test half-open circuits with probe requests

## Severity: Medium ## Location `crates/hero_aibroker_lib/src/providers/keypool.rs` ## Finding The KeyPool has health tracking per key but no circuit breaker pattern: - Keys are marked with `FailureKind` and backoff - No tripping threshold that removes a provider entirely - No half-open state for recovery testing - Failed providers continue receiving traffic (on less-preferred keys) ## Impact - A provider experiencing a partial outage continues receiving requests - Requests fail slowly (waiting for timeout) instead of failing fast - No automatic isolation of unhealthy providers ## Recommendation - Implement circuit breaker states: Closed → Open → Half-Open - Trip the circuit after N consecutive failures - Periodically test half-open circuits with probe requests