refactor(remote-config): add API sources failover with health check endpoint support#3812
Open
tonidero wants to merge 2 commits into
Open
refactor(remote-config): add API sources failover with health check endpoint support#3812tonidero wants to merge 2 commits into
tonidero wants to merge 2 commits into
Conversation
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
tonidero
changed the title
tonidero
force-pushed
the
api-source-health-checker
branch
from
July 21, 2026 14:42
442910a to
630c17f
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3812 +/- ##
=======================================
Coverage ? 80.59%
=======================================
Files ? 410
Lines ? 17014
Branches ? 2493
=======================================
Hits ? 13712
Misses ? 2337
Partials ? 965 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tonidero
force-pushed
the
api-source-health-checker
branch
2 times, most recently
from
July 21, 2026 16:01
d18c71c to
0f59e70
Compare
Generated by 🚫 Danger |
tonidero
changed the title
tonidero
force-pushed
the
api-source-health-checker
branch
from
July 21, 2026 16:43
0f59e70 to
d35d1c3
Compare
Completes the API source failover feature behind the usesRemoteConfigAPISources dangerous setting (still internal and defaulting to false, so released behavior is unchanged): - SourceHealthChecker: probes <source>/v1/health/connectivity; only a 2xx is healthy. Blocking, so it must be called from a background thread. Concurrent checks per source share one request and results are cached for 10s. - DefaultRemoteConfigSourceProvider: the API failover list restarts from the top 10 minutes after it first advances, so the SDK periodically retries the primary source (and re-arms an exhausted list) instead of sticking on a backup forever. - APISourceFailover: the decision layer. Connection-level failures and 5xx responses can fail over, but only after the current source's health check fails; a 2xx health response means the source is healthy and the original error surfaces instead. 4xx responses never fail over. Malformed source URLs are reported unhealthy and skipped. - HTTPClient: performRequest resolves a source per attempt and walks the source list on failed attempts, transparently retrying the in-flight request on the next source. ETag refresh retries and the static per-endpoint fallback URLs keep their existing behavior, with static fallback acting as the last resort once sources are exhausted or decline to fail over. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tonidero
force-pushed
the
api-source-health-checker
branch
from
July 21, 2026 17:50
d35d1c3 to
51c160a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Completes the health-check-gated API source failover feature. Everything remains behind the
usesRemoteConfigAPISourcesdangerous setting, so released behavior is unchanged.Failure classification per request attempt on the current source:
GET <source>/v1/health/connectivity(lazy, never proactive). Health 2xx → source healthy, no failover, surface the original error/result. Health non-2xx or unreachable → mark source unhealthy, transparently retry the in-flight request on the next source.Note
Medium Risk
Changes core HTTP request routing and retry logic for API traffic behind a dangerous setting; incorrect health-check or failover decisions could misroute requests or mask errors, though default production behavior is unchanged when the flag is off.
Overview
Adds health-check-gated failover across remote-config API hosts when
usesRemoteConfigAPISourcesis on.HTTPClientno longer only picks a source URL up front; it loops through sources (capped at 5 attempts), and on 5xx or connection failure asksAPISourceFailoverwhether to retry on the next host.APISourceFailoverpicks the current source (same eligibility as before: default base URL, endpoint opts in, not a fallback-host attempt) and on failure probesGET <source>/v1/health/connectivityviaSourceHealthChecker(cached ~10s, deduped concurrent probes). Health 2xx → treat the source as up and surface the original error; unhealthy →reportUnhealthyand transparently retry the same in-flight request on the next source. 4xx never triggers failover or health checks.DefaultRemoteConfigSourceProvidernow re-arms the API list every 10 minutes after the first failover advance so the SDK periodically retries the primary; blob failover is unchanged.SourceFailover.reportUnhealthyreturns whether it actually advanced.PurchasesFactorywiresAPISourceFailover+SourceHealthCheckerintoHTTPClientinstead of passing the raw source provider. Broad unit and backend integration tests cover the new behavior.Reviewed by Cursor Bugbot for commit 64e7f06. Bugbot is set up for automated code reviews on this repo. Configure here.