Allow usage of stale-but-valid ingestion auth tokens#334
Open
mkeesey wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves ingestion resiliency by allowing the ingestion client to continue using a previously minted (still valid) Kusto ingestion authorization context when token refresh via Kusto management commands fails, and by throttling repeated refresh attempts to avoid hammering an unhealthy control plane.
Changes:
- Add stale-but-valid auth token reuse and refresh attempt throttling to
resources.Manager.AuthContext. - Track auth token refresh/validity/throttle timestamps in
Manager, with injectable time source for deterministic testing. - Add hermetic tests covering stale-token reuse, throttling behavior, expiry behavior, and concurrent callers.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
azkustoingest/internal/resources/resources.go |
Implements cached-token reuse + throttled refresh attempts, adds time injection for testing. |
azkustoingest/internal/resources/resources_test.go |
Adds a deterministic Mgmt stub and new tests for refresh failure, throttling, expiry, and concurrency. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Author
|
I have this in draft status right now to exercise these changes with our own service code first. |
The Kusto ingestion client periodically (every 1h) attempts to call a management command on the destination Kusto cluster to refresh a token used to authenticate to Kusto-cluster owned storage. On occasion, we experience issues where our Kusto cluster's control plane has brief unavailability for several minutes at a time - this can completely block ingestion from clients for long periods of time, despite the tokens themselves having much longer lifetimes and the destination storage remaining available. Instead of failing fast when new tokens cannot be minted, this PR adds a mechanism to reuse old-but-valid tokens when new tokens can't be created. This should allow these clients to continue uploading to storage while the Kusto cluster is temporarily unavailable. This also adds a throttling mechanism where we do not continually hammer the destination Kusto cluster during token expirations, creating more of a thundering herd when the Kusto control plane is already having issues.
Author
|
I think this one is ready for review now. Apologies it sat in draft form for so long - took me some time to get back to this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Kusto ingestion client periodically (every 1h) attempts to call a management command on the destination Kusto cluster to refresh a token used to authenticate to Kusto-cluster owned storage. On occasion, we experience issues where our Kusto cluster's control plane has brief unavailability for several minutes at a time - this can completely block ingestion from clients for long periods of time, despite the tokens themselves having much longer lifetimes and the destination storage remaining available.
Instead of failing fast when new tokens cannot be minted, this PR adds a mechanism to reuse old-but-valid tokens when new tokens can't be created. This should allow these clients to continue uploading to storage while the Kusto cluster is temporarily unavailable.
This also adds a throttling mechanism where we do not continually hammer the destination Kusto cluster during token expirations, creating more of a thundering herd when the Kusto control plane is already having issues.