Add object store multi cache sharding#23177
Draft
davelopez wants to merge 9 commits into
Draft
Conversation
Introduces support for distributing cached datasets across multiple directories with weighted distribution. Adds core classes to handle shard selection based on dataset identifiers. Updates cache monitor to track multiple targets and modifies configuration parsing to support the new directory structure. Provides example configuration to guide setup.
Updates the base caching class to utilize the new shard manager. Replaces direct path and size attributes with properties backed by the shard configuration. Modifies internal methods to accept object identifiers, enabling deterministic cache path resolution per dataset. Ensures cache operations route files correctly while maintaining backward compatibility.
Updates all caching-enabled backends to integrate the sharding infrastructure. Initializes the shard manager during setup and passes object identifiers to download, upload, and validation methods. Ensures each backend correctly resolves cache paths, respects shard-specific limits, and serializes the updated configuration format.
Introduces comprehensive unit tests for the shard manager and distributed caching behavior. Validates deterministic selection, weight distribution, configuration fallbacks, and target sizing. Updates existing monitor tests to accommodate the new initialization signature. Verifies end-to-end operations across multiple directories using a stub backend.
Updates the core caching validation logic to require an object ID, removing the fallback to a default cache target. This simplifies target resolution and ensures consistent sharding behavior. Callers across cloud, onedata, and S3 backends are updated to pass the ID explicitly. The unused staging path setter and related import are removed. Test stubs are adjusted to match the new signature.
Deletes the legacy cache size configuration utility function as it is no longer referenced. Updates the shard manager documentation to explicitly note that the weight parameter is intentionally omitted for single-shard configurations, improving clarity for future maintenance.
Refactors the shard consistency test by moving the root path resolution into a dedicated helper function. This reduces code duplication and improves readability within the test suite.
Adds helper functions to verify shard paths and locate shard roots. Updates tests to use these helpers instead of inline logic. Improves code consistency and readability across the test suite.
Eliminates extraction of unused variables from kwargs in the caching object store. Reduces code clutter and simplifies the method signature handling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP TBD
How to test the changes?
License