Feat: Unify remote filesystem dispatch for exporters (groundwork for lakehouse support) - #1021
Merged
Conversation
- add create_filesystem_for_path in data_juicer/utils/fs_utils.py as the single dispatch point for building PyArrow filesystems by path scheme (side-effect-free: never mutates the input extra-args dict and returns the remaining args; pyarrow imported lazily) - refactor S3/HDFS filesystem initialization in Exporter/RayExporter to delegate to the unified helper - make validate_s3_path/validate_hdfs_path and exporter branch checks case-insensitive on the scheme; single-slash typos like s3:/x and hdfs:/x now raise ValueError instead of being treated as local paths - add tests/utils/test_fs_utils.py and extend s3/hdfs utils tests
cmgzn
requested review from
Dludora,
Qirui-jiao,
cyruszhang,
fengrui-z and
yxdyc
July 29, 2026 09:21
fengrui-z
reviewed
Aug 3, 2026
Align the remaining scheme checks in the export chain with the case-insensitive dispatch introduced for exporters, so uppercase schemes (S3://, HDFS://) are handled consistently end to end: - exporter/ray_exporter: encryption-skip check and shard-path construction now use urlparse(...).scheme.lower(); scheme stripping is case-insensitive (split on '://') - default/ray/partitioned executors: S3 credential injection guard uses urlparse(...).scheme.lower() - config: remote-path detection and log-filename path extraction use urlparse (netloc/path) instead of case-sensitive startswith/replace - file_utils.is_remote_path: case-insensitive scheme check
Dludora
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This is the first groundwork PR in the incremental split of #911 (lakehouse data source support). It converges the duplicated "detect path prefix → extract credentials/config → build a PyArrow FileSystem" logic scattered across exporters into a single dispatch extension point. Follow-up Iceberg/Delta/Hudi PRs only need to extend the scheme branches in one place instead of re-implementing per-callsite logic.
Builds on the HDFS support introduced in #1014.
Changes
data_juicer/utils/fs_utils.pywithcreate_filesystem_for_path(path, extra_args=None) -> (fs, remaining_args):s3://→create_pyarrow_s3_filesystem,hdfs://→create_pyarrow_hdfs_filesystem(config includespathso host/port can be inferred), anything else →(None, copy);extra_argsdict; the backend-specific keys (S3_FS_KEYS/HDFS_FS_KEYS) are consumed and removed from the returnedremaining_argscopy;RayExporter: the two parallel S3/HDFSifblocks that poppedaws_*/hdfs_*keys fromexport_extra_argsare replaced by a single call to the helper;self.s3_filesystem/self.hdfs_filesystemattributes and their usage in write methods are unchanged.Exporter: the HDFS branch (PyArrow fs +ArrowFSWrapper) now delegates to the helper; the S3 branch keeps building fsspecstorage_options(different backend contract) but its entry check is aligned to the same scheme semantics.validate_s3_path/validate_hdfs_pathand the exporter branch checks now treat the scheme case-insensitively, consistent withurlparsesemantics.Behavior notes
s3:/bucket/xorhdfs:/user/xnow raise a clearValueErrorat exporter construction time (previously they were silently treated as local paths).S3://...,HDFS://...) are now correctly accepted and routed to the corresponding filesystem.Related
Tests
tests/utils/test_fs_utils.py(15 cases): local/relative/file://paths, S3/HDFS dispatch with mocked factories (exact conf/remaining-key assertions), input-dict immutability (deep-compare), uppercase schemes, query-parameter paths, malformed paths (ValueError),extra_args=None.tests/utils/test_s3_utils.py/test_hdfs_utils.pywith uppercase-accept and single-slash-reject cases.tests/utils/test_fs_utils.py+test_s3_utils.py+test_hdfs_utils.py(49 passed),tests/core/test_exporter.py+tests/core/test_ray_exporter.py(32 passed), load-strategy HDFS/S3 tests unaffected.