Skip to content

[Data] Generate sortable, collision-resistant Dataset IDs - #65075

Open
yuhuan130 wants to merge 1 commit into
ray-project:masterfrom
yuhuan130:feat/data-dataset-ulids
Open

[Data] Generate sortable, collision-resistant Dataset IDs#65075
yuhuan130 wants to merge 1 commit into
ray-project:masterfrom
yuhuan130:feat/data-dataset-ulids

Conversation

@yuhuan130

Copy link
Copy Markdown
Contributor

Description

This PR replaces counter-based Dataset IDs with 22-character, Base62-encoded ULIDs.

Each ULID contains a timestamp and random data, making it sortable by creation time and practically collision-free.

The full ID format remains:

{dataset_name}_{dataset_ulid}_{run_index}

This PR also:

  • Generates a new Dataset ULID during deserialization.
  • Keeps the run index for repeated executions and training epochs.
  • Adds RAY_DATA_USE_LEGACY_DATASET_IDS as a temporary compatibility option.

Tests

Focused Dataset ID tests: 5 passed

Also updates test_dataset_id_train_ingest test in python/ray/data/tests/test_stats.py to verify that the same generated Dataset ID is used across epochs while the run index increases.


Closes #65073

Signed-off-by: Alex Chien <alexchien130@gmail.com>
@yuhuan130
yuhuan130 requested a review from a team as a code owner July 28, 2026 10:58

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces time-sortable, practically unique Dataset ULIDs encoded in Base62, replacing the legacy counter-based Dataset IDs by default, while providing a configuration option to fall back to the legacy behavior. The review feedback focuses on improving robustness and compatibility, specifically by replacing parenthesized context managers in tests to maintain Python 3.8 compatibility, optimizing the Base62 encoding function to avoid string concatenation in a loop, and using getattr when accessing use_legacy_dataset_ids to prevent potential AttributeErrors during initialization and deserialization.

Comment thread python/ray/data/tests/unit/test_dataset_id.py
Comment thread python/ray/data/_internal/dataset_id.py
Comment thread python/ray/data/dataset.py
Comment thread python/ray/data/dataset.py
@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Data] Prevent Dataset ID collisions

1 participant