Skip to content

Dataset relative files fail on Windows and filesystem URIs #4502

Description

@ShiroKSH

Summary

json_dataset() and csv_dataset() do not reliably resolve relative sample files on Windows or when the dataset path is a filesystem URI.

On Windows, the existing tests/dataset/test_dataset.py::test_dataset_image_paths test fails on current main because images/ballons.png remains relative instead of resolving beside images.jsonl.

Root cause

The readers store Dataset.location with os.path.abspath(), while resolve_sample_files() derives the parent by splitting that value on the fsspec filesystem separator.

  • Windows abspath() produces backslashes, while LocalFileSystem.sep is /, so the split does not find the parent directory.
  • os.path.abspath() treats file://, s3://, and other filesystem URIs as local path strings, corrupting the stored location before relative files are resolved.

Reproduction

uv run pytest tests/dataset/test_dataset.py::test_dataset_image_paths -vv

Current result on Windows:

AssertionError: assert WindowsPath('images/ballons.png').exists()

Expected behavior

Dataset locations should retain their filesystem scheme, and relative sample files should resolve against the dataset's parent directory on local and remote filesystems.

I can open a focused PR using the repository's filesystem-aware path helpers, with regression coverage for local paths and filesystem URIs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions