Skip to content

file_size() performs unnecessary file-open work for every input file #7480

Description

@yongster

Describe the bug

file_size() needs only object metadata, but its Rust implementation currently opens each FileReference through DaftFile::load_blocking() before reading the size.

For a non-byte-range FileReference, DaftFile::load() first calls ObjectSource::get_size(), then also calls ObjectSource::supports_range(), and may construct a reader. file_size() subsequently returns only the size already obtained during the load path.

This adds unnecessary I/O and blocking work for every input file. The overhead is especially noticeable when evaluating file_size() over many remote objects, where supports_range() may require an additional request.

To Reproduce

import daft
from daft.functions import file, file_size

df = daft.from_pydict(
    {
        "path": [
            "s3://my-bucket/path/file-1.parquet",
            "s3://my-bucket/path/file-2.parquet",
            # many more remote files
        ]
    }
)

df.select(file_size(file(df["path"]))).collect()

Inspecting the current Rust execution path:
1. Size::call invokes DaftFile::load_blocking() for each file.
2. DaftFile::load() invokes get_size().
3. The same method then invokes supports_range(), although file_size() does not read file contents or need range support.
4. Size::call obtains the already-known size from the loaded file object.

### Expected behavior

file_size() should fetch only the size metadata required for its result.
For batches of file references, it should use an asynchronous/vectorized metadata path so independent remote size requests can run concurrently, without opening files or checking range-read support.

### Component(s)

Built-in Functions, Multimodal Operations (files/images/etc.)

### Additional context

uggested implementation:
- Add an async helper that resolves a FileReference and directly calls ObjectSource::get_size().
- Convert the file_size UDF to AsyncScalarUDF, following the existing file_exists() pattern.
- Preserve null propagation and existing error behavior.
- Add tests covering local files and a mocked object source, including a regression check that the size-only path does not call supports_range() or read file contents.
Relevant code:
- src/daft-file/src/functions.rs (Size::call)
- src/daft-file/src/file.rs (DaftFile::load)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions