Describe the bug
file_size() needs only object metadata, but its Rust implementation currently opens each FileReference through DaftFile::load_blocking() before reading the size.
For a non-byte-range FileReference, DaftFile::load() first calls ObjectSource::get_size(), then also calls ObjectSource::supports_range(), and may construct a reader. file_size() subsequently returns only the size already obtained during the load path.
This adds unnecessary I/O and blocking work for every input file. The overhead is especially noticeable when evaluating file_size() over many remote objects, where supports_range() may require an additional request.
To Reproduce
import daft
from daft.functions import file, file_size
df = daft.from_pydict(
{
"path": [
"s3://my-bucket/path/file-1.parquet",
"s3://my-bucket/path/file-2.parquet",
# many more remote files
]
}
)
df.select(file_size(file(df["path"]))).collect()
Inspecting the current Rust execution path:
1. Size::call invokes DaftFile::load_blocking() for each file.
2. DaftFile::load() invokes get_size().
3. The same method then invokes supports_range(), although file_size() does not read file contents or need range support.
4. Size::call obtains the already-known size from the loaded file object.
### Expected behavior
file_size() should fetch only the size metadata required for its result.
For batches of file references, it should use an asynchronous/vectorized metadata path so independent remote size requests can run concurrently, without opening files or checking range-read support.
### Component(s)
Built-in Functions, Multimodal Operations (files/images/etc.)
### Additional context
uggested implementation:
- Add an async helper that resolves a FileReference and directly calls ObjectSource::get_size().
- Convert the file_size UDF to AsyncScalarUDF, following the existing file_exists() pattern.
- Preserve null propagation and existing error behavior.
- Add tests covering local files and a mocked object source, including a regression check that the size-only path does not call supports_range() or read file contents.
Relevant code:
- src/daft-file/src/functions.rs (Size::call)
- src/daft-file/src/file.rs (DaftFile::load)
Describe the bug
file_size()needs only object metadata, but its Rust implementation currently opens eachFileReferencethroughDaftFile::load_blocking()before reading the size.For a non-byte-range
FileReference,DaftFile::load()first callsObjectSource::get_size(), then also callsObjectSource::supports_range(), and may construct a reader.file_size()subsequently returns only the size already obtained during the load path.This adds unnecessary I/O and blocking work for every input file. The overhead is especially noticeable when evaluating
file_size()over many remote objects, wheresupports_range()may require an additional request.To Reproduce