fix(rest): resolve a FileIO without io-impl or warehouse configured#13
Open
smaheshwar-pltr wants to merge 3 commits into
Open
fix(rest): resolve a FileIO without io-impl or warehouse configured#13smaheshwar-pltr wants to merge 3 commits into
smaheshwar-pltr wants to merge 3 commits into
Conversation
smaheshwar-pltr
force-pushed
the
rest-fileio-fallback
branch
from
June 27, 2026 23:21
005c648 to
3849099
Compare
MakeCatalogFileIO and MakeTableFileIO previously returned InvalidArgument when neither "io-impl" nor "warehouse" was configured, preventing a catalog that relies solely on vended storage credentials (no client-side IO config) from being created or loading tables. Now: - MakeCatalogFileIO falls back to a local FileIO so the catalog can be created; per-table resolution supplies remote access. - MakeTableFileIO infers the implementation from "io-impl", then the table's metadata-location scheme, then the "warehouse" scheme, else local. The metadata location is threaded through TableFileIO. This mirrors PyIceberg's load_file_io, which selects the FileIO from the metadata-location scheme. Adds tests for metadata-location detection, precedence over a (logical) warehouse, and the local fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
smaheshwar-pltr
force-pushed
the
rest-fileio-fallback
branch
from
July 12, 2026 01:41
3849099 to
0950161
Compare
smaheshwar-pltr
commented
Jul 12, 2026
| RestCatalogProperties::kWarehouse.key()); | ||
| if (!metadata_location.empty()) { | ||
| ICEBERG_ASSIGN_OR_RAISE(const auto detected_kind, | ||
| DetectBuiltinFileIO(metadata_location)); |
Owner
Author
There was a problem hiding this comment.
Selecting the table FileIO from the metadata-location scheme (then the warehouse) mirrors PyIceberg's load_file_io, so a catalog that vends credentials works without a client-side io-impl/warehouse:
load_file_io: https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/io/__init__.py#L349_infer_file_io_from_scheme: https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/io/__init__.py#L337- call site (passes
metadata_location): https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/catalog/rest/__init__.py#L837
(Java reaches the same outcome differently — a single ResolvingFileIO that resolves per file operation — so it isn't the precedent for this load-time selection.)
smaheshwar-pltr
force-pushed
the
rest-fileio-fallback
branch
from
July 14, 2026 18:18
750accb to
be02260
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resolve a REST catalog
FileIOwhen neitherio-implnorwarehouseis configured:MakeCatalogFileIOfalls back to a localFileIO(instead of returningInvalidArgument).MakeTableFileIOinfers the implementation in orderio-impl> the table'smetadata-locationscheme >warehousescheme > local. The metadata location is threaded throughTableFileIO.Why
A catalog that relies solely on vended storage credentials carries no client-side
io-impl/warehouse, so both factories previously failed and such a catalog could be neither created nor used.Selecting the table FileIO from the
metadata-locationscheme mirrors PyIceberg'sload_file_io, which picks the FileIO from the metadata-location scheme (then the warehouse):load_file_io/_infer_file_io_from_scheme: https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/io/__init__.py#L349 , https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/io/__init__.py#L337metadata_location): https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/catalog/rest/__init__.py#L837(Java reaches the same outcome differently — a single
ResolvingFileIOthat resolves per file operation — so it is not the precedent for this load-time selection.)Why one PR (not two)
The two changes are one capability and are non-functional apart: with only the
MakeCatalogFileIOfallback a vended-table load still throws inMakeTableFileIO; and themetadata-locationbranch is unreachable without the catalog fallback (a configured catalog's merged config always carriesio-impl/warehouse). Splitting would ship dead code.Notes / follow-ups
metadata-locationscheme fails fast, rather than silently building a warehouse-derived FileIO that could not read the table anyway.metadata-locationstill reuses the catalog FileIO; resolving that is a follow-up.