feat(io): expose FileIO properties#14
Open
smaheshwar-pltr wants to merge 2 commits into
Open
Conversation
smaheshwar-pltr
commented
Jun 27, 2026
Comment on lines
+218
to
+219
| std::unordered_map<std::string, std::string> PropertiesForLocation( | ||
| std::string_view location) const override; |
Owner
Author
There was a problem hiding this comment.
I don't get this - what is the precedent for properties just per-location? Look at the prior PR - I wrote PR description + maybe comments citing sources of how different implementations provide properties to engines, did we get inspiration from one of those? Or if not why are we different from the other Iceberg implementations? Quite confused here
smaheshwar-pltr
force-pushed
the
rest-fileio-properties
branch
from
June 27, 2026 23:21
fc46f62 to
13a013b
Compare
smaheshwar-pltr
commented
Jun 27, 2026
| return *best; | ||
| } | ||
|
|
||
| std::unordered_map<std::string, std::string> ArrowS3FileIO::properties() const { |
Owner
Author
There was a problem hiding this comment.
Exposing the FileIO's configuration as a flat map is how every other implementation surfaces it to engines:
- Java
FileIO.properties(): https://github.com/apache/iceberg/blob/874e4096e5d16fddbbf43b91f20e248616232cc3/api/src/main/java/org/apache/iceberg/io/FileIO.java#L105 - Rust
config().props()(&HashMap): https://github.com/apache/iceberg-rust/blob/47eaef7185916eb86add59d913eb872a06810465/crates/iceberg/src/io/storage/config/mod.rs#L78 - PyIceberg
table.io.properties: https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/io/__init__.py#L260-L263 - Trino reads
baseTable.io().properties()directly: https://github.com/trinodb/trino/blob/c9a2b907234b94c230cc26e2993411d4afdd77c5/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/IcebergRestCatalogFileSystemFactory.java#L60
Real REST servers return vended credentials in a table's config, so they appear here; structured storage-credentials stay on the separate SupportsStorageCredentials channel, as in Java.
smaheshwar-pltr
force-pushed
the
rest-fileio-properties
branch
from
June 28, 2026 00:03
13a013b to
fe44eb3
Compare
Add FileIO::properties(), returning the FileIO's configuration by const reference. The default returns an empty map; ArrowS3FileIO returns the properties it was configured with. This lets an engine that performs its own I/O read the resolved configuration of a table's FileIO, matching the property surface other Iceberg implementations expose (Java FileIO.properties(), Rust config().props(), Python table.io.properties; Trino reads baseTable.io().properties()). Vended credentials returned in a table's config are part of that configuration and are surfaced here; structured storage-credentials remain on the separate SupportsStorageCredentials channel, as in Java. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
smaheshwar-pltr
force-pushed
the
rest-fileio-properties
branch
from
June 28, 2026 00:05
fe44eb3 to
20f9c65
Compare
smaheshwar-pltr
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add
FileIO::properties(), returning the FileIO's configuration byconstreference. The default returns an empty map;ArrowS3FileIOreturns the properties it was configured with.Why
An engine that performs its own I/O needs to read the resolved configuration off a loaded table's FileIO. Every other Iceberg implementation exposes this as a flat property map:
FileIO.properties(): https://github.com/apache/iceberg/blob/874e4096e5d16fddbbf43b91f20e248616232cc3/api/src/main/java/org/apache/iceberg/io/FileIO.java#L105Storage::config().props()(returns&HashMap): https://github.com/apache/iceberg-rust/blob/47eaef7185916eb86add59d913eb872a06810465/crates/iceberg/src/io/storage/config/mod.rs#L78table.io.properties: https://github.com/apache/iceberg-python/blob/01a86f987d1924ca3246a2b985b78890ec0c0fbd/pyiceberg/io/__init__.py#L260-L263baseTable.io().properties()directly: https://github.com/trinodb/trino/blob/c9a2b907234b94c230cc26e2993411d4afdd77c5/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/IcebergRestCatalogFileSystemFactory.java#L60Scope / notes
storage-credentialsinstalled viaSupportsStorageCredentialsare a separate channel used for per-prefix routing and are not included, matching Java (whoseFileIO.properties()likewise excludes them).{}). Note it adds a method to the exportedFileIOvtable.