[#12051] fix(flink-connector): Move Paimon Hadoop options to Hadoop conf#12121
[#12051] fix(flink-connector): Move Paimon Hadoop options to Hadoop conf#12121jarredhj0214 wants to merge 4 commits into
Conversation
| moveHadoopOptionsToConf(paimonOptions, hadoopConf); | ||
| try { | ||
| CredentialPropertyUtils.applyPaimonCredentials( | ||
| CredentialPropertyUtils.getCredentials(catalog()), mutableOptions); |
There was a problem hiding this comment.
CredentialPropertyUtils.applyPaimonCredentials may put credentials to mutableOptions.
We need to confirm one thing: should these properties be removed from paimonOptions?
There was a problem hiding this comment.
CredentialPropertyUtils.applyPaimonCredentialsmay put credentials to mutableOptions.We need to confirm one thing: should these properties be removed from
paimonOptions?
Good point. These two sets of properties are intentionally handled differently.
The Hadoop filesystem options from catalog properties (hadoop.*, fs.*, and dfs.*) are removed from paimonOptions and moved into Hadoop Configuration.
The credentials injected by CredentialPropertyUtils.applyPaimonCredentials are vended credentials, and they need to stay in paimonOptions because Paimon native S3/OSS FileIO loaders declare these keys as required options. If we remove them from paimonOptions, native Paimon S3/OSS FileIO may fail to initialize.
I added a comment to make this distinction explicit.
What changes were proposed in this pull request?
Move Paimon Flink connector Hadoop filesystem options (
hadoop.*,fs.*, anddfs.*) from Paimon catalog options into HadoopConfigurationbefore creating the inner Paimon catalog.This creates the inner Paimon catalog with a
CatalogContextthat carries the sanitized Paimon options plus Hadoop configuration, and keeps vended Paimon S3/OSS/JDBC credentials available as Paimon options.Why are the changes needed?
Paimon can log dynamic/catalog options and expose sensitive filesystem credentials such as OSS/BOS access keys when they remain in Paimon options. Moving Hadoop filesystem credentials to Hadoop configuration follows the Flink Hive connector pattern and avoids passing these sensitive Hadoop properties through Paimon options.
Fix: #12051
Does this PR introduce any user-facing change?
No API changes. Existing Hadoop filesystem catalog properties continue to be accepted, but the Flink Paimon connector forwards them via Hadoop
Configurationinstead of Paimon catalog options.How was this patch tested?
Added unit coverage for:
hadoop.*/fs.*filesystem options into Hadoop configurationLocal checks:
git diff --checkCould not run Gradle tests locally because only JDK 8 is installed and the build requires JDK 17.