Skip to content

[#12051] fix(flink-connector): Move Paimon Hadoop options to Hadoop conf#12121

Open
jarredhj0214 wants to merge 4 commits into
apache:mainfrom
jarredhj0214:codex/flink-paimon-hadoop-conf-credentials
Open

[#12051] fix(flink-connector): Move Paimon Hadoop options to Hadoop conf#12121
jarredhj0214 wants to merge 4 commits into
apache:mainfrom
jarredhj0214:codex/flink-paimon-hadoop-conf-credentials

Conversation

@jarredhj0214

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Move Paimon Flink connector Hadoop filesystem options (hadoop.*, fs.*, and dfs.*) from Paimon catalog options into Hadoop Configuration before creating the inner Paimon catalog.

This creates the inner Paimon catalog with a CatalogContext that carries the sanitized Paimon options plus Hadoop configuration, and keeps vended Paimon S3/OSS/JDBC credentials available as Paimon options.

Why are the changes needed?

Paimon can log dynamic/catalog options and expose sensitive filesystem credentials such as OSS/BOS access keys when they remain in Paimon options. Moving Hadoop filesystem credentials to Hadoop configuration follows the Flink Hive connector pattern and avoids passing these sensitive Hadoop properties through Paimon options.

Fix: #12051

Does this PR introduce any user-facing change?

No API changes. Existing Hadoop filesystem catalog properties continue to be accepted, but the Flink Paimon connector forwards them via Hadoop Configuration instead of Paimon catalog options.

How was this patch tested?

Added unit coverage for:

  • moving hadoop.*/fs.* filesystem options into Hadoop configuration
  • keeping vended S3/OSS credentials in Paimon options for native Paimon FileIOs
  • keeping JDBC credentials in Paimon options

Local checks:

  • git diff --check

Could not run Gradle tests locally because only JDK 8 is installed and the build requires JDK 17.

@geyanggang
geyanggang requested a review from diqiu50 July 21, 2026 08:51
@diqiu50 diqiu50 changed the title [#12051] fix(flink): move Paimon Hadoop options to Hadoop conf [#12051] fix(flink-connector): move Paimon Hadoop options to Hadoop conf Jul 22, 2026
@diqiu50 diqiu50 changed the title [#12051] fix(flink-connector): move Paimon Hadoop options to Hadoop conf [#12051] fix(flink-connector): Move Paimon Hadoop options to Hadoop conf Jul 22, 2026
moveHadoopOptionsToConf(paimonOptions, hadoopConf);
try {
CredentialPropertyUtils.applyPaimonCredentials(
CredentialPropertyUtils.getCredentials(catalog()), mutableOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CredentialPropertyUtils.applyPaimonCredentials may put credentials to mutableOptions.

We need to confirm one thing: should these properties be removed from paimonOptions?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CredentialPropertyUtils.applyPaimonCredentials may put credentials to mutableOptions.

We need to confirm one thing: should these properties be removed from paimonOptions?

Good point. These two sets of properties are intentionally handled differently.

The Hadoop filesystem options from catalog properties (hadoop.*, fs.*, and dfs.*) are removed from paimonOptions and moved into Hadoop Configuration.

The credentials injected by CredentialPropertyUtils.applyPaimonCredentials are vended credentials, and they need to stay in paimonOptions because Paimon native S3/OSS FileIO loaders declare these keys as required options. If we remove them from paimonOptions, native Paimon S3/OSS FileIO may fail to initialize.

I added a comment to make this distinction explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug report] Flink Paimon connector may expose sensitive catalog properties in logs

2 participants