Add YAML tests for writing to Iceberg CDC sink

[ahmedabu98]

Adds YAML tests for:

1. Delta to Iceberg CDC (batch)
2. Iceberg to Iceberg CDC (streaming and batch)

Adds a small fix to let Python recognize columns that start with an underscore "-" when converting Beam Rows to native objects (needed for xlang, yaml)

Removing @RequiresStableInput so the sink can run on the local FnApiRunner (which Python and YAML use). This is semantic anyways because GroupByKey right before the commit already keeps its input stable. Only real behavior change is on Flink runner, which isn't included in tests yet

[codecov]

Codecov Report

❌ Patch coverage is 37.03704% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.78%. Comparing base (76f6e6d) to head (67ffe17).
⚠️ Report is 160 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/yaml/integration_tests.py	0.00%	17 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40542      +/-   ##
============================================
+ Coverage     56.16%   58.78%   +2.61%     
- Complexity     2288    13973   +11685     
============================================
  Files          1124     2595    +1471     
  Lines        178336   272476   +94140     
  Branches       1489    11305    +9816     
============================================
+ Hits         100170   160170   +60000     
- Misses        75645   106278   +30633     
- Partials       2521     6028    +3507     

Flag	Coverage Δ
java	65.05% <ø> (-10.74%)	⬇️
python	79.91% <37.03%> (+0.35%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Assigning reviewers:

R: @tvalentyn for label python.
R: @damccorm for label java.
R: @derrickaw for label yaml.
R: @chamikaramj for label iceberg.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[chamikaramj]

Thanks!

[chamikaramj]

Could you mention in the PR description why these were removed ? Won't this change the behavior for non-Dataflow runners ?

[ahmedabu98]

Done. It would only change behavior for Flink, but we need it to run Python/YAML pipelines

[chamikaramj]

Could you also try these against Dataflow ?

[ahmedabu98]

We don't seem to have Dataflow YAML tests set up. I ran the following on Dataflow though and got the expected output:

DeltaLake to Iceberg:

pipeline:
  type: chain
  transforms:
    - type: ReadFromDeltaCDC
      config:
        table: "gs://ahmedabualsaud/delta/orders"
        start_version: 0
        end_version: 3
        include_metadata_columns: ["_change_type", "_commit_version"]
        hadoop_config:
            "fs.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
            "fs.AbstractFileSystem.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
            "fs.gs.auth.type": "APPLICATION_DEFAULT"
            "fs.gs.project.id": "apache-beam-testing"
    - type: WriteToIceberg
      config:
        table: "shop.orders_from_delta"
        catalog_properties:
          type: "hadoop"
          warehouse: "gs://ahmedabualsaud/iceberg_cdc"
        mode: "merge-on-read"
        equality_columns: ["id"]
        change_type_column: "_change_type"
        change_type_map:
          insert: "INSERT"
          update_preimage: "UPDATE_BEFORE"
          update_postimage: "UPDATE_AFTER"
          delete: "DELETE"
        sequence_number_column: "_commit_version"

Iceberg to Iceberg:

pipeline:
  type: chain
  transforms:
    - type: ReadFromIcebergCDC
      config:
        table: "shop.orders_from_delta"
        catalog_properties:
          type: "hadoop"
          warehouse: "gs://ahmedabualsaud/iceberg_cdc"
        config_properties:
          "fs.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
          "fs.AbstractFileSystem.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
          "fs.gs.auth.type": "APPLICATION_DEFAULT"
          "fs.gs.project.id": "apache-beam-testing"
        include_metadata_columns: ["_change_type", "_commit_snapshot_sequence_number"]
    - type: WriteToIceberg
      config:
        table: "shop.orders_from_iceberg"
        catalog_properties:
          type: "hadoop"
          warehouse: "gs://ahmedabualsaud/iceberg_cdc"
        mode: "merge-on-read"
        equality_columns: ["id"]
        change_type_column: "_change_type"

[damccorm]

4 reviewers is too many :) assigning to Cham since you have the best context and have already started the review - thanks

R: @chamikaramj

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers