Spark: Implement variant extraction pushdown for shredded VARIANT columns

[qlong]

Change

This PR is part of the work to support variant extraction pushdown, the core change is to engineschema that now maps slots to paths in variant.

• Add SparkVariantExtractionScanBuilder implementing SupportsPushDownVariantExtractions so Spark can push variant_get paths from Filter/Project nodes into Iceberg scans.
• Gate behind new spark.sql.iceberg.variant-extraction-push-down.enabled (default on).
• Use an all-or-nothing batch policy: decline the entire batch if any extraction has an unsupported path, unsupported target type, references a non-variant column, or is a full-variant slot (expectedDataType = VariantType, path $).
• Avoid partial scan rewrites that break multi-variant tables and plans where variant_get above join/aggregate barriers still references the original column.
• Override readSchema() on batch query scans to expose annotated extraction structs to executors.
• Add TestVariantShreddingPushdown for DSv2 plan shape and query correctness.

issue: #16448

Notes for reviewers

• PathUtil.java is mostly copied from existing PR Api: Support variant extract and fix manifest bounds byte order #15384, will rebase once that PR is merged.
• Merge order:
  1. Spark: Add selective shredded variant extraction Parquet readers #16714
  2. Spark: Implement variant extraction pushdown for shredded VARIANT columns #16715

End to end testing

Requires #16714 for end-to-end testing. To try the full pushdown + selective read path without merging locally, use this branch:

https://github.com/qlong/iceberg/tree/variant-extraction-integration-test

Test Results

See performance improvements in #16714

Co-authored with Claude Sonnet 4.6

[qlong]

@rdblue @steveloughran @nssalian PTAL when you get a chance.

[qlong]

Spark side fix for column pruning when variant is pushed down. apache/spark#56556