[Task]: Migrate Java SDK GcsUtil from gcsio to the java-storage client #40712

What needs to happen?

Umbrella issue for moving the Java SDK's GCS access (GcsUtil, used by GcsFileSystem and anything reading or writing gs:// paths) off the gcsio / google-api-services-storage stack (GcsUtil V1) and onto the Cloud Storage Java client library google-cloud-storage (GcsUtil V2).

Background

GcsUtil is a facade over two implementations:

• GcsUtilV1: the existing gcsio-based implementation.
• GcsUtilV2: the new implementation built on com.google.cloud.storage.Storage.
  V2 was introduced behind the use_gcsutil_v2 experiment. It then gained feature and performance parity with V1, with parity integration tests (GcsUtilIT) running both implementations side by side.

Rollout plan

1. Implement V2 behind an opt-in experiment (use_gcsutil_v2). Done.
2. Reach feature and performance parity, with test coverage. Done.
3. Make V2 the default. Keep disable_gcsutil_v2 as a fallback; use_grpc_for_gcs also stays on V1. In progress.
4. Let V2 run as the default for at least 3 releases and fix whatever turns up.
5. Remove V1 and the gcsio dependency, and retire the use_gcsutil_v2 / disable_gcsutil_v2 experiments.

Experiments

Completed work

• GCS client library migration in Java SDK, part 1 (GCS client library migration in Java SDK - part 1 #36876)
• GCS client library migration in Java SDK, part 2 (GCS client library migration in Java SDK - part 2a #37502)
• GCS client library migration in Java SDK, part 2b (GCS client library migration in Java SDK - part 2b #37592)
• Remove test-only functions in GcsUtil and move related unit tests to V1 (GCS client library migration in Java SDK - part 2c #37597)
• GCS client library migration in Java SDK, part 3 (GCS client library migration in Java SDK - part 3 #37900)
• Extend GCS performance metrics and unify the GCS metric namespace (Extend GCS performance metrics and unify the GCS metric namespace #40142)
• Fill gaps between GcsUtilV2 and GcsUtilV1 (Fill gaps between GcsUtilV2 and GcsUtilV1 #40244)
• Route the remaining GCS calls through GcsUtilV2 when enabled (Route the remaining GCS calls through GcsUtilV2 when enabled #40273)
• Re-arrange GcsUtil unit tests and integration tests (Re-arrange GcsUtil unit tests and integration tests  #40279)
• Add V2 unit tests and V1/V2 parity ITs for GcsUtil (Add V2 unit tests and V1/V2 parity ITs for GcsUtil #40319)
• Parallelize per-file rewrites in GcsUtilV2 copy and move (Parallelize per-file rewrites in GcsUtilV2 copy and move #40376)
• Close GcsUtilV2's small-file performance gaps with GcsUtilV1 ([Java] Close GcsUtilV2's small-file performance gaps with GcsUtilV1 #40677)

Remaining work

• Use GcsUtil V2 by default, with disable_gcsutil_v2 as the fallback ([Java] Use GcsUtil V2 by default #40713)
• Decide whether V2 should support gRPC, or whether use_grpc_for_gcs goes away along with V1 (mentioned in GCS client library migration in Java SDK - part 3 #37900 (comment))
• Let V2 run as the default for at least 3 releases
• Remove GcsUtilV1 and the gcsio dependency
• Retire the use_gcsutil_v2 and disable_gcsutil_v2 experiments

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner