Repository navigation
[Task]: Add perf tests #38559
Description
What needs to happen?
For context, please see https://s.apache.org/beam-delta-lake-source.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
durgaprasadml commentedon May 30, 2026
Hi @chamikaramj, I would like to work on this issue.
I’ve been analyzing the Delta Lake source implementation and the existing Beam IO integration/performance test patterns. My plan is to:
I’ll follow the existing Beam IO connector architecture and keep the implementation modular and contribution-ready.
chamikaramj commentedon May 30, 2026
Sounds good. As discussed in the PR, let's wait for the source implementation with SDF support (#38706) to go in first and then you can add perf testing on top of that.
chamikaramj commentedon May 30, 2026
For perf testing, let's make sure that that we deploy a real Delta Lake instance with a large input size (for example, 100 GB).
We should also use a distributed Beam runner instead of a local runner so that the work get parallelized.
Results can be published to here: https://metrics.beam.apache.org/d/1/getting-started?orgId=1&viewPanel=123125
Documentation on testing: https://beam.apache.org/documentation/io/testing/
durgaprasadml commentedon May 30, 2026
I’ll wait for the SDF-based source implementation in #38706 to land first and then continue the perf-testing work on top of that implementation.
github-actions commentedon Oct 28, 2026
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.