Repository navigation
Performance Regression or Improvement: sideinpts_python_batch_1gb_1kb_10workers_1000window_99key_percent_dict:runtime #40005
Description
Performance change found in the
test: sideinpts_python_batch_1gb_1kb_10workers_1000window_99key_percent_dict for the metric: runtime.
For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.
Test description: python | Side Input | 1 GB dictionary, 99% of keys, 1000 fixed windows
Test Link -
Test Dashboard - http://metrics.beam.apache.org/d/-E9aGlFGk/side-input-load-tests?orgId=1&from=now-90d&to=now&viewPanel=9
timestamp: Thu Sep 3 20:17:32 2026, metric_value: 1808.00
timestamp: Wed Sep 2 19:47:35 2026, metric_value: 1512.00
timestamp: Tue Sep 1 19:55:57 2026, metric_value: 1275.00
timestamp: Mon Aug 31 19:41:16 2026, metric_value: 1816.00 <---- Anomaly
timestamp: Sun Aug 30 19:04:58 2026, metric_value: 827.00
timestamp: Sat Aug 29 19:02:56 2026, metric_value: 1097.00
timestamp: Fri Aug 28 23:01:31 2026, metric_value: 980.00
timestamp: Thu Aug 27 22:52:17 2026, metric_value: 870.00
timestamp: Wed Aug 26 20:06:49 2026, metric_value: 955.00
timestamp: Tue Aug 25 19:18:13 2026, metric_value: 1217.00
timestamp: Mon Aug 24 19:33:43 2026, metric_value: 1004.00
timestamp: Sun Aug 23 19:04:30 2026, metric_value: 970.00
timestamp: Sat Aug 22 19:29:03 2026, metric_value: 971.00
timestamp: Fri Aug 21 19:09:20 2026, metric_value: 1040.00
Activity
aIbrahiim commentedon Sep 11, 2026
I investigated side input dataflow batch runs around the alert and checked Grafana for (batch / python, from 20 August until now):
https://metrics.beam.apache.org/d/-E9aGlFGk/side-input-load-tests?orgId=1&from=1787173200000&to=now&viewPanel=9
Grafana shows the runtime is usually around 15 to 20 min, with some spikes. Near the alert there is one fast run around 13 to 14 mins then a couple of 30 min spikes, and recent points are back in the normal range
I also compared GHA runs: Test 8 got slower on 31 August, but the similar Test 7 (1% keys) did not jump the same day and container tag was still beam-master-20260817 around that time
It looks unstable or flaky rather than a real slowdown
tvalentyn commentedon Sep 22, 2026
We should try to get to the bottom of this. Do you repro the regression via A/B testing, for example by passing --sdk_harness_container_image ?
aIbrahiim commentedon Oct 2, 2026
@tvalentyn I ran the Side Input Test 8 twice with --sdk_container_image, same test, only the worker image changed
First with beam-master-20260817: ~1803s (~30 min)
https://github.com/apache/beam/actions/runs/36714011882
Then with beam-master-20260831: ~1766s (~29 min)
https://github.com/apache/beam/actions/runs/36838732957
The times are basically the same, so the container change doesn’t look like the cause here and with Grafana also looking noisy (usually ~15 to 20 min, with some spikes), I still think this is unstable load test variance rather than a real lasting regression