Fix flaky Python tests

[reuvenlax]

No description provided.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[github-actions]

Assigning reviewers:

R: @tvalentyn for label python.
R: @derrickaw for label yaml.
R: @igorbernstein2 for label bigtable.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[derrickaw]

LGTM for yaml area

[tvalentyn]

thanks for the help fixing flakes

[tvalentyn]

Let's use existing utils or tenacity for the retry decorator. I'd try

from apache_beam.utils import retry

gcp_retry = retry.with_exponential_backoff(
    num_retries=3,
    initial_delay_secs=2,
    retry_filter=retry.retry_on_server_errors_timeout_or_quota_issues_filter
)

self.input_topic = gcp_retry(self.pub_client.create_topic)(
    name=self.pub_client.topic_path(self.project, INPUT_TOPIC + self.uuid)
)
...

[reuvenlax]

done

[tvalentyn]

            ('PermissionDenied' in str(exn) or 'NotFound' in str(exn) or

would this silently disable the test due to permission issues?

[reuvenlax]

hmmm, yes. I don't think we should be skipping tests to hide vertex failures. Removing this.

[tvalentyn]

nit: we could use tenacity (already a beam dep for retries like this):

from tenacity import Retrying, stop_after_attempt, wait_fixed

# MongoDbContainer's entrypoint restarts mongodb after initial scripts; wait
# until the server is stable, accepting connections and responds to ping.
for attempt in Retrying(stop=stop_after_attempt(15), wait=wait_fixed(1)):
    with attempt:
        mongo_client = mongo_container.get_connection_client()
        mongo_client.admin.command('ping')

[reuvenlax]

done

[github-actions]

Reminder, please take a look at this pr: @tvalentyn @derrickaw @igorbernstein2

[codecov]

Codecov Report

❌ Patch coverage is 6.25000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.91%. Comparing base (76f6e6d) to head (5edc935).
⚠️ Report is 160 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/yaml/integration_tests.py	0.00%	6 Missing ⚠️
.../apache_beam/yaml/test_utils/datadog_test_utils.py	0.00%	5 Missing ⚠️
sdks/python/apache_beam/io/gcp/bigtableio.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #40297       +/-   ##
=============================================
+ Coverage     56.16%   79.91%   +23.74%     
=============================================
  Files          1124      487      -637     
  Lines        178336    82308    -96028     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65775    -34395     
+ Misses        75645    16533    -59112     
+ Partials       2521        0     -2521     

Flag	Coverage Δ
go	?
java	?
python	79.91% <6.25%> (+0.35%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[tvalentyn]

please don't squash reviewed and unreviewed commits as it hides the diff.