Support core dump analysis with pystack and gdb. by tvalentyn · Pull Request #39484 · apache/beam

tvalentyn · 2026-07-24T14:13:18Z

Support analyzing core files with pystack when python SDK processes crash with segfaults.

Sample usages:

python my_pipeline.py --runner=DataflowRunner --profiler_agent=coredump --disk_size_gb=100

Stack traces from the crash diagnostics are saved into text files that are then synced to GCS. Excerpt of the analysis is published directly into logs: we attempt to find print the stacktrace for the thread that has the GIL and surface it; the complete analysis can be retrieved from GCS (--profle_location or gs://<temp_location>/profiles by default).

Arguments to pystack can be customized via the --profiler_extra_args pipeline options.
For more information, see: https://bloomberg.github.io/pystack/ and pystack --help .

Currently this will be supported with Dataflow from Beam 2.76.0 but it should be possible to extend support to other runners.

Requires either pystack, or optionally gdb be installed in the SDK container image.
Pystack might be able to provide most insight, but if that is not sufficient users can install gdb into the container image, then we also attempt to run various gdb analysis commands and save the output into a file.

To control how often core files are postprocessed and saved, use the --profile_upload_interval_sec and --profile_postprocess_interval_sec.

Core files are deleted after the analysis to save disk space. We don't dump cores on demand, but only perform a core analysis when when a crash happened.

The --profiler_stop_after_sec and --profiler_stop_after_crash options are also supported however it does not stop core files from being saved to disk, but stops the cleanup, which may be undesirable.

Sample analysis of a manufactured segfault (retrieved from GCS).

core.python.31-20260728181728.txt

core.python.31-20260728181728.gdb.txt

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-07-24T14:13:21Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

github-actions · 2026-07-24T17:04:17Z

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

gemini-code-assist · 2026-07-28T18:06:52Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

claudevdm · 2026-07-30T01:16:23Z

+	// options experiments (which is automatically set by the Python SDK). This ensures
+	// that core dumps are written in /tmp/ with a prefix matching "core.".
+	coreDir := "/tmp"
+	interval := 5 * time.Second


Is the default meant to be 5 seconds? It seems the pipeline option defaults to 60 sec

beam/sdks/python/apache_beam/options/pipeline_options.py

Lines 1724 to 1726 in 58bac32

'--profile_postprocess_interval_sec',

type=int,

default=600,

So the 5 seconds will only survive if someone explicitly sets the pipeline option to 0, otherwise it will default to 600 sec?

Is 600 too long to catch segfault on startup?

yeah, this 5 second constant is useless, let's remove this altogether.

Re 600... not sure; probably won't eat a ton of CPU to do it more frequently, we can lower to 60sec. the ticker should not run concurrently, perhaps the issue might happen if the core cannot be processed or deleted, and keeps getting reprocessed again and again... hopefully that's not common

actually i'll keep it 600 sec since the same param is used by other profilers, which might be resource-intensive. Users can set the lower value manually but I suspect this will be good enough.

claudevdm · 2026-07-30T01:25:30Z

+		return
+	}
+
+	prefix := "core."


Is it a bit broad to match on any file starting with core.? Maybe we can add a bit more validation/regex?

good point, fortunately we control the pattern, so we can specify a more unique prefix. beam_py_coredump perhaps.

changed the prefix.

claudevdm · 2026-07-30T01:46:32Z

 						logger.Warnf(ctx, "Python (worker %v) exited %v times: %v\nrestarting SDK process",
 							workerId, errorCount, err)
 					} else {
 						logger.Fatalf(ctx, "Python (worker %v) exited %v times: %v\nout of retries, failing container",


The last crash's core dump seems to depend on the ticker firing before Fatalf exits the container (default interval is 600s, and fresh cores <2s old are skipped).

Should we do one explicit core-dump sweep + GCS sync right before failing the container, so the crash that killed the worker is always analyzed regardless of timer timing?

yes, the last crash is not super reliable... dataflow might also tear down the VM before processing finishes.

we could try to save core files into a semi-persist dir, which would survive container restarts, instead of /tmp/ . Let me check if that will work. I wary of increasing complexity of the boot.go.

Added a callback + explicit postprocessing before crashing the container.

claudevdm · 2026-07-30T01:56:04Z

+		case <-ticker.C:
+			processNewCoredumps(ctx, logger, pcfg, coreDir)
+			if isProfilerDisengaged(pcfg) {
+				return


Is this a race with the 2s file-age guard in processNewCoredumps? With --profiler_stop_after_crash.

Should we do a final synchronous sweep here that waits out the age guard (sleep >2s, re-run processNewCoredumps, sync results to GCS) before returning?

good catch, yes, it is a race and we should fix this corner case.

Change the logic to wait until the age is older than 2 seconds.

claudevdm · 2026-07-30T01:56:07Z

+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+			processNewCoredumps(ctx, logger, pcfg, coreDir)


Should we syncProfilesToGCS here if the write was success?

currently profile postprocessing and upload are not coordinated (for this and other profilers). Since we write postprocessed content to a semi-persist directory, it should survive container restarts, so it should not be critical, and the logic is simpler. it is also possible to adjust the uploads to be more frequent.

…tloader

…ing the container.

github-actions Bot added python docker labels Jul 24, 2026

github-actions Bot added the Next Action: Reviewers label Jul 24, 2026

tvalentyn marked this pull request as draft July 28, 2026 16:59

tvalentyn force-pushed the pystack_coredump branch 2 times, most recently from 62f50c9 to 2a68dfc Compare July 28, 2026 17:56

tvalentyn marked this pull request as ready for review July 28, 2026 18:06

tvalentyn changed the title ~~[feat] Support core dump analysis with pystack.~~ Support core dump analysis with pystack and gdb. Jul 29, 2026

claudevdm reviewed Jul 30, 2026

View reviewed changes

tvalentyn added 10 commits July 31, 2026 11:36

Refactor process tracking and signal handling in Python container boo…

e3d4f43

…tloader

Support collecting and processing dumped core files with pystack.

fa7e534

Terminate postprocessing after profiler was disengaged.

74fc9fc

Also save core analysis in text files.

44cb07f

Support gdb

8ca1062

Log gdb commands.

39610a0

Change prefix

ec754be

Respect the postprocessing interval supplied in options.

e8927e1

Register a callback to do a final profile postprocessing before crash…

08421a6

…ing the container.

Wait until core file creation finishes.

b5ce587

tvalentyn force-pushed the pystack_coredump branch from b7d95d3 to b5ce587 Compare July 31, 2026 18:38

Uh oh!

Conversation

tvalentyn commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist Bot commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tvalentyn Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tvalentyn Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tvalentyn commented Jul 24, 2026 •

edited

Loading

tvalentyn Jul 31, 2026 •

edited

Loading

tvalentyn Jul 30, 2026 •

edited

Loading