Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/geant4-perf-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,25 @@ per-event data products, so the module cannot initialise at
registration time — a phlex design question (one-time/run-level
resources vs per-event products), not an aegir bug.

## Reproducibility and worker-kernel churn

Measured with `gun_st_full` (200 events, fixed gun seed):

- `phlex -j 1`: **reproducible** — the validation histograms
`h_mc_multiplicity`, `h_mc_momentum`, and `h_mc_pdg` are bin-by-bin
identical across runs (checked by `scripts/compare_histograms.py`).
- `concurrency: 1` with `-j 12`: **not reproducible.** Even though only
one `simulate` call runs at a time, TBB hands successive calls to
different pool threads; each new thread lazily builds its own
`G4WorkerRunManagerKernel` with its own RNG stream, so the event
sequence samples different streams run to run. This directly confirms
the worker-kernel churn concern: the kernel count is bounded by the
number of distinct TBB threads that ever touch `simulate`, not by the
module's `concurrency`.
- Consequence for validation: physics comparisons must either run
`-j 1` (exact) or compare distributions statistically. The
single-threaded benchmark recipes in the justfile now pin `-j 1`.

## Voluntary context switches

~2 per event (4.1–4.3 k per 2000-event run), independent of thread
Expand Down
12 changes: 6 additions & 6 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ bench-geant4: build
hyperfine \
--warmup 1 \
--min-runs 3 \
--reference 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_bench.jsonnet)' \
--reference 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_bench.jsonnet) -j 1' \
--reference-name 'G4 single-threaded' \
-n 'G4 multi-threaded ({{g4_concurrency}}T)' 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} --ext-str concurrency={{g4_concurrency}} workflows/gun_mt_noop.jsonnet)' \
--export-markdown bench-geant4.md \
Expand All @@ -95,9 +95,9 @@ bench-geant4-io: build
hyperfine \
--warmup 1 \
--min-runs 3 \
--reference 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_bench.jsonnet)' \
--reference 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_bench.jsonnet) -j 1' \
--reference-name 'G4 ST noop' \
-n 'G4 ST full' 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_full.jsonnet)' \
-n 'G4 ST full' 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_full.jsonnet) -j 1' \
--export-markdown bench-geant4-io.md

# Sweep G4 concurrency levels
Expand All @@ -117,7 +117,7 @@ bench-gun-noop: build
--min-runs 5 \
--reference 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{events}} workflows/gun_noop.jsonnet)' \
--reference-name 'gun noop (no G4)' \
-n 'gun + G4 ST noop' 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_bench.jsonnet)' \
-n 'gun + G4 ST noop' 'PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c <(jsonnet --ext-str events={{g4_events}} workflows/gun_st_bench.jsonnet) -j 1' \
--export-markdown bench-gun-noop.md

# --- Profiling / scaling sweep ---
Expand Down Expand Up @@ -162,7 +162,7 @@ profile_freq := "99"
profile-flamegraph: build
perf record -F {{profile_freq}} -g --call-graph=dwarf -o perf.data -- \
env PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c \
<(jsonnet --ext-str events={{profile_events}} workflows/{{profile_workflow}}.jsonnet)
<(jsonnet --ext-str events={{profile_events}} workflows/{{profile_workflow}}.jsonnet) -j 1
@if command -v flamegraph.pl >/dev/null && command -v stackcollapse-perf.pl >/dev/null; then \
perf script -i perf.data | stackcollapse-perf.pl | flamegraph.pl \
> flamegraph_{{profile_workflow}}.svg; \
Expand All @@ -183,5 +183,5 @@ trace_events := "50"
profile-trace: build
AEGIR_TRACE_FILE=trace_{{trace_workflow}}.json \
PHLEX_PLUGIN_PATH={{plugin_path}} phlex -c \
<(jsonnet --ext-str events={{trace_events}} workflows/{{trace_workflow}}.jsonnet)
<(jsonnet --ext-str events={{trace_events}} workflows/{{trace_workflow}}.jsonnet) -j 1
@echo "Wrote trace_{{trace_workflow}}.json — open in https://ui.perfetto.dev"
Loading