feat: per-pod chaos selector — asymmetric BN latency injection (#3348)

[Nana-EC]

Problem

Chaos tests injected identical latency across all block nodes simultaneously.
Because every BN was equally degraded, no divergence occurred — the tests were
verifying nothing meaningful about single-node resilience.

Four additional bugs meant chaos was silently never applied at all:

1. envsubst missing on CI runner — chaos manifest template substitution
   produced empty files; kubectl apply on an empty file returned no error.
2. Loss field format — test YAMLs specified loss: "50%" but Chaos Mesh
   requires a plain number string ("50"); the admission webhook rejected the
   manifest and the event was still marked PASS.
3. Silent kubectl apply failure — return code was not checked, so rejected
   manifests caused chaos to silently not apply.
4. BackfillPlugin LIVE_TAIL (greedy: true) — even when chaos did apply,
   backfill polled unchaosed peers every ~60s and pulled missing blocks before
   the mid-chaos snapshot, collapsing any spread to zero.

Changes

Infrastructure fixes

• .github/workflows/solo-e2e-test.yml — install gettext-base so envsubst
  is available on the runner.
• solo-test-runner.sh / execute_inject_latency — strip trailing % from
  loss values; fail fast if kubectl apply returns non-zero.

Per-pod instance selector

• network-latency.yaml.tmpl — added ${SOURCE_NAME_FILTER} /
  ${TARGET_NAME_FILTER} placeholder lines in each labelSelectors block.
  When set they inject app.kubernetes.io/instance: <name> at the correct
  indentation. When empty the line is blank — existing tests unaffected.
• solo-test-runner.sh — extracts optional source.name / target.name
  from event args; builds compound dryrun selector for pod-count validation;
  exports the two filter env vars.

Mid-chaos measurement

• snapshot-block-heights event — queries lastAvailableBlock on every BN
  mid-chaos and writes a snapshot file.
• blocks-diverged assertion — reads the snapshot; fails if
  max − min < min_spread. Proves chaos produced detectable divergence.
• blocks-converged assertion — queries live heights at assertion time;
  fails if spread exceeds tolerance_blocks. Proves streams survived and nodes
  caught up.

Topology change

• paired-3, 7cn-3bn-distributed — all BNs set to greedy: false
  to prevent BackfillPlugin from pulling blocks over the unchaosed peer path
  and masking divergence. See feat: add bandwidth-throttle chaos event type to E2E latency tests to reproduce cumulative-lag production failure #3379 for the bandwidth-throttle follow-on that
  restores greedy: true.

Test definitions

latency-stress — survival test. One BN (BN1) receives high latency +
moderate loss. At this tier TCP throughput still exceeds the block data rate
so lag is constant (~2.5 blocks), not cumulative; divergence is not asserted.
Assertions confirm BN1 stays healthy, keeps receiving blocks, and converges
after chaos clears.

Pod	CN↔BN latency	BN→BN peers
BN1	5000 ms ± 1250 ms + 50% loss, bidirectional	2500 ms ± 625 ms, direction=to
BN2/3	—	—
CNs	100 ms ± 20 ms (background)	—

latency-severe — divergence test. Three-tier gradient; BN1 most degraded,
BN2 mid, BN3 reference. At 8000 ms + 75% correlated loss, burst-loss RTO chains
freeze the CN↔BN1 stream. Spread of ~39 blocks observed consistently.

Pod	CN↔BN latency	BN→BN peers
BN1	8000 ms ± 2000 ms + 75% loss (corr 75%), bidirectional	4000 ms ± 1000 ms, direction=to
BN2	4000 ms ± 1000 ms + 40% loss (corr 50%), bidirectional	2000 ms ± 500 ms, direction=to
BN3	— (reference)	—
CNs	1000 ms ± 250 ms (corr 75%), bidirectional	—

Convergence tolerance is wide (150 blocks) at this tier because the CN↔BN1
stream is severed and BN1 does not reconnect within the test window.
blocks-increasing and block-rate-floor target BN3 only.

Known limitation / follow-on

latency-stress has no block-divergence assertion because the current
tc-netem approach cannot produce cumulative lag when block payloads are
small (~2 KB). #3379 / #3387 (bandwidth throttle via tc tbf) will cap
ingest below block production rate, creating sustained lag with greedy: true.

Test results

Run	Test	Outcome	Spread
31114793205	latency-severe	✅ PASS	39
31105361075	latency-stress	✅ PASS	— (survival)

Closes #3348

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@             Coverage Diff              @@
##               main    #3353      +/-   ##
============================================
+ Coverage     83.37%   83.58%   +0.21%     
- Complexity     2626     2678      +52     
============================================
  Files           228      229       +1     
  Lines         11801    11983     +182     
  Branches       1368     1396      +28     
============================================
+ Hits           9839    10016     +177     
- Misses         1431     1438       +7     
+ Partials        531      529       -2     

see 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jsync-swirlds]

The high latency options seem a bit too high (around 10X any real-world latency that doesn't involve deep-space radio links), in my opinion.

[jsync-swirlds]

Approving to get testing in place, but the extreme latency involved here, compared to larger test environments, suggests we need to look at more than just latency to find the proper edges.

[Nana-EC]

Approving to get testing in place, but the extreme latency involved here, compared to larger test environments, suggests we need to look at more than just latency to find the proper edges.

Yep, already in motion - #3387
Will continue to explore additional items.
Also captured your thoughts in #3503