feat: bandwidth-throttle chaos event type for cumulative-lag testing (#3379) - #3387
Draft
Nana-EC wants to merge 40 commits into
Draft
feat: bandwidth-throttle chaos event type for cumulative-lag testing (#3379)#3387Nana-EC wants to merge 40 commits into
Nana-EC wants to merge 40 commits into
Conversation
Nana-EC
force-pushed
the
feature/3379-bandwidth-throttle-chaos
branch
from
August 20, 2026 21:27
a0e50e1 to
68fc549
Compare
…riggered (#3348) BackfillPlugin with greedy=false only scans up to the BN's own last stored block (determineEndCap returns storeMax). With asymmetric latency BN1 never misses blocks — TCP retransmit keeps the stream whole — so the non-greedy scan finds no historical gaps and BackfillPlugin stays silent. With greedy=true, determineEndCap queries peers for their max available block (getPeerMaxAvailableBlock). The latency gradient means BN3 is always 1-2 blocks ahead of BN1 at any scan boundary. That range is classified as LIVE_TAIL and submitted to the liveTailScheduler, producing INFO log: "Starting backfillGap type=[LIVE_TAIL] range=[N-N+1]" This fires every 60s scan cycle throughout the chaos window, making the backfill-triggered assertion reliable. Changes: - paired-3.yaml: greedy: false → greedy: true on all three BNs; regenerated out/ overlays will pick this up at task-up time - latency-stress.yaml: restore loss: "75%" (packet loss widens the live-tail gap at scan boundaries, making detection more reliable); restore backfill-triggered with grep: "backfillGap" (targets INFO log exactly) - latency-severe.yaml: same — restore backfill-triggered on BN1 with the "backfillGap" grep pattern; update header comments to explain mechanism Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
All three BNs already have peers configured; enabling greedy=true so BackfillPlugin proactively closes live-tail lag from faster peers, consistent with the same change made to paired-3. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
backfill-triggered could not reliably fire under latency/loss chaos: TCP absorbs all packet loss on kind's loopback bridge so BN1 receives every block and never falls behind peers. The BackfillPlugin's LIVE_TAIL mechanism only triggers when BN1's storeMax < peers' storeMax at a 60s scan boundary, which doesn't happen when TCP delivers all blocks. blocks-converged (tolerance 5) tests what actually matters: after chaos clears, do all BNs converge to the same block height? This confirms that CN↔BN streams survived throughout and any latency-induced lag closed — regardless of whether TCP delivery or backfill achieved convergence. Header comments updated to reflect the new test intent. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
… tests
Both latency-stress and latency-severe now:
1. Capture a snapshot-block-heights event at peak chaos (t=180s stress,
t=210s severe) — records per-BN lastBlock while the fault is live
2. Assert blocks-diverged (min_spread=3) — confirms chaos caused ≥3-block
spread across BNs, proving the latency gradient had real effect
3. Assert blocks-converged (tolerance=5) — confirms all BNs returned to
the same height after chaos cleared
This replaces the single end-state-only check with a proper before/after
pair that detects both divergence under chaos and recovery after it.
Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…failure mode latency-stress: CN↔BN1 800ms→3000ms ±750ms, loss 75%→50% (corr 50%), BN1→BN peers 1000ms→1500ms ±375ms — borderline 'approaching threshold' zone (expected spread 3-8 blocks at peak chaos). latency-severe: CN↔CN 600ms→1000ms, CN↔BN1 800ms→5000ms ±1500ms + 75% loss (corr 75%), CN↔BN2 400ms→2500ms ±750ms + 40% loss (corr 50%), BN1→BN 1000ms→2500ms, BN2→BN 500ms→1000ms — full production failure mode (expected spread 40-80 blocks at peak chaos). At 5000ms one-way + 75% correlated loss on BN1, TCP retransmit backoff (10s→20s→40s→80s→160s) creates genuine cumulative block lag matching the geographic deployment failure observed in production environments. Closes #3348 Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
At 3000ms one-way latency (stress tier), base block lag = 3000ms × 0.5 blk/s = 1.5 blocks. TCP throughput at this latency+loss is ~700 KB/s — far above the ~1-5 KB/s block data rate — so no cumulative queue builds. min_spread: 3 is unreachable with pure netem; set to 1 (onset of lag, borderline). At 5000ms + 75% correlated loss (severe tier), base lag = 2.5 blocks plus burst delays from retransmit backoff. min_spread: 3 may be hit transiently but is unreliable; set to 2 (consistently reachable at peak chaos). Reproducing the cumulative-lag production failure mode requires bandwidth throttling (tc tbf), not just latency+loss. Tracked in #3379. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
… measurement With greedy=true, BackfillPlugin's LIVE_TAIL closes any CN-latency-induced lag within one 60s scan cycle by pulling from BN3 via the unchaosed peer path. No amount of CN↔BN1 latency creates sustained spread — the gap is erased before the snapshot fires. greedy=false forces BN1 to rely solely on its direct CN gRPC stream, making the one-way latency visible as a block-height difference at snapshot time. This is a deliberate testing trade-off: not production-faithful on the backfill dimension, but the only way to measure latency-induced divergence without bandwidth throttling. The bandwidth-throttle approach (which works with greedy=true) is tracked in #3379. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Add Chaos Mesh bandwidth throttle support to the E2E test harness. - New chaos template: network-bandwidth.yaml.tmpl — uses Chaos Mesh action: bandwidth (tc tbf) to cap throughput. Mirrors the structure of network-latency.yaml.tmpl with BANDWIDTH_RATE/LIMIT/BUFFER vars. - New function execute_inject_bandwidth in solo-test-runner.sh — args match execute_inject_latency (name, source/target kind+name, bidirectional) plus rate/limit/buffer with sane defaults (1mbps / 1000000 / 10000). - Event dispatch: inject-bandwidth → execute_inject_bandwidth; clear-bandwidth reuses execute_clear_latency (same kctl delete networkchaos logic, same chaos_resource_name scheme). Closes part of #3379. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Add tests/bandwidth-lag.yaml: - Throttles CN→BN1 and all BN-peers→BN1 to 500kbps using the new inject-bandwidth event type; BN2/BN3 run at full speed as reference. - 500kbps rate: just above the ~400kbps needed to deliver 0.5 blk/s at 100 KB/block, so lag builds slowly and is measurable. - Chaos window: t=30–335s (305s); snapshot at t=180s (peak chaos). - Asserts min_spread≥5 at peak, tolerance≤10 blocks after recovery (BN1 may have a larger backfill gap than in latency tests). - Header comment explains the rate rationale and tuning guidance. - Register bandwidth-lag in both test-definition descriptions in .github/workflows/solo-e2e-test.yml (workflow_call + workflow_dispatch). Closes #3379. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…ing dash
- yq '.bidirectional // true' treats boolean false as falsy and returns
the default true, making bidirectional=false impossible to express.
Use 'if has("bidirectional") then .bidirectional else true end' instead.
- chaos_resource_name slices the combined slug to 63 chars which can
leave a trailing '-' that kubectl apply rejects. Strip it after slicing.
Fixes items from #3312.
Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…d test:scripts task - Update sample-logs-with-backfill.txt: class name was org.hiero.block.node.verification.VerificationServicePlugin (old module); corrected to org.hiero.block.node.block.verification.VerificationServicePlugin. - Add envsubst to the Taskfile check task: chaos latency injection uses envsubst to render network-latency.yaml.tmpl but it was not validated. - Add test:scripts Taskfile task to run test-chaos-assertions.sh without needing a live cluster (previously run-only manually). All 20 script unit tests pass. Closes items from #3312. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…-lag testing Adds two new bandwidth throttle test tiers derived from the production traffic profile (39 CNs worst case / 13 in 3-BN DR split): - bandwidth-lag-severe.yaml (200 kbps): models the DR-split scenario; BN1 receives blocks at ~50% of normal rate, producing ~37 blocks of divergence in the 150s chaos window. Assertion: min_spread ≥ 15. - bandwidth-lag-stress.yaml (50 kbps): models near-complete link failure; BN1 starved to ~12.5% of normal rate, producing ~66 blocks of divergence. Extended load-stop window (510s) to allow post-chaos backfill to begin. Assertion: min_spread ≥ 30. CI: adds both tests to the test-definition description in workflow_dispatch and workflow_call; expands Chaos Mesh install condition to also trigger on bandwidth tests (previously only chaos/latency). Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Nana-EC
force-pushed
the
feature/3379-bandwidth-throttle-chaos
branch
from
August 25, 2026 02:48
68fc549 to
b7b37d9
Compare
The 500kbps cap on CN→BN1 and peer→BN1 in the bandwidth-lag test produced 0 block spread in CI (all BNs at block 176 at peak chaos), failing the blocks-diverged assertion (min_spread=5). The cap wasn't tight enough to push BN1's ingest rate far enough below the observed block production rate within the 150s chaos window. Lowering to 100kbps, per the fallback already documented in the test file's rate rationale comment. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
bandwidth-lag failed with 0 block spread at both 500kbps and 100kbps caps, identical results despite a 5x tighter rate. This points to the NetworkChaos resource not actually taking effect rather than the cap being too loose, but execute_inject_bandwidth/execute_inject_latency never checked whether the applied resource reached AllInjected before the test proceeded — a silent injection failure looked identical to "chaos had no measurable effect" downstream. Add wait_for_networkchaos_injected, called after apply in both inject functions, to fail fast with kubectl describe output when injection doesn't succeed within 30s. Also fixes execute_inject_bandwidth not checking kctl apply's exit code (inject-latency already did). Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
bandwidth-lag has now produced identical zero block-spread results across three CI runs (500kbps, 100kbps, and with AllInjected confirmed) — Chaos Mesh reports the rule as successfully injected, but block production shows no measurable ingest slowdown on BN1. Given the sustained ~6000 TPS load, blocks are almost certainly large enough that a genuine 100kbps cap on the ingest path should cause severe, easily detectable lag. A real effect this small points to the throttle not actually shaping the traffic that carries blocks to BN1, despite Chaos Mesh's own AllInjected status. Add dump_bandwidth_chaos_diagnostics, called after successful injection in execute_inject_bandwidth, to log tc qdisc counters and the chaos-mesh ipset from the matched source pods on the next run — this should show directly whether the intended traffic is being shaped, without needing live cluster access mid-test. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
The tc/ipset diagnostics added in 779f1bf failed outright in CI: "exec: tc: executable file not found in \$PATH" on both the CN (root-container) and BN (block-node-server) app images, since neither ships iproute2/ipset. That left the actual question — whether the Chaos Mesh bandwidth rule is shaping real traffic — still unanswered. Switch to `kubectl debug --target=<container> --image=nicolaka/netshoot`, which attaches an ephemeral container that shares the existing pod's network namespace (no extra privilege or node access needed), and netshoot ships tc/ip/ipset. Target container name is derived from the selector's label key since CN and BN pods use different container names. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
The netshoot-based diagnostics added in abbb7be produced no output and no error in CI — `kubectl debug` without -i/--stdin creates the ephemeral container and returns immediately without waiting for it to start or attaching to it (per its own --attach flag docs: default false unless -i is set). That made every DIAG block silently empty. Add -i so --attach defaults to true, which waits for the container to run and streams its output back, as intended. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Root cause of 5 consecutive CI runs showing 0 block spread regardless of rate (500kbps, then 100kbps) found via the tc qdisc dump added in 38631f2: chaos-mesh's bandwidth "rate" field uses "kbit" for kilobits/s and "kbps" for kilobytes/s (8x kbit) — confirmed directly in CI output, where a configured "100kbps" produced an actual `qdisc tbf ... rate 800Kbit`. All three bandwidth-lag test tiers wrote their rate rationale in kilobits/s (matching networking convention and their own math, e.g. "~50 KB/s = 400kbps needed") but used the "kbps" suffix, so every configured cap was applied 8x looser than intended — enough slack that BN1 never fell behind regardless of the nominal number. Switch all rate values (bandwidth-lag: 100kbps, bandwidth-lag-severe: 200kbps, bandwidth-lag-stress: 50kbps) to the "kbit" suffix with the same numeric value, and update the rationale comments to document the unit distinction so it isn't reintroduced. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
With correct kbit units (2ec035a), the tc dump now shows something more concrete than a loose rate: the tbf (throttled) band shows 0 bytes sent for the *entire* chaos window on every CN and BN pod, while all real traffic flows through the untouched sfq band instead. That points to the ipset-based packet classifier not matching the intended traffic at all, rather than a rate/timing issue. The ipset dump added in abbb7be has been failing with "Kernel error received: Operation not permitted" the whole time — the default netshoot debug container lacks NET_ADMIN. Add --profile=netadmin so `ipset list` can actually run and show whether BN1's pod IP is present in the matched set. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
The netadmin-profile ipset dump (6853f2b) confirmed the tbf ipset had exactly one correct-looking member (BN1's pod IP), yet the throttled tc band still showed 0 bytes for the entire chaos window. Root cause: CN and BN-peers reach BN1 via its k8s Service DNS name, resolving to its ClusterIP, not its pod IP directly. chaos-mesh's bandwidth rule is installed inside the *selector* pod's own netns and matches on pod IPs — with selector=sender/direction=to, the packet is still addressed to BN1's ClusterIP at that point (DNAT to the pod IP happens later, on the host side), so it never matches the ipset. This is a documented chaos-mesh/Service interaction limitation. Flip all three bandwidth-lag test tiers to selector=block-node-1 / target=sender / direction=from, installing the shaping on BN1's own ingress instead, filtered by source IP — which survives the Service DNAT since only the destination gets rewritten. Add an explicit `direction` arg to inject-bandwidth (overriding the bidirectional-derived to/both default) since the existing bidirectional boolean has no way to express "from". Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
The last CI run silently skipped every event after inject-cn-bn1-bw and jumped straight to assertions — 5 events (inject-peer-bn1-bw, snapshot-peak-chaos, both clear-bandwidth events, load-stop) never ran. Root cause: run_events' main loop is `... | jq -c '.[]' | while read -r event; do ...; done` — the whole loop shares stdin with that pipe. dump_bandwidth_chaos_diagnostics (introduced in abbb7be/38631f2b) calls `kubectl debug -i`, which keeps container stdin open and reads from whatever fd0 it inherits. With no redirection, that was the *same* pipe carrying the remaining event JSON lines, so kubectl debug silently consumed them before the next `read -r event` could see them. Redirect stdin from /dev/null on both kubectl debug invocations so they no longer compete with the event loop's piped input. -i is still needed so --attach defaults to true; only the OS-level fd is isolated. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
The selector=BN1/direction=from "ingress" attempt (afe10e4) still showed 0 bytes in the throttled tc band in CI. Root cause: chaos-mesh's bandwidth action is always egress-only, installed on the *selector* pod's own interface — direction only changes which IP field (src vs dst) the ipset filters on, not whether shaping is ingress or egress. Filtering BN1's own egress packets for source=CN can never match (BN1's outbound packets are always sourced from BN1 itself), so this approach was structurally incapable of shaping anything, regardless of rate or units. Revert to the original selector=sender/direction=to (the only combination that installs a real egress qdisc on the traffic's actual source), but resolve the *target* as BN1's Service ClusterIP via chaos-mesh's externalTargets field instead of a pod selector. This matches the packet's real destination address at the point tc/ipset filters it, before kube-proxy's DNAT ever rewrites it to the pod IP — the actual root cause identified via the netadmin ipset dump. - execute_inject_bandwidth: add target.kind=service handling that resolves the named Service's ClusterIP via `kubectl get svc -o jsonpath` and renders it as externalTargets, replacing the pod-selector `target:` block for that case only (inject-latency and pod-selector bandwidth targets are unaffected). - network-bandwidth.yaml.tmpl: replace the static target: block with a ${TARGET_BLOCK} placeholder populated by the script, since the two target modes render structurally different YAML. - All 3 bandwidth-lag test tiers: target: { kind: service, name: block-node-1 } with direction: to, replacing the ingress attempt. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #3387 +/- ##
============================================
- Coverage 83.59% 83.55% -0.04%
+ Complexity 2581 2578 -3
============================================
Files 219 219
Lines 11336 11336
Branches 1328 1328
============================================
- Hits 9476 9472 -4
- Misses 1365 1370 +5
+ Partials 495 494 -1 see 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
dump_bandwidth_chaos_diagnostics (and the netshoot/kubectl-debug plumbing it needed) was scaffolding added purely to find the Service-ClusterIP-vs- pod-IP ipset mismatch, now fixed via externalTargets (f1bf1f7) and documented in chaos-templates/README.md. It added ephemeral-container overhead to every bandwidth injection with no ongoing value now that the root cause is known and handled. wait_for_networkchaos_injected is kept — it's a real hardening (fail fast on injection failure), not exploratory instrumentation. Also trims the inline target.kind=service comment down to a pointer at the new README rather than re-explaining the mechanism inline. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…oach Documents how solo-test-runner.sh wires test YAML events into these NetworkChaos templates, and the three non-obvious Chaos Mesh bandwidth- action behaviors that cost most of the debugging effort in PR #3387: rate units (kbit vs kbps), egress-only tc shaping, and Service-ClusterIP vs pod-IP targeting (externalTargets). Also documents how to manually verify a fault is actually shaping traffic via kubectl debug + netshoot, since AllInjected=True alone doesn't confirm that. Intent: an engineer picking up this test suite should be able to read this once and know what to change for a new latency/bandwidth scenario, without rediscovering these gotchas from scratch. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
The per-file header comments had accumulated a blow-by-blow debugging narrative (every failed selector/direction/unit attempt) as the real root causes were found across several commits. That history now lives in chaos-templates/README.md as a proper reference; trim each test file back to a concise design rationale plus a pointer to the README, so the test files read as current-state documentation rather than a debugging log. Also cross-reference finding 005 (CN↔BN reconnect deadlock after severe bandwidth starvation, agent/proposals/solo-chaos-spike/findings/ — local, not committed) from both tiers, since it directly affects what recovery-assertion failures mean at these rates. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…tigation 50kbit/s (this tier's previous rate) is tighter than bandwidth-lag's 100kbit/s tier, which was enough to permanently deadlock the CN↔BN1 reconnect after chaos cleared (finding 005 — CN never adjusts its requested resume block after a BlockNodeOutOfRange rejection, so it retries a stale block forever). Running stress at 50kbit/s would almost certainly hit the same deadlock, producing no new signal beyond what the base tier already showed. Loosen to 500kbit/s as a provisional value to get a runnable, non-guaranteed-failure test while the deadlock is investigated separately, and loosen blocks-diverged/blocks-converged thresholds to match (previous values assumed a ~66-block spread that this looser cap won't produce). Not yet validated against a real block-size/timeout recalculation — retune once finding 005 is understood well enough to pick a rate deliberately. severe (200kbit/s) is left unchanged per direction — its fate pending the same investigation. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
A second confirmed reconnect-deadlock run (finding 005) plus a closer
read of the CN logs showed the previous rates (50-500kbit/s across all
three tiers) were never actually testing a severity gradient — every
one of them sits ~1000x below the real critical zone, so they all hit
the identical outright per-request-timeout deadlock regardless of the
specific number. Loosening 50kbit/s to 500kbit/s (prior commit) made
no difference for exactly this reason.
Derived two real floors from CN log evidence (block ~2.1MB, ~0.5 blk/s,
CN per-request send timeout ~2.5s):
- sustained-need floor ~8.5 Mbit/s (block_bytes*8*block_rate)
- per-request timeout floor ~6.8 Mbit/s (block_bytes*8/timeout_s) —
below this a single block can't transfer within CN's timeout at
all, which is what triggers the permanent reconnect deadlock.
Re-tiered around that zone, per direction: bandwidth-lag covers a
minimal constraint, bandwidth-lag-stress a significant but manageable
one, and bandwidth-lag-severe the highest constraint before the
breaking point (not past it):
- bandwidth-lag: 100kbit/s -> 15mbit/s (~1.5% of a 1 Gbps link)
- bandwidth-lag-stress: 500kbit/s -> 9mbit/s (~0.9%, just above the
sustained-need floor)
- bandwidth-lag-severe: 200kbit/s -> 7mbit/s (~0.7%, just above the
per-request timeout floor — deliberately the
tier most likely to brush against finding
005's deadlock)
`limit` scaled to ~3MB (roughly one block's worth of queue depth) for
all three tiers to match the new rate scale. Assertion thresholds reset
to conservative starting points since none of these exact rates have
run in CI yet. Documented the full floor derivation in
chaos-templates/README.md so the next person tuning these doesn't have
to rediscover it.
Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…uted topologies bandwidth-lag's blocks-converged/blocks-increasing/block-rate-floor assertions failed on both topologies: once a BN's live publisher connection is torn down under chaos, non-greedy backfill caps its gap-scan window at its own last stored block, so it never looks past that point even though its peers (with independent, unthrottled publishers) are far ahead. Enabling greedy lets the scan window extend to the peers' max available block instead, so a stalled BN can catch back up via backfill regardless of whether its live stream ever recovers. fan-out-3cn-2bn is left as-is: it already sets greedy: true, but BN2 has no publisher of its own and depends entirely on backfill from BN1, so there is no independent unthrottled source to backfill from once BN1's stream dies - running bandwidth-lag against that topology also doesn't match the test's documented topology requirement (paired-3). Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…ss takes effect
execute_load_start read a load-start event's test_class arg, defaulted it,
and logged it, but never exported NLG_TEST_TYPE before invoking
solo-load-generate.sh, which selects its class from
${NLG_TEST_TYPE:-CryptoTransferLoadTest}. Every in-test load-start therefore
always ran CryptoTransferLoadTest regardless of what a test YAML's
load-start.args.test_class asked for.
Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…E export Extracts execute_load_start from solo-test-runner.sh (same sed/eval approach as test-topology-decisions.sh, since the runner has top-level arg parsing that exits under set -u with no --test flag) and mocks solo-load-generate.sh with a stub that captures the environment it actually received. Confirmed this fails against the pre-fix code (NLG_TEST_TYPE unset) and passes with the fix. Wired into task test:scripts so it actually runs. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…s takes effect Same gap as execute_load_start (fixed in 77ae3e2): execute_load_stop read a load-stop event's test_class arg, defaulted it, and logged it, but never exported NLG_TEST_TYPE before invoking solo-load-generate.sh stop. Without this, fixing load-start alone would start a non-default load class correctly but load-stop would still target CryptoTransferLoadTest and fail to actually stop it. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Extends the existing execute_load_start regression test with the matching execute_load_stop case, and isolates every case with an explicit unset of NLG_TEST_TYPE/NLG_ARGS/NLG_MAX_TPS beforehand so a missing export fails deterministically instead of possibly passing on leaked state from an earlier case in the same run. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Exercises the test_class override now that execute_load_start/ execute_load_stop actually honor it, instead of always defaulting to CryptoTransferLoadTest regardless of what a test YAML asks for. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Derives real average bytes-per-block from files_recent_total_bytes_stored / files_recent_blocks_written (both already exposed by BlockFileRecentPlugin) over a sample window, asserted against a min_bytes floor. Gives a direct number instead of inferring block size from the presence/absence of CN-side slow-request warnings. Same "assertions run after all events complete" constraint as block-rate-floor: this measures the post-load-stop/recovery window, not bytes-per-block during active chaos. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Green/red/zero-blocks-written/missing-baseline cases, same fixture-mock pattern as assert_block_rate_floor's coverage in this file. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
min_bytes is a sanity floor, not a tuned expectation -- the point is to read the printed bytes/block value and settle whether a given load (CryptoTransfer vs HCS) is actually producing blocks big enough to stress the bandwidth cap, instead of inferring size from the absence of CN-side slow-request warnings. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
Queried files_recent_total_bytes_stored/files_recent_blocks_written directly, but MetricKey.addCategory(METRICS_CATEGORY) prefixes every metric with "blocknode_" (BlockNodePlugin.METRICS_CATEGORY = "blocknode") before exposition -- same reason block-rate-floor queries blocknode_publisher_highest_block_number_inbound rather than the bare Java key. Confirmed against CI run 33209880050: the assertion failed with "no baseline (... unavailable)" because the unprefixed names don't exist on /metrics. Also tightens the fixture mock in test-chaos-assertions.sh to gate on the correct prefixed names instead of matching whatever name it was given -- the previous version would have passed against either name, which is how this shipped without the test catching it. Verified: reverting just the query names now makes the fixture test fail (confirmed locally, reverted after confirming). Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…oadTest HCSLoadTest ran for the full chaos window (confirmed via CI timestamps) but produced no detectable divergence across 3 consecutive runs -- zero CN-side slow-request/error events at the same 7 Mbit/s cap that reliably trips double-digit disconnects with CryptoTransferLoadTest. Its individual transactions are apparently too small in bytes to stress this cap regardless of submission rate. CryptoTransferLoadTest is the one that's actually been producing the effect this test needs. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
…l counter suffix blocknode_files_recent_blocks_written is a LongCounter; OpenMetrics/Prometheus exposition appends _total to Counter-type metrics, so the actual exported name is blocknode_files_recent_blocks_written_total. blocknode_files_recent_total_bytes_stored is a Gauge and is unaffected. Confirmed against real Grafana metric names. Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
network-bandwidth.yaml.tmpl— renders aNetworkChaosresource withaction: bandwidth(wrapstc tbf, token bucket filter). Caps throughput between source and target pods to a configurable rate.execute_inject_bandwidthfunction insolo-test-runner.sh— mirrors the structure ofexecute_inject_latencybut exportsBANDWIDTH_RATE,BANDWIDTH_LIMIT, andBANDWIDTH_BUFFERinstead ofLATENCY/JITTER/CORRELATION. Includes the same CRD-presence retry and pod-selector dryrun checks.inject-bandwidthroutes toexecute_inject_bandwidth;clear-bandwidthreusesexecute_clear_latency(samekctl delete networkchaos/chaos_resource_namelogic).bandwidth-lag.yaml(baseline, 500kbps) — confirms the failure mode exists; lag builds slowly at just-below-threshold rate.bandwidth-lag-severe.yaml(200kbps) — severe tier; models the DR-split scenario (13 of 39 CNs reaching BN1); expects ≥15 blocks divergence at peak chaos.bandwidth-lag-stress.yaml(50kbps) — stress tier; models near-complete link failure (all 39 CN→BN1 paths effectively blocked); expects ≥30 blocks divergence.fix(chaos): correct bidirectional yq coercion and resource-name trailing dash— yq.bidirectional // truenow usesif has("bidirectional") then .bidirectional else true endso an explicitbidirectional: falseis respected; chaos resource names strip trailing-after 63-char slice.chore(chaos): update stale fixture class name, add envsubst check, add test:scripts task—Taskfile.ymlnow checksenvsubstis installed before chaos injection;test:scriptstask added; stale fixture class name corrected.bandwidth-lag,bandwidth-lag-severe, andbandwidth-lag-stressadded to thetest-definitiondescription string in bothworkflow_callandworkflow_dispatchtriggers. Chaos Mesh install condition now also fires onbandwidthtests.Why bandwidth throttling is needed
The existing
latency-stressandlatency-severetests injecttc netemlatency + packet loss viaaction: netem. Even at 5000 ms RTT + 75% correlated loss, TCP throughput remains ~700 KB/s — roughly 140,000× the block data rate (~1–5 KB/block). TCP has no trouble keeping up; it adds a fixed one-way delay but causes no cumulative queue build-up. The largest realistic block lag isone-way-latency × block_rate ≈ 2–3 blocks.The production failure mode is different: a BN that falls permanently and increasingly further behind the live tip over time. That is a throughput-constrained failure.
action: bandwidth/tc tbfis the only Chaos Mesh mechanism that directly caps ingest throughput, causing the block queue to grow monotonically until chaos clears.Bandwidth test tier rationale
All three tests use the same paired-3 topology (CN1 → BN1 exclusively; BN2/BN3 unthrottled as reference). Rate selection is derived from the production traffic profile:
The
limitparameter is scaled proportionally to each rate (~10s of burst capacity at the throttled rate):BN-peer paths are throttled identically in each tier so greedy backfill cannot neutralise the ingest cap during the chaos window. The stress tier uses an extended
load-stopdelay (510s vs 420s) to give BN1 more time post-chaos to begin closing its large backfill gap.Test plan
TOPOLOGY=paired-3(≥ 3 BNs required for spread measurement)CHAOS_ENABLED=true task chaos:installCHAOS_ENABLED=true TOPOLOGY=paired-3 task test:run TEST_FILE=tests/bandwidth-lag.yamlCHAOS_ENABLED=true TOPOLOGY=paired-3 task test:run TEST_FILE=tests/bandwidth-lag-severe.yamlCHAOS_ENABLED=true TOPOLOGY=paired-3 task test:run TEST_FILE=tests/bandwidth-lag-stress.yamlblocks-divergedsnapshot shows BN1 lagging by ≥ min_spread at each tierblocks-convergedpasses after chaos clears (tolerance scales with tier)blocks-divergedfails, check block sizes in run output; lower rate one tier and re-runtest-definition: bandwidth-lag-severeorbandwidth-lag-stresswithtopology: paired-3Closes #3379
Also addresses #3312