perf(bomet): live-deployment validation run + deployment-configurable harness

[dhruv-1001]

closes #1854

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: 2c431cd2-95d9-4f36-a3d3-de25ce880f95

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[subhashini-egov]

@dhruv-1001 Can we add the hardware provisioned on this machine?

[dhruv-1001]

Added a Machine section with the full spec:

CPU	AMD EPYC-Rome, 16 vCPU
Memory	30 GiB
Disk	305 GB SSD (non-rotational)
OS / runtime	Ubuntu 24.04.4 LTS, Docker 29.4.0
Virtualisation	KVM guest
Services	59 containers (full DIGIT stack)

Two corrections while checking this, both now fixed in the doc:

1. I had described bomet as bare metal — it isn't. systemd-detect-virt reports kvm, so it's a VM sharing 16 vCPU across the whole stack.
2. I had called it a live production deployment. It's a shared dev environment that happens to carry real data and real daily usage (~2,250 complaints, 20–100/day), so I've described it that way instead.

I also recorded the idle baseline — load ~5.5–6.9 with 4–7 GB free before any test load. That matters for reading the rest: this box isn't quiet at rest, so every figure sits on top of existing load.

[subhashini-egov]

What's the max number of concurrent users? How long did we sustain the test? What was the ramp up period? @dhruv-1001

[dhruv-1001]

Added a Test Shape section. Every ramp uses warmup → ramp-up → sustained hold → ramp-down, and only the main scenario is measured (warmup is excluded from all thresholds and figures):

Scenario	Warmup	Ramp-up	Sustained at peak	Ramp-down	Peak VU
ramp-2vu	1 VU / 2m	2m	5m	1m	2
ramp-10vu	2 VU / 2m	2m	5m	1m	10
ramp-50vu	5 VU / 2m	3m	5m	2m	50
ramp-nvu (75–150)	VU/10 / 2m	3m	5m	2m	75–150

Direct answers:

• Max concurrent users tested: 150 VU (up from 50 — I added the 75/100/125/150 steps to answer your other comment).
• Sustained: 5 minutes at each peak.
• Ramp-up: 2 minutes to 10 VU, 3 minutes above that.

One caveat now recorded in the doc: the 2/10/50 VU runs were taken overnight (00:47–01:15 EAT, idle load ~5.5) while the 75–150 VU ceiling runs ran during working hours (12:47–13:40 EAT, idle load ~6.9, less free RAM). The ceiling figures are therefore slightly conservative.

[subhashini-egov]

0.7/s seems too small a throughput no? @dhruv-1001

[dhruv-1001]

Good catch — you were right, and it was two separate things.

1. The number was genuinely understated. Those were run averages, which dilute the peak with the warmup, ramp-up and ramp-down phases. Recomputed as steady-state over the 5-minute hold at peak:

Test	Was (run-avg)	Steady-state	API req/s
ramp-2vu	0.154/s	0.214/s	0.86
ramp-10vu	0.730/s	1.054/s	4.20
ramp-50vu	3.332/s	5.165/s	20.66

So the real figure was 40–55% higher than published. The doc now reports steady-state and adds an API req/s column, since one lifecycle = 4 API calls.

2. Even corrected, it's small on purpose — it's think-time-bound, not server-bound. Each lifecycle carries ~8s of scripted sleep (4 random 1–3s pauses modelling a human) plus ~1.6s of requests, so one VU can do at most one lifecycle per ~9.5s. Theoretical max is VU / 9.5:

Peak VU	Theoretical	Measured	Ratio
10	1.05/s	1.054/s	100%
50	5.26/s	5.165/s	98%
75	7.89/s	7.947/s	101%
100	10.53/s	9.798/s	93%
150	15.79/s	10.851/s	69%

Up to 75 VU throughput tracks the client-side think time to within 1–2%, which means the server imposed no measurable throughput limit at all below that. The divergence from 100 VU onward is where the server actually becomes the constraint — that's the real signal, and it was missing from the original section.

[subhashini-egov]

What's our upper limit? HOw many TPS can we handle before things start to break? @dhruv-1001

[dhruv-1001]

This one needed new tests — nothing broke at 50 VU, so the original PR genuinely didn't know. I've now stepped 75 → 100 → 125 → 150 VU until a threshold tripped.

Answer:

Max sustainable concurrent users	125 VU
Max sustained throughput	~43 API req/s (~10.8 lifecycles/s)
Breaking point	150 VU — end-to-end p95 hits 16.11s vs the 15s threshold
Failure mode	Latency, not errors

Full curve (steady-state over the 5-minute hold):

Peak VU	Lifecycles/s	API req/s	http p95*	txn p95	Success	HTTP fail	Verdict
50	5.165	20.66	510ms	11.70s	100%	0%	pass
75	7.947	31.78	446ms	11.31s	100%	0%	pass
100	9.798	39.20	754ms	11.99s	100%	0%	pass
125	10.785	43.15	1316ms	13.50s	100%	0%	last clean level
150	10.851	43.37	2312ms	16.11s	100%	0%	BREACH

* includes ~185ms network RTT.

Three things worth flagging:

• Throughput plateaus at ~43 req/s. Going 125 → 150 VU bought +0.5% throughput while p95 latency grew 76%. That's a saturated system, not a cliff.
• Nothing ever failed. http_req_failed stayed at 0.000% and transaction_success at 100% at every level including the breach — roughly 77,000 requests in the ceiling runs alone. Under saturation this stack queues rather than shedding load or erroring. The breach is purely on end-to-end duration; http_req_duration p95 (2312ms) was still well inside its own 5000ms threshold.
• CPU is the binding constraint. Idle time hit 2–3% from 100 VU up, while memory never dropped below 1.3 GB of 30 GB and no container was OOM-killed. Adding CPU should move this ceiling; adding RAM won't.

For context, ~43 req/s sustains roughly 932,000 lifecycles/day — about 93× the 10,000 txn/day design target, against Bomet's actual 20–100/day.

Implementation note: I added ramp-nvu.js (parameterised by __ENV.VUS) rather than using burst.js, because burst.js declares no thresholds and tags its scenario default, so the {scenario:main}-scoped thresholds never evaluate — it can't fail, which makes it useless for ceiling work.

All 19,235 test complaints were removed afterwards via a gated transaction that preserved every non-test row, including ones filed by real users mid-test.

[dhruv-1001]

@subhashini-egov all four comments addressed in 87d7611 — replies inline. Summary:

#	Ask	Resolution
1	Hardware	Machine table added. Corrected: it's a KVM guest, not bare metal, and a shared dev env rather than "live production" as I'd written
2	Concurrency / duration / ramp	Test-shape table added. 150 VU max, 5 min sustained at each peak
3	"0.7/s seems too small"	You were right — those were run averages. Steady-state is 40–55% higher; also explained it's think-time-bound below ~75 VU
4	Upper limit / TPS	New tests. Ceiling measured: 125 VU / ~43 API req/s; breaks at 150 VU on latency, not errors

Two things you didn't ask for but should know about, both in the same commit:

1. This PR had a regression, now fixed. My earlier change added trailing locality/city params to createComplaint(), but seed-1m.js, seed-calibrate.js and variable-throughput.js duplicate the lifecycle instead of importing pgrLifecycle, so they still called it with 7 args and would have written complaints with city/locality undefined. Fixed both ways — the params now default to the seed values so any un-updated caller is safe, and those three scenarios are made env-aware like pgr-lifecycle.js.

2. Credential-leak risk in the repo, unrelated to this PR. performance/k6/config/environments.js is created by copying the .example and holds real deployment credentials. It's gitignored in the upstream load-tests repo but was not ignored here, so a git add -A would have committed it. Added the rule. No leak has occurred — only the .example is tracked — but worth knowing it was possible.

On test data: all 19,235 complaints created during the ceiling runs were removed via a gated transaction that verified non-test rows were preserved exactly, including complaints filed by real users while the tests ran. Server ended healthy — 59/59 containers up, no OOM kills, no restarts.

One caveat on the ceiling numbers: they were measured during Kenyan working hours on a box with an existing idle load of ~6.9 and ~4 GB free, so they're slightly conservative versus an idle box.

[dhruv-1001]

Restructured — the raw k6 output is out, the run is now a docs set

This PR was 156,514 insertions across 56 files, almost all of it raw k6 output. It is now 718 insertions across 14 files.

What changed

Raw artefacts removed. The seven performance/results/20260824-* run directories (metrics.csv, k6-output.json, gzipped CSVs) and the seven results/baseline/bomet-*.json summaries are no longer committed — ~22 MB that could not be reviewed in a diff. Nothing in the repo reads them. A gitignore rule keeps future run artefacts out; the March 2026 runs stay tracked, since ignore rules don't affect already-tracked files.

The 153-line section appended to performance/docs/findings.md is now 19 lines — a summary and a pointer.

The run is published as its own docs set at performance/docs/run-24-08-26/, mirroring the structure already in performance/docs/:

index.md	Overview and what makes this run different
executive-summary.md	Headline numbers
findings.md	Machine, methodology, capacity curve, degradation
recommendations-transition-plan.md	Sizing and scaling in business terms

Numbers are mapped through the framework this site already uses: steady-state throughput over the 5-minute hold, lifecycles × 4 for API TPS, lifecycles × 86,400 for daily capacity, and the existing 1 VU ≈ 20–30 real users conversion. VitePress builds clean, so the new nav has no dead links.

Headline

125 VU sustained · 43.3 API req/s · ~935K lifecycles/day · zero failed requests at every level. The 150 VU breach is on end-to-end transaction_duration (16.14s p95 vs a 15s budget), which includes ~8s of scripted think time. http_req_duration — actual server latency — was 2,326ms against its own 5,000ms budget, so by the metric that measures the server directly nothing had broken at 150 VU.

Three things the write-up now says that it didn't before

1. Bomet is running without any of the three PR perf: add missing PGR address index + disable workflow fuzzy search #248 database fixes. Verified against this repo's deploy path: no idx_eg_pgr_address_v2_parentid (only a locality index), no composite workflow indexes, no GIN trigram index, no jit = off, and EGOV_WF_FUZZYSEARCH_ISFUZZYENABLED unset. The March tier figures explicitly assume all three are applied — so Bomet sits outside that assumption. Its database is small (~2,250 complaints) so this costs little today, but the March data shows an unfixed deployment degrading ~9.4× by 100K records.

2. 125 VU and the March ~300 VU are not the same measurement. March marks where errors appear (connection exhaustion, PgBouncer timeouts). Bomet's 125 marks where a latency budget breaks with zero errors. Bomet's error ceiling was never probed — deliberately, on a live deployment — and is strictly higher. The earlier version of this section left the two numbers side by side without reconciling them.

3. The host metric sample logs were not retained. The load/CPU/memory table is carried forward from the run-time analysis and cannot be re-derived from what's in the repo. Said so explicitly rather than presenting it as reproducible.

Every table states how it was computed, so the figures can be regenerated from a rerun.

[Lokendra-egov]

@dhruv-1001 Two things before merging:

1. race-probe.js will crash on dev/prod targets. env.serviceCodes/env.localities have no fallback there (unlike every other scenario file in this PR), so a plain k6 run scenarios/race-probe.js throws on iteration one.

2. apply-cpu-profile.py's new docker ps lookup isn't scoped to a compose project anymore. remove_limits() now resets CPU quotas on every labeled container on the host, not just this stack's, risky on a shared box.

[dhruv-1001]

@Lokendra-egov Both correct, both fixed in 47359f63.

1. race-probe.js on dev/prod

Confirmed exactly as you described — it read the keys raw where every other scenario resolves them through a fallback:

const svcCodes = env.serviceCodes;                  // undefined outside the bomet block
const serviceCode = svcCodes[i % svcCodes.length];  // throws before the first request

SERVICE_CODES and LOCALITIES are now exported from pgr-lifecycle.js and imported here, so the probe uses the same already-resolved values rather than duplicating the 33 seed codes.

There was a third gap in the same three lines, which I'd rather flag than leave for someone to hit: env.citizenPhone and env.citizenName were also passed straight through, and createComplaint has no default for either. That one wouldn't have thrown — it would have quietly filed complaints with an undefined mobile number, which is worse than crashing because nothing surfaces it. Both now default to the generated identities pgr-lifecycle.js already uses.

2. apply-cpu-profile.py scoping

Also confirmed. get_running_containers() ran docker ps with no --filter, so it matched every compose-managed container on the host and remove_limits() cleared quotas across all of them.

The constraint worth naming: resolving by the com.docker.compose.service label was a deliberate earlier fix, because docker compose ps requires a compose filename that multi-file deployments don't have. So the fix had to keep label resolution and add scoping on top:

• Project comes from --project, else $COMPOSE_PROJECT_NAME, else auto-detected when exactly one compose project is running.
• With several running and none named, it refuses rather than guessing. Single-host behaviour is unchanged; the shared-host case is the one that's now safe.
• remove_limits() additionally skips containers already at cpu.max, so it only touches what a profile actually limited.
• playbook-profile.yml threads an optional compose_project through.

Verification

• All five resolve_project paths exercised: explicit, env var, single auto-detect, several → refused, none → refused
• --remove --project no-such-project exits 1 having touched nothing
• All 14 k6 harness files pass node --check; the script compiles

One thing your review exposed beyond the two bugs: my pre-push gate was node --check on every scenario, which proves syntax and nothing else. Neither of these would ever have failed it — the first needs a target whose config lacks optional keys, the second needs a second compose project on the host. Both are runtime-shape problems that a parse check cannot see. Worth a smoke run per target in CI rather than a syntax gate; I haven't added that here.