Versioned, reproducible measurement program for Cloud World Model, a Canvas Cloud AI product by Kevin Brown (GitHub canvascloudai, kevin@canvascloud.ai).
The public accuracy page (GET https://www.cloudworldmodel.ai/api/accuracy-benchmark) consumes this dataset. It does not get to vote on it.
v1 owned campaign exists (473f1339-f712-4096-96d6-3d4fc07cb427, filled in holdout/REPORT.md). The typical campaign typical-v1-20260927c is in typical/REPORT.md. After an owned calibration fit (engine 1.2.5, fit only on the 20/100/200 RPS runs), the typical-v1 300 RPS after-fit holdout is 84.2 on the accuracy page's cost basis (82.4 without cost), against 24.3 before the fit (27.0 without cost). Later-day (us-east-2) and second-region (us-west-2) holdouts at 300 RPS scored 76.0 and 81.1 (page cost basis; 73.3 and 79.0 without cost, vs 82.4), driven mainly by P95 (Day and region holdouts). After-fit scores are in typical/scores-after-fit.json. Campaign typical-scale-v1-20261006 is the preregistered app-server count change (1/2/3 × m5.large at 100/200/300 RPS, 3 reps, 27 rungs; App-server count change). Both 2→1 and 2→3 verdicts are "Does not predict the change": CPU direction and size passed at every rung, and P95 at 300 RPS did not (measured +129.9% and -49.8%, predicted direction none). The 1×-300 error rise of +0.0146 pp failed C8. Diminishing returns holds at 300 RPS (gain 2.30× vs 1.99×, DB CPU flat) and CWM does not capture it; it does not hold at 100 or 200 RPS. Connection-class errors were zero, and the 3× peak was 243 connections against a max of 500. Sessions 1–2 reported RDS 8.0.46 and session 3 reported 8.0.46-rds.20260908 (possible confound; see the campaign evidence). A future P95 recalibration fitted on these runs would make them fit data rather than holdouts, and would need a new holdout run. Campaign typical-p95-v1-20261010 is the preregistered holdout of the two-input typical P95 model (1/2/3 × m5.large, 300/200/100 RPS, 3 reps; P95 two-input model holdout). Both verdicts pass: absolute 9/9 cells in tolerance, median score 87.6, and the model beats engine 1.2.18 at every 300 cell (87.6/89.7/68.8 vs 19.1/33.5/61.5). The server-change verdict passes both 300 transitions and 4/4 others. All 9 applies reported RDS 8.0.46-rds.20260908. DB CPU stayed ≤ ~28%, so the DB-saturation regime and anything above 300 RPS are not validated. This is validation only, and engine coefficients are unchanged. Coefficients and the calibration id are unchanged. Burst on this CRUD workload measured ≈ 0% errors. Coefficients are fitted per-metric by OLS on idle / normal / peak only (calibrate/coefficients.yaml). Holdouts report fit_prediction and delta. Do not retune coefficients to raise the public score. Do not copy the public 2% / 9.55% cell as owned data. results/ still has no isExample: false run JSON (CI rejects that path).
Public source of this program: github.com/canvascloudai/cwm-bench. Origin namespace: cloudworldmodel (name the repo cwm-bench).
CWM's public accuracy benchmark uses canonical topology:
ALB → 2 × m5.large → db.r5.large MySQL 8.0 Single-AZ, us-east-1
Live as of 2026-08-29: overallScore 94.7%. The hole is Burst at 1000 RPS:
| Reference (cited) | Simulated | |
|---|---|---|
| Error rate | 2.00% | 9.55% (0% accuracy on that cell) |
| Throughput | 980 RPS | 905 RPS |
The 2% is cited as RDS connection timeout (~500 max connections) from AWS documentation, not a company-owned CloudWatch run. Coefficients on that page are least-squares-fit to documented scenarios. That public 9.55% cell is not owned data.
This repo now holds an owned v1 campaign. On this CRUD workload, Burst-class canonical / pool-bound / app-bound runs measured ≈ 0% errors. That does not match the public 2% literature cell. Do not add a CPU-error term for CRUD Burst. cpu-only shows CPU can fail when forced. Coefficients are a per-metric OLS fit of idle / normal / peak. Do not retune anything to raise 94.7%.
The owned campaign's primary region was us-east-2 (not the us-east-1 pin below). Second-region holdout was us-west-2.
This repository is the program that produced that dataset: terraform for the topology, a tiny reference app, k6 rungs and diagnostics, schemas the accuracy page must ingest, a per-metric OLS fitter that refuses a composite score, and a filled holdout report.
Also in CONTRIBUTING.md and in comments on CI.
- Never optimize the composite accuracy score.
- Fit per-metric only (CPU, P50/P95/P99, goodput, error-by-class, connections) on a declared fit split.
- Hold out Burst, plus a later day and a second region.
- Coefficients ship with measurement SHA, fit split, holdout deltas. A coefficients change without a new measurement ID is rejected.
- v1 owned campaign is present in
holdout/REPORT.md. Coefficients are fitted per-metric from owned idle / normal / peak. Do not invent CloudWatch. Do not copy the public 2% / 9.55% Burst cell as owned data. Cost from the public price list can be measured. - gp2
BurstBalancehitting 0 is a third error bucket (iops_throttle), distinct from CPU failures and DB connection failures. Do not fold IOPS throttle into either.
| Tier | Pin |
|---|---|
| Region | us-east-1 (a second region is holdout) |
| Edge | Application Load Balancer |
| App | 2 × m5.large, Amazon Linux 2023, Node 20, gp2 root 30 GiB |
| Database | 1 × db.r5.large, MySQL 8.0, Single-AZ, gp2 100 GiB |
| Generator | c6i.xlarge (4 vCPU compute-optimized) running pinned k6 v0.54.0 |
AMI is resolved from the public SSM parameter /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 unless you set ami_id. That parameter drifts. Record resolved_ami_id at apply time. See terraform/README.md.
APP_POOL_SIZE defaults to 250 so two nodes can present 500 sessions toward the parameter-group max_connections override (also default 500). That pairing is a declared topology choice so the pool-bound diagnostic can saturate the cited cap. It is not a CloudWatch observation. The RDS MySQL formula DBInstanceClassMemory/12582880 is higher on db.r5.large (~1365). We override to 500. Details in terraform/README.md.
gp2 burst is 3000 IOPS; baseline is GiB × 3. When BurstBalance hits 0, count iops_throttle. The app never emits that class.
| Name | Target RPS | Script |
|---|---|---|
| Idle | 10 | load/scenarios.js SCENARIO=idle |
| Normal | 100 | SCENARIO=normal |
| Peak | 500 | SCENARIO=peak |
| Burst | 1000 | SCENARIO=burst |
Protocol: warmup, then 15 min steady. Defaults: WARMUP=5m, DURATION=15m. Mix: 70% GET product, 20% list, 10% write.
Burst is holdout. Do not put it in the fit set.
load/diagnostics.js — same protocol, 1000 RPS. These classify saturation. They do not assert a root cause of the public 9.55% vs 2% cell.
DIAGNOSTIC |
App config | Path |
|---|---|---|
pool-bound |
expects APP_POOL_SIZE=250 (default apply) |
CRUD mix |
app-bound |
expects APP_POOL_SIZE=40 (re-apply first) |
CRUD mix |
cpu-only |
pool n/a | GET /api/cpu-spin only |
If generator CPU exceeds ~70% in the steady window, discard the run.
A second workload lives in app-typical/ and stays off unless Terraform is applied with app_profile=typical and app_workers=2. The default remains the lean app (app_profile=lean, app_workers=1). Scenario keys are typical-fit-20, typical-fit-100, typical-fit-200, typical-holdout-300, and optional typical-saturation-500. Those keys run only in us-east-2. Two further holdout applies use the same typical mix at 100 RPS and 300 RPS: typical-later-day and typical-later-day-300 (us-east-2, a later UTC day than the typical-v1-20260927c fit date 2026-09-27) and typical-second-region and typical-second-region-300 (us-west-2 only). Each apply runs both rates before destroy. They do not loosen the us-east-2 lock on the other typical keys, and they are not the lean later-day / second-region keys. The frozen definition is typical/PREREGISTRATION.md (section 12 is the holdout amendment; section 13 points at the app-count campaign typical/scale-v1/PREREGISTRATION.md). Nine validation keys, typical-scale-{1x,2x,3x}-{100,200,300}, run that campaign in us-east-2. Each app count is its own apply (-var app_count=N). They are not a change to the frozen typical-v1 scores. Runner steps are typical/RUNBOOK-CANVASCLOUDAI.md. The measured campaign is typical/REPORT.md, with the numeric export in holdout/exports/typical-v1-20260927c.summary.md. That report also records the engine 1.2.5 after-fit holdout; the scores are typical/scores-after-fit.json, and python3 typical/after-fit/score_export.py reproduces them from typical/after-fit/. The day and region holdouts are holdout/exports/typical-holdouts-v1-20261006.summary.md, typical/scores-holdouts-v1-20261006.json, and typical/campaign/typical-holdouts-v1-20261006/. The app-server count campaign is holdout/exports/typical-scale-v1-20261006.summary.md, typical/scores-scale-v1-20261006.json, typical/scale-v1/score_scale.py, and typical/campaign/typical-scale-v1-20261006/. The P95 model holdout is holdout/exports/typical-p95-v1-20261010.summary.md, typical/scores-p95-v1-20261010.json, typical/p95-v1/score_p95_v1.py, and typical/campaign/typical-p95-v1-20261010/. The next campaign, typical-dbsat-v1 (DB breakpoint discovery on 2/3/4 × m5.large, ascending 300 → 1,500 RPS), is preregistered in typical/dbsat-v1/PREREGISTRATION.md. Its scorer is typical/dbsat-v1/score_dbsat_v1.py, the runner-side controller contract is typical/dbsat-v1/CONTROLLER-SPEC.md, and the descriptive engine 1.2.21 predictions are typical/dbsat-v1/predictions/engine-1.2.21/. It is not run yet.
app/ Node 20 reference CRUD + seed SQL (8 categories, 200 products, 0 orders)
app-typical/ Flag-gated typical profile (separate package and seed)
load/ k6 scenarios + diagnostics + typical mix + later-day / second-region holdout keys
terraform/ Canonical topology (AWS provider 5.x). fmt/validate in CI; no apply.
typical/ Pre-registration, runbook, predictions, typical/REPORT.md, scores, and typical/after-fit/
schema/ draft 2020-12. EXAMPLE fixtures only (isExample: true).
results/ README + .gitkeep. No runs.
calibrate/ Per-metric OLS (idle/normal/peak). Holdout deltas reported. Refuses composite scores.
holdout/ REPORT.md filled from owned lean campaign 473f1339…; typical exports are holdout/exports/typical-v1-20260927c.summary.md, holdout/exports/typical-holdouts-v1-20261006.summary.md, holdout/exports/typical-scale-v1-20261006.summary.md, and holdout/exports/typical-p95-v1-20261010.summary.md
scripts/ ci.sh + worker-adapter.mjs (Admin Benchmarks worker contract)
The Admin Benchmarks worker capability-checks a pinned revision, then calls this contract. Unknown commands and unknown scenarios fail.
node scripts/worker-adapter.mjs wait-ready --json
node scripts/worker-adapter.mjs run --scenario <scenario-key> --json
node scripts/worker-adapter.mjs collect --scenario <scenario-key> --jsonwait-ready returns adapterVersion (1.6.1) and the full
supportedScenarios list (idle, normal, peak, burst,
pool-bound, app-bound, cpu-only, later-day, second-region,
typical-fit-20, typical-fit-100, typical-fit-200,
typical-holdout-300, typical-saturation-500, typical-later-day,
typical-later-day-300, typical-second-region,
typical-second-region-300, typical-scale-{1x,2x,3x}-{100,200,300},
and typical-p95-{1x,2x,3x}-{300,200,100}).
later-day is a real later-UTC-day holdout, not an alias of normal.
second-region is a real us-west-2 holdout, not a rename of the
us-east-1 run. typical-* keys require app_profile=typical and two
workers. They require region us-east-2, except typical-second-region
and typical-second-region-300, which require us-west-2.
typical-later-day and typical-later-day-300 also require a later UTC
day than the fit campaign (CWM_FIT_CAMPAIGN_DATE or adapter state).
typical-scale-* keys are holdout-split validation in us-east-2 and
require topology.app_count 1, 2, or 3. typical-p95-* keys are the
P95-model holdout on the same counts, ladder 300 then 200 then 100.
Every other typical key expects
2 app servers. A mismatch fails with APP_COUNT_MISMATCH before k6 starts.
A lean key on a typical stack, or the
reverse, fails with PROFILE_MISMATCH before k6 starts. When the stack
is up, wait-ready reports each app node's profile, workers, and
gitSha from /api/meta. Burst is not a capability skip. collect must
return a complete run (required CloudWatch datapoints, including ALB,
plus k6 summary.json with latency percentiles and error-class
counts) or it fails with COLLECT_INCOMPLETE. Empty CloudWatch stays
null. Nothing is invented. See scripts/README.md.
The Admin Benchmarks v1 campaign is collected (see holdout/REPORT.md).
Burst-class CRUD error on that workload is ≈ 0%. Coefficients are a
per-metric OLS fit of idle / normal / peak. The adapter still requires
a complete collect for burst; it will not write fake CloudWatch into
results/.
App check used by the worker and CI:
cd app && npm ci --ignore-scripts && npm run checkYou need an AWS account you control. This repo ships no account IDs and no credentials.
cd terraform
terraform init
terraform fmt
terraform validate
terraform plan -var='test_id=YYYYMMDD-your-campaign' -out=tfplan
terraform apply tfplan
terraform outputSet ami_id and app_source_git_ref to this commit SHA for a campaign you intend to keep. App and generator user_data check out that exact URL and SHA. They do not fall back to main, master, HEAD, or another repository. Copy alb_dns, generator_ip, dashboard_url, resolved_ami_id into the campaign record.
Destroy when finished: terraform destroy -var='test_id=YYYYMMDD-your-campaign'. RDS delete waits up to 60 minutes. If subnet-group / ENI / SG destroy races RDS deletion, use scripts/terraform-destroy-retry.sh (cleanup only; not a campaign result). An interrupted apply should default to cleanup, not resume measurement — the worker owns that policy.
The worker must isolate campaign working directories by claim. This repository does not fix the lost-checkout race of a shared campaign directory disappearing mid-apply.
Full notes: terraform/README.md.
On the generator (SSM Session Manager; scripts live at /opt/cwm-bench/load):
source /opt/cwm-bench/TARGET.env
export CAMPAIGN_ID=YYYYMMDD-your-campaign
export RUN_ID=idle-1
export SPLIT=fit
export RESULTS_DIR=results/raw/$CAMPAIGN_ID/$RUN_ID
mkdir -p "$RESULTS_DIR"
SCENARIO=idle k6 run --out json="$RESULTS_DIR/k6.json" load/scenarios.jsRepeat for normal, peak, burst (SPLIT=holdout on Burst). Then the three diagnostics. Copy results/raw/... off the instance. Never overwrite.
Full invocation: load/README.md.
results/raw/{campaign_id}/{run_id}/
One campaign per directory. Never overwrite. After CloudWatch is pulled, write a schema-valid run.json (schema/run.schema.json). CI rejects any file under results/ that claims isExample: false until you actually have one — and you must not invent it.
The only JSON run document in this repository today is schema/example-run.json. It is marked EXAMPLE. It does not look like a campaign and it does not use the live CWM Burst numbers.
- Collect schema-valid runs.
split=fitexcludes Burst, a later day, and a second region. - Fit each metric independently (OLS on the intended loss in
calibrate/README.md). - Write
calibrate/coefficients.yamlwithmeasurement_sha,fit_split,holdout_deltas,created_at. - Fill
holdout/REPORT.md. A coefficients PR without those deltas, or with Burst in the fit set, is rejected. - The accuracy page reads this dataset. It does not average it with documentation citations to hide Burst.
v1 holdout measurements are recorded. coefficients.yaml is a per-metric OLS fit of idle / normal / peak (metrics.* non-null; holdout fit_prediction / delta present). python3 calibrate/calibrate.py recomputes that fit from calibrate/fit_input.yaml and checks the committed file. --composite-score exits non-zero.
| Artifact | State |
|---|---|
| Topology as code | Present |
| Reference app + seed | Present |
| k6 rungs + diagnostics | Present |
| Run / campaign schemas | Present (EXAMPLE fixture only) |
| Raw results | None under results/ (CI still rejects isExample: false). Typical collect JSON is in typical/campaign/typical-v1-20260927c/. Day and region holdout evidence is in typical/campaign/typical-holdouts-v1-20261006/. App-server count evidence is in typical/campaign/typical-scale-v1-20261006/. P95 model holdout evidence is in typical/campaign/typical-p95-v1-20261010/ (request-level k6 files kept off-repo) |
| Coefficients | Per-metric OLS on idle / normal / peak; metrics.* fitted |
| Holdout tables | Filled from owned campaign 473f1339-f712-4096-96d6-3d4fc07cb427; predictions + deltas |
| Burst | Owned CRUD Burst-class error ≈ 0% on this workload (not the public 2% / 9.55% cell) |
GitHub Actions (.github/workflows/ci.yml) and bash scripts/ci.sh:
- terraform fmt + validate (no apply)
- schema-validate EXAMPLE fixtures
k6 inspecton load scripts- calibrate provenance (fitted metrics + holdout deltas; Burst not in fit_split)
- holdout v1 tests (REPORT filled, metrics fitted, no owned 9.55)
- calibrate OLS reproducibility + refuse composite score
- reject
results/files that claimisExample: false - worker-adapter unit tests (AWS mocked; no fabricated campaign)
python3 -m pip install -r calibrate/requirements.txt
# terraform >= 1.5, k6 0.54.x, Node 20 for app --check
bash scripts/ci.sh