Benchmarking - DO NOT MERGE#4487
Open
prakhargarg105 wants to merge 529 commits into
Open
Conversation
translateInfraSource was emitting YAML for nested-map var values, which terraform's -var parser rejects. Switch to json.Marshal so the comment that already says "JSON-encoded" is now true. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
terraform -chdir=<stack> changes the working directory before resolving -backend-config, so a repo-relative backend.hcl path is interpreted inside the stack directory and fails to read. Resolve to absolute up front. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three issues caught on the first real apply against AWS:
1. Security group descriptions used an em-dash; AWS rejects non-ASCII in
GroupDescription. Replace with hyphens.
2. aws_s3_bucket_lifecycle_configuration rule now requires an explicit
filter or prefix in newer provider versions. Add filter {} to fall
under the all-objects default.
3. The destroy defer was registered after the shared apply succeeded,
so a failed shared apply left orphan resources. Move the defer
before any apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wal_level isn't a user-settable parameter on AWS RDS — setting it directly fails ModifyDBParameterGroup with "Could not find parameter with name: wal_level". The RDS-specific equivalent is rds.logical_replication=1, which makes RDS set wal_level=logical for us. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
16.4 is no longer available in us-east-2 (RDS retires patch versions). 16.14 is the current latest in the postgres16 parameter group family. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
combineReset was writing \$POSTGRES_DSN as a shell-variable reference, but the SSM script environment doesn't carry that variable, so psql fell back to the local Unix socket and failed before the first sweep point. Substitute the DSN value directly (same pattern as renderWorkloadScript and runSeeder), and wrap with set -euo pipefail + ON_ERROR_STOP=1 so a real psql failure surfaces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
postgres_cdc and other enterprise connectors refuse to start without a license. Add a --license-file flag (defaulting to $REDPANDA_LICENSE_FILEPATH), upload the license to the staging bucket alongside the binary + config, and set REDPANDA_LICENSE_FILEPATH=/opt/bench/license.jwt in the bench script env so Connect picks it up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
stat() succeeds even when macOS TCC / sandbox blocks read, so the validation passed but stageArtefacts later failed after ~8 minutes of terraform apply. Open the file for read at validate-time so the permissions error surfaces before any AWS provisioning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add *.license, *.jwt, rpcn_license, and rpcn.license patterns so a license file dropped at the repo root for benchmarking doesn't get accidentally committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
postgres_cdc unconditionally reads its `tls:` field and overwrites the
sslmode parsed from the DSN (see input_pg_stream.go:303). When the
scenario didn't set `tls:`, FieldTLS returned a disabled tls.Config, so
postgres_cdc connected without encryption and RDS rejected the
replication slot with 'no pg_hba.conf entry ... no encryption'.
Add tls: { enabled: true, skip_cert_verify: true } so the replication
stream is encrypted. skip_cert_verify is acceptable for this benchmark
because the connection stays inside the bench VPC.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
postgres_cdc uses NewTLSField (not NewTLSToggledField), so there's no 'enabled' toggle — the presence of the tls: block implies TLS. Drop the 'enabled: true' line that caused lint failure at every sweep point. Also: if the FIRST sweep point captures zero samples (Connect failed to start or the connector errored for the whole window), bail out of the sweep with a clear error rather than burning the remaining ~50min of sweep windows watching the same failure repeat. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the brief 30-line README with a complete guide covering first-time setup, running benches, architecture, adding scenarios + connectors, known limitations (SSM truncation, TLS field shape, RDS quirks, SSO timeout, macOS TCC, clock skew), troubleshooting, and cost estimates. Captures everything learned during the foundation's smoke runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add .claude/ (Claude Code harness state) and /runner (stray binary sometimes left at the repo root by go-run when invoked from there) so neither lands in commits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The benchmark processor emits bytes/sec via humanize.Bytes which uses SI suffixes (B, kB, MB, GB). The previous regex matched only MB/sec, so any sweep point under ~1 MB/sec was silently dropped to zero samples. Normalise every unit to MB/sec so percentiles across a sweep stay unit-consistent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SSM GetCommandInvocation caps StandardOutputContent at ~24KB, which the 1s rolling-stats lines overflow within ~2 minutes. Sweep summaries were landing with 1-2 samples and meaningless percentiles. Connect now writes to /tmp/bench-<vcpu>.log on the runner; the script uploads to s3://<results-bucket>/runs/<session>/sweep-<vcpu>.log after clean termination, and MatrixRunner fetches+parses it via a new LogFetcher interface. SSM stdout only carries framework status echos plus a per-minute heartbeat (the latest rolling-stats line), well under the SSM cap so the operator still sees live throughput. On the early-abort path (0 samples on the first point), the log tail is dumped to stdout so Connect's underlying error is visible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous workload (5K writes/sec, single goroutine) capped the sweep at exactly the producer's ceiling (6 MB/s) and Connect had spare CPU at every point — the sweep measured the producer, not Connect. - Scenario: write_rate_per_sec 5000 -> 80000 (~96 MB/s target) so the producer is no longer the binding constraint at low vCPU counts. - Seeder workload(): single goroutine + single connection capped around 30-40K inserts/sec on c8g.large regardless of what was asked. Spread across 8 workers with pgxpool MaxConns=8, each pushing a smaller batch per 100ms tick. RDS can now actually receive at the requested rate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First AWS run with the bucket-1 framework fixes — 899 samples per sweep point captured cleanly through S3, percentiles real, no SSM truncation. Findings: - 1 vCPU is CPU-bound at ~76 MB/s — the postgres_cdc read ceiling for this row shape and batch size. - 2-4 vCPU is producer-bound at ~95-99 MB/s — Connect has headroom; the load-gen + RDS write path caps at the workload's 80K writes/sec. A larger workload (or per-point table reset) is needed to find the multi-vCPU ceiling. - 8 vCPU is omitted, with rationale. By sweep point 4 the orders table had grown to ~305M rows during the earlier points; per-insert latency on the b-tree stretched until the producer effectively stalled and Connect read 0 msg/sec for the full window. Cost: ~$6 across two retries (one diagnostic, one publishable). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Covers orphan-cleanup Lambda (TTL-based, auto-destroy via AWS API), cost-check subcommand (Cost Explorer, last-7d + MTD + per-instance-type), Prometheus snapshot capture (continuous 10s scrape, curated subset in result JSON, anomaly correlation), and SUMMARY.md auto-refresh (marker-delimited section, regen subcommand for offline use). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each plan is independent and subagent-executable: - cost-check (smallest, no infra, CE auth path) - SUMMARY.md auto-refresh (pure code, marker-bounded section) - Prom snapshot capture (sidecar scrape + curated parse + anomaly context) - Orphan-cleanup Lambda (biggest — Lambda code + IAM + EventBridge + SNS) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds SummariseCosts (renamed from plan's Summarise to avoid collision with stats.go's existing Summarise([]Sample)) together with cost_test.go covering aggregate, empty, and error-propagation cases. Last7Days boundary is exclusive of the 7-days-ago anchor (matching the "14..20" intent in test comments). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces summary.go with the summaryRow type and walkResults/derivedRow functions that discover <results-root>/<connector>/<scenario>/*.json files, pick the newest per scenario (lexicographic sort on timestamp-prefixed names), and return one summaryRow per scenario sorted alphabetically. Five table-driven tests in summary_test.go cover: empty dir, single file, newest-wins, multi- scenario alphabetical order, and all-zero-peaks sentinel behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the summary-section.md.tmpl embedded template and a renderSection function to summary.go that executes it, with TDD coverage for the one-row, empty-rows, and zero-peak-shows-dash cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Connect's iceberg output is commit-latency-bound: serialized ~320ms Glue snapshot commits with no commit-linger, so under default config it commits ~300 records at a time at ~0.03 CPU cores and is flat across vCPU (~412 rec/s). The redpanda input alone sustains ~57k rec/s into a drop output, so the input is not the limiter — the small per-commit record count is. Thread a top-level `buffer` from the scenario pipeline block to the Connect config root (mirrors the existing cache_resources passthrough) so a fast input can be decoupled from the commit-bound output. With a memory buffer + larger batches, each commit carries ~50k records instead of ~300. Validated (connect-only 1-vCPU smoke): mean 412 -> 10,984 rec/s (~27x), peak ~35k rec/s, and CPU 0.03 -> 0.99 cores (now CPU-bound, so it scales with vCPU instead of being flat). orders-sink-smoke.yaml carries the tuned config (buffer 512MiB, max_in_flight 8, count 50000, period 10s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Smoke ran Connect-only at 1 vCPU for 15min: median 5.7 MB/s, 3865 msg/sec, 0 anomalies. Producer-bound — load-gen sustained ~4K PutItems/sec out of 5K target before hitting BatchWriteItem InternalServerError near end. Connector ceiling is well above this; needs higher PutItem rate to probe. Also fixes scenario YAML: `billing: PROVISIONED` was being forwarded to `terraform apply` by translateInfraSource and rejected as an undeclared variable. The TF module hardcodes PROVISIONED anyway, so the key is removed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Auto-generated results appended across the iceberg-sink bench work: 1-vCPU smokes, the full [1,2,4,8] both-engine sweep, and the buffered Connect runs. See results/iceberg/orders-sink*/ for the raw JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match Connect's commit cadence to KC's Tabular coordinator (10s) using input-side batching, so the comparison is tuned-vs-tuned rather than default-vs-tuned. Add a pipeline.input_options passthrough in sinkTopology.Pipeline that merges scenario-supplied redpanda-input tuning (e.g. unordered_processing) into the input, protecting the bench-managed connection fields (brokers/topic/group) from being clobbered. orders-sink.yaml now configures input_options.unordered_processing (enabled, checkpoint_limit 100000, batching count 50000 / period 10s) + output max_in_flight 8 / batching 50000 / 10s. Input batching adds the read-ahead the default path lacks (measured ~110x over baseline, output_sent ~46k rec/s at 1 vCPU), so each commit carries ~10s of records like KC's coordinator. Trades cross-partition ordering (fine for a sink; the buffer variant is the order-preserving equivalent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repetitive 'x' padding compressed ~150x in Parquet, making committed- bytes/sec meaningless and giving KC an absurd ~7 B/record. Sample each record's payload from a 16 MiB pool of random alphanumeric bytes (JSON-safe) via a random window, and vary region/status across small sets. Payloads are now distinct and barely compressible, so Parquet file sizes are representative and the MB/s axis is symmetric across engines for a fair head-to-head. Pool + window keeps seeding fast (no per-byte RNG on the hot path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tic payload) Tuned-vs-tuned head-to-head: Connect input batching at KC's 10s commit cadence, high-entropy realistic payload. Both records/s and MB/s now agree. Connect scales (15k->178k rec/s across 1->8 vCPU), KC plateaus (56k->88k); KC leads at 1-2 vCPU, Connect crosses over by 4 vCPU and ends ~2x ahead at 8. results/iceberg/orders-sink/2026-06-04T19-01-06Z.json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…quota
The reset bash recreated the source table with hardcoded
WriteCapacityUnits=10000 — fine for the 1-vCPU smoke (reset doesn't
fire) but would tank the table mid-sweep. Wire read_capacity /
write_capacity through the dynamodb stack outputs and the
aws_dynamodb_cdc engineSpec's ExtraEnvVars, so the bash references
${READ_CAPACITY}/${WRITE_CAPACITY} and tracks scenario sizing
automatically.
Also resize the workload after the 120K WCU smoke hit AWS's default
per-table quota (47-min CreateTable hang) and the 10K PutItems/sec
smoke saturated 40K WCU late in the run. New sizing: 40K WCU
provisioned, 9K PutItems/sec sustained — 10% headroom under quota.
Smoke redpanda-data#3 result (10 MB/s p50 @ 1 vCPU) appended to dynamodb.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The drop+recreate-table-between-points pattern (mirroring postgres/mysql) doesn't fit DDB. The aws_dynamodb_cdc connector reads from Streams, not table items, so leftover rows can't leak between sweep points. All we need to reset is the connector's CHECKPOINT table — dropping it forces the next process to fall back to start_from:latest and skip to the current stream tail. The table-recreate path also kept hitting AWS's per-table teardown latency (20-30min for a 40K WCU table), causing TF's default 10m delete timeout to fire mid-destroy. Override to 30m so the final teardown completes cleanly; this is the price of keeping the source table around across sweep points (~150GB by end of full sweep, negligible storage cost). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seeder's writeBatch returned the first BatchWriteItem error verbatim — including transient AWS errors like InternalServerError and ProvisionedThroughputExceededException — which killed the worker goroutine and cascaded to the whole workload exiting. This masqueraded as a Connect-side throughput drop in earlier runs: the bench saw rates collapse from 36 MB/s to 0 (workload fully died) or to ~11 MB/s (workload partially died, some workers continued) after ~9 minutes, and the connector was incorrectly suspected. Two changes: - writeBatch now retries up to 8 times with exponential backoff (50ms→2s, ~6s total) on any non-context-cancellation error, not just UnprocessedItems. - After max retries, the batch is dropped rather than returned as an error. Sustaining the configured rate is more important than exact-once delivery for a bench — better to underdeliver one batch than terminate the load generator. Verified at 1 vCPU: throughput now sustained 36 MB/s for 10 minutes (was: 36 MB/s for 9 min, then collapse to 0 or ~11), 0 anomalies. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the four-point sweep result (36→15→12→0 MB/s) with the analysis showing it's an artefact of the bench reset strategy not fitting DDB Streams' shard lifecycle, not a Connect anti-scaling behaviour. Lists falsified hypotheses (AWS GetRecords throttle, GOMAXPROCS lock contention, downstream ShouldThrottle backpressure) and the load-gen-side bugs found during the investigation (4K cap, transient error worker death) which were fixed at d19c10b / 817a6a3 / e2fb34e. Identifies the open root cause hypothesis (shard rotation outpacing the connector's 30s shard refresh interval) and three paths forward for someone returning to this work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…iant Switch orders-sink.yaml from input batching (unordered_processing) to a top-level memory buffer (512 MiB) + output batching at KC's 10s cadence. The buffer decouples the fast redpanda input from the commit-latency-bound iceberg output the same way, but PRESERVES cross-partition ordering (no unordered_processing). Measured equivalent throughput in isolation. Input batching variant preserved in git history (a1d88d9). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ults Add iceberg-findings.md — the full RPCN-vs-KC writeup: fair tuned-vs-tuned comparison (Connect scales, crosses over KC at 4 vCPU; KC efficient at 1-2), the commit-latency-bound default trap (~412 rec/s) and the buffer/input-batching fixes, the buffer-vs-input-batching ordering/throughput tradeoff, negative results (max_in_flight inert, coalescing doesn't fix low-core), and the low-core per-record-CPU gap + recommendations. Plus the buffer-variant full sweep results appended to iceberg.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single-table dynamodb_cdc is capped at ~36 MB/s by DynamoDB's per-table 40K WCU quota, which Connect already saturates at 1 vCPU — so the CPU sweep showed a flat line, not a scaling curve. Create N source tables via for_each so the producer can exceed the per-table quota: total provisioned WCU = len(table_names) x write_capacity, while each table stays within its own 40K quota. - dynamodb-bench module: table_name (string) -> table_names (list), one table per name via for_each; outputs kept as joined strings. - runner translateInfraSource: JSON-encode []any so YAML sequences forward as valid HCL list -vars (covers infra.source.table_names); unit test added. - scenario cdc.yaml: 2 tables (76K total WCU, ~72 MB/s offered), total rate 18K/sec, account-level WCU quota pre-flight documented; stale drop+recreate comments removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2-table sweep on c8g.4xlarge (1/2/4/8 vCPU). Broker-derived p50: 36 -> 66 -> 73 -> 73 MB/s (rolling-stats p50: 40 -> 72 -> 81 -> 82). Near-linear 1->2 vCPU (~1.8x), plateau by 4 vCPU at the source ceiling. Confirms the CON-485 DescribeStream shard-pagination fix end-to-end: the 8-vCPU point holds a steady 82 MB/s (p5 78, p95 84) where the earlier single-table run collapsed to 0 MB/s from shard-rotation stall. Connect is source-bound at 4+ vCPU; its own ceiling is higher and would need 3+ tables (account WCU quota increase) to find. Raw JSON in S3 (results/*.json is gitignored). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Connect oracledb_cdc vs Debezium Oracle on RDS Oracle 19c, both mining the same redo via LogMiner. Adds the rds-oracle TF module, oracle stack, cdc-rows-oracle seeder (seed/workload/exec), and the orders-cdc scenario. Smoke + full [1,2,4,8] sweep run clean end-to-end (both engines stream Oracle CDC). Two non-obvious fixes baked in: - Reset self-stages the seeder from S3 (stage/<seeder>) before TRUNCATE: reset runs on the runner host, but the seeder is only staged on the load-gen host during seed. Oracle has no psql/mysql CLI so it shells out to the seeder's exec subcommand; postgres/mysql dodge this via network CLIs. Affects any no-CLI-engine reset. - stream_snapshot: true (not false). With an empty checkpoint cache, false falls through to FindStartPos -> oldest available redo SCN, so at low vCPU Connect grinds historical redo and never reaches the live workload (0 MB/s, no error). true snapshots the empty table instantly and streams from CURRENT_SCN, matching Debezium. Known limitation: the bench is currently source-bound (~13 MB/s flat across 1-8 vCPU; workload lands ~10K rows/sec vs 150K target), so it does not yet expose Connect CPU scaling. IDENTITY sequence CACHE 100000 helped only at the margin; root cause (write commit/redo path vs Connect LogMiner read cadence) still under investigation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the finding from the oracledb_cdc vs Debezium AWS sweep: Connect streams Oracle CDC at ~13 MB/s on a single LogMiner session, flat across 1-8 vCPU and flat across scn_window_size (20K-200K). Read-bound on LogMiner fetch — write path proven to do 41 MB/s, CPU and SCN-window have no effect. Matches Debezium and Joseph Woodward's independent local bench (redpanda-data#4082): a LogMiner protocol limit, not a Connect limit. Records why no CPU-scaling curve is produced and why multi-table splitting (which scaled DynamoDB via real stream shards) is neither effective nor representative for Oracle's single shared redo log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-ran the full both-engines sweep on benchmarking @ 2c2341b (after merging upstream/main with the oracledb_cdc perf work). Connect rose 13 -> 19 MB/s (+46%) from connector efficiency (redpanda-data#4533 scanner, redpanda-data#4531 session handling) — still flat across 1-8 vCPU, still single-session LogMiner-bound. At sustained throughput Connect (19) edges Debezium (~17); Debezium's high 1-2 vCPU figures are non-reproducible warm-up bursts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # benchmarking/aws/runner/kcconnectors.go # benchmarking/aws/runner/main.go # benchmarking/aws/runner/scenario.go # benchmarking/aws/scenarios/dynamodb/cdc.yaml # benchmarking/aws/terraform/modules/dynamodb-bench/main.tf # benchmarking/aws/terraform/modules/dynamodb-bench/outputs.tf # benchmarking/aws/terraform/modules/dynamodb-bench/variables.tf # benchmarking/aws/terraform/shared/runner-user-data.tftpl # benchmarking/aws/terraform/stacks/dynamodb/main.tf # benchmarking/aws/terraform/stacks/dynamodb/variables.tf # docs/benchmark-results/SUMMARY.md # docs/benchmark-results/dynamodb.md
- Rename seeders/cdc-rows -> cdc-rows-postgres so the four CDC seeders read
consistently (cdc-rows-{postgres,mysql,oracle}, cdc-ddb). Updates the
postgres scenario, runner tests + testdata, and doc references.
- Add runner/doc.go: a package reading guide mapping the ~45 files into
logical groups (entry, specs, topology, render, metrics, infra, output).
- Refresh README: accurate Status table + file-reference tree for all five
connectors (postgres/mysql/oracle/dynamodb/iceberg), drop done "future work".
No behaviour change. Builds, vets, runner tests pass, all 5 scenarios validate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # .gitignore # docs/benchmark-results/iceberg.md
Apply the upstream Iceberg tuning recipes (e49fdbf) to the sink bench scenarios and record full-sweep results on the merged upstream code (benthos v4.76.0, incl. shredder schema-field alloc caching). Scenarios: - orders-sink (Recipe A, order-preserving): memory buffer + batch_policy + output batching, max_in_flight=16. Restoring output-level batching is what un-starves the committer's coalescer (0.8 -> 12.7 MB/s at 1 vCPU smoke). - orders-sink-recipe-b (new, Recipe B, unordered): unordered_processing input batching, max_in_flight=32, no buffer. - orders-sink-smoke: mirror the order-preserving primary sweep. Head-to-head (mean MB/s, 178 GB, c8g.4xlarge, connect vs kafka_connect): 1v: A 15.9 / B 38.3 / KC 45 2v: A 69 / B 64 / KC 63 4v: A 114 / B 99 / KC 71 8v: A 109 / B 122 / KC 74 B wins at 1 and 8 vCPU (2.4x at 1v); A edges 2-4v. Both beat KC at 2+ vCPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rite-up Curated summary of the vCPU-scaling sweep: environment, the two Connect tuning-recipe configs (order-preserving A, unordered B), mean-MB/s results table, scaling analysis, and recommendation. Companion to the auto-appended iceberg.md tables. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Connect mongodb_cdc vs Debezium MongoDB comparison bench. MongoDB is not an RDS engine, so the source is a self-hosted single-node replica set (rs0) on EC2 rather than a managed DB. Artifacts: - terraform/modules/mongodb-ec2: EC2 mongod on local NVMe, cloud-init installs mongodb-org + initiates rs0. Runs in a PUBLIC subnet with a public IP: the shared VPC has no NAT gateway, so private-subnet instances have no egress (they can't install packages or register with SSM). SG still restricts 27017 to the bench client SGs, so mongod is not internet-exposed. 100 GB oplog. - terraform/stacks/mongodb + shared bench_host_instance_profile output. - scenarios/mongodb/orders-cdc.yaml: memory checkpoint cache (mongodb_cdc requires one), bash reset via the seeder's exec (drop+recreate collection). - seeders/cdc-rows-mongodb: seed/workload/exec via mongo-driver/v2, with a connection-retry that tolerates mongod cloud-init still starting. - engineSpecs + kcConnectorSpecs["mongodb_cdc"] (Debezium MongoDB via mongodb.connection.string) + main.go collection.include.list formatting. - Debezium MongoDB 2.7.3 plugin in the runner cloud-init. Validated by a 1-vCPU both-engines smoke: stable non-zero throughput for both engines (connect ~26 MB/s, kafka_connect ~20 MB/s), no change-stream collapse. WORKLOAD RATE: MongoDB's oplog is a capped collection — unlike slot/binlog CDC, a consumer that lags gets its change stream invalidated when the oplog wraps, collapsing throughput to zero. The scenario therefore uses a sustainable rate (20K/sec) rather than the 150K/sec saturation the RDS scenarios use. Multi-vCPU sweeps will need per-point rate tuning (see scenario notes). KNOWN CAVEAT (follow-up): the broker-derived metric under-reads the Connect engine (~2 MB/s) vs its self-reported ~26 MB/s, while KC's agree (~20/20). Cause: the seeder reuses one payload per worker (highly compressible), and Connect's redpanda output compresses on the wire while Debezium's producer does not — so the broker sees compressed bytes for Connect only. The broker-derived head-to-head column is therefore not yet apples-to-apples for this connector. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finalizes the mongodb_cdc vs Debezium MongoDB sweep (builds on 156a110). Config changes: - seeder: cycle a 4096-entry pool of distinct random payloads (was one reused payload per worker). Reused payloads are trivially compressible, so Connect's redpanda output compressed on the wire while Debezium's producer did not, skewing the broker-byte metric ~13x. The pool exceeds a compression batch (~850 docs) so batches are incompressible for both engines. - oplog: 100 -> 500 GB. At saturation the CDC consumer lags the ~230 MB/s ingest; the oplog must hold a full window of backlog at the 1-vCPU worst case without wrapping (wrap invalidates the change stream -> throughput collapses). - scenario: saturating rate (200K/sec) + [1,2,4,8] sweep. NO per-point rate overrides: mongodb_cdc is single-cursor-bound (see below), so every point hits the same ceiling and a saturating rate + big oplog reveals it cleanly. 8/8 head-to-head (1-vCPU smoke + this sweep, broker-derived MB/s, apples-to-apples): vCPU: 1 2 4 8 connect broker: 23.2 24.4 26.5 26.7 (self-report flat 33 from 2 vCPU) kafka_connect: 19.4 16.6 26.5 17.2 (noisy, no trend) Connect rises to ~27 and holds steady; Debezium is slower and erratic (its buffer-lock warnings show as throughput jitter). Connect wins on both speed and stability. KEY FINDING — mongodb_cdc is SINGLE-CURSOR-bound, not vCPU-bound: it streams via one database-level db.Watch (internal/impl/mongodb/cdc/input.go), so throughput plateaus from 2 vCPU (extra consumer cores can't parallelize one ordered cursor) and NO connector config raises it (multiple collections just add a $match filter to the same cursor). The only lever is a sharded cluster, where mongos merges per-shard cursors that tail in parallel. This differs fundamentally from the slot/binlog sweeps (postgres/mysql), which are consumer-CPU-bound and do scale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Curated 8/8 sweep results for the mongodb_cdc bench: broker-vs-broker head-to-head table (Connect ~27 MB/s steady vs Debezium ~17-26 noisy), the single-cursor-ceiling finding (flat past 2 vCPU; only a sharded cluster scales it), and methodology/caveats. Replaces the auto-appended per-run noise with a clean canonical summary; raw per-run JSON stays under benchmarking/aws/results/ (gitignored). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record mongodb / orders-cdc head-to-head result (best 2 vCPU, Connect 33 MB/s vs KC 17 MB/s, +16 MB/s / +50%) and bump the last-refreshed date to 2026-07-17. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.