docs: publish global charts and refresh benchmark navigation

README.md

@@ -1,144 +1,104 @@
 # CostBench
 
-**An open benchmark for real-time analytics system cost-performance across the complete analytics path.**
+**An open benchmark for real-time analytics cost-performance across the complete analytics path.**
 
-CostBench measures how much performance each dollar actually buys you — not only when a query runs, but across the full system path that makes real-time analytics possible: fresh data in, query-ready data maintained, fast answers out, and the cost of keeping that path running.
+CostBench measures the work and cost required to make continuously arriving data query-ready and
+serve analytical queries over it. The current full-path quotes study includes accepted runs for
+ClickHouse Cloud, Snowflake, Google BigQuery, and Amazon Redshift Serverless.
 
 > [!NOTE]
-> **Why static query benchmarks are incomplete**
->
-> Most database benchmarks load a static dataset once, let the system fully prepare it, then run each query multiple times and report the fastest hot-cache result. That measures repeated reads over stable, already-prepared data. It does not show the time and cost of making fresh data query-ready, or how queries behave when they run continuously over data that keeps changing, while ingest, maintenance, and refresh work keep running in parallel.
-
-That is why CostBench focuses on **full-path cost-performance** to answer the question that matters:
-
-> **Where do you get the most real-time analytics performance per dollar across the full path?**
-
-## What CostBench measures
-
-A real-time analytics benchmark needs to measure the system end to end:
+> A static query benchmark starts after data has been loaded and prepared. CostBench also measures
+> continuous ingest, maintenance of query-ready structures, freshness, and query serving while that
+> work remains active.
 
 ![The five stages of full-path cost-performance](docs/images/full-path-cost-performance-stages.gif)
 
-1. Fresh data arrives continuously.
-2. The system writes that data and makes it query-ready.
-3. Raw data is kept in a layout that supports fast drill-down queries.
-4. Pre-aggregated data is maintained for low-latency dashboard queries.
-5. Queries are served continuously while ingest, maintenance, and refresh work keep running.
-
-CostBench captures the cost and latency of those stages together, because different systems spend work in different places. One system may spend more on ingest. Another may spend more on background clustering, refresh compute, or query execution. A read-only query benchmark cannot show that full tradeoff.
-
-CostBench is therefore **not a bulk-load or backfill benchmark**. It simulates a real-time analytics system where fresh data is continuously generated at the source and must become query-ready as it arrives.
-
-## Methodology: full-path real-time cost-performance
-
-The CostBench methodology measures the full real-time analytics path:
-
-- continuous ingest at a fixed rate,
-- raw-data organization for efficient drill-down queries,
-- continuously maintained pre-aggregations,
-- freshness of derived data,
-- continuous query serving while data keeps arriving,
-- and the cost of keeping all of that running.
-
-This is the methodology described in the first end-to-end real-time analytics blog: [The end-to-end cost-performance of real-time analytics: Snowflake vs. ClickHouse Cloud](https://clickhouse.com/blog/real-time-analytics-cost-performance-snowflake-vs-clickhouse)
-
-
-## Legacy methodology: read-side and write-side components
-
-Earlier CostBench work measured important parts of the analytics path separately. Those results are still useful, but they are no longer the main methodology.
-
-### Read-side cost-performance
-
-The original read-side benchmark measured query cost-performance over already-loaded, already-prepared datasets. It compared how much query performance each dollar bought across major cloud data warehouses.
-
-That methodology is useful for understanding query-engine efficiency, but it does not measure the cost of making fresh data query-ready, maintaining derived data, or serving queries while ingest and maintenance continue in parallel.
-
-Related blog:
-
-- **[How the 5 major cloud data warehouses compare on cost-performance](https://clickhouse.com/blog/cloud-data-warehouses-cost-performance-comparison)** — full read-side results at 1B / 10B / 100B rows, including the interactive explorer.
-
-Repository path:
-
-- **[Legacy read-side benchmark →](query-side-only/)**
-
-### Query-ready raw data
-
-The next step measured one write-side component of the real-time path: what it costs to keep continuously ingested raw data query-ready.
-
-That benchmark focused on newly written raw data and physical organization. It did not include continuously maintained pre-aggregations or continuous query serving while ingest and maintenance continued. The full-path methodology expands on that by measuring ingest, raw-data organization, pre-aggregation freshness, refresh cost, and query execution together.
-
-Related blog:
-
-- **[Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse](https://clickhouse.com/blog/write-side-cost-performance-snowflake-clickhouse)** — measuring what it costs to keep continuously ingested data query-ready.
-
-## Systems covered
-
-CostBench has benchmark coverage across major cloud data warehouses, depending on the methodology and workload:
+## Start here
 
-- ClickHouse Cloud
-- Snowflake
-- Databricks SQL Serverless
-- Google BigQuery
-- Amazon Redshift Serverless
+| Area | Purpose |
+|---|---|
+| [Full-path benchmarks](full-path-realtime/) | Current end-to-end real-time methodology and workloads |
+| [Quotes benchmark](full-path-realtime/quotes/) | Accepted multi-provider study, evidence map, and reproduction order |
+| [Global visualizations](full-path-realtime/quotes/global/visualizations/) | Provider-neutral chart manifest and reproducible renderers |
+| [Legacy query benchmark](query-side-only/) | Read-side comparison over already-prepared data |
 
-The full-path real-time methodology is being expanded over time across more systems, ingest paths, datasets, and concurrency levels.
+## What the full-path benchmark measures
 
-## How CostBench compares cost-performance
+The benchmark keeps the analytics path live from source to answer:
 
-Cloud data warehouses expose cost through different units: credits, DBUs, slot-seconds, compute units, RPUs, serverless service credits, warehouse runtime, and storage. CostBench normalizes those vendor-specific billing models into a common question:
+1. Events arrive continuously at a fixed target rate.
+2. The provider writes those events into its raw-data path.
+3. The raw layout remains usable for drill-down queries.
+4. A derived aggregate is maintained for dashboard queries.
+5. Dashboard and drill-down workloads run while ingest and maintenance continue.
 
-> How much work did the system need to complete the workload, and what did that work cost?
+The published evidence covers:
 
-For full-path benchmarks, CostBench combines:
+- ingest progress and successful row counts;
+- raw and aggregate query latency during active ingestion;
+- persisted materialized-view freshness;
+- complete fresh-data-path and matched query cost;
+- provider configuration and pricing assumptions;
+- source JSONL, reconciled windows, generated CSV, SVG, PNG, and provenance summaries.
 
-- fresh-data path cost,
-- query cost,
-- query runtime,
-- freshness behavior,
-- and the system configuration required to keep the workload running.
+This is not a bulk-load benchmark. Systems are evaluated as continuously operating real-time
+analytics paths, including provider-specific components such as background refresh compute,
+serverless ingestion services, or a required broker layer.
 
-This makes it possible to compare systems even when they move work to different parts of the architecture, such as ingest compute, background clustering, materialized-view refresh, scheduled refresh warehouses, or query-serving compute.
+## Current accepted quotes evidence
 
-## Open and reproducible
+| System | Accepted evidence | Comparison role |
+|---|---|---|
+| ClickHouse Cloud | [`results_t2/`](full-path-realtime/quotes/clickhouse-cloud/results_t2/) | Pairwise reference and full-path baseline |
+| Snowflake | [`results/t2/`](full-path-realtime/quotes/snowflake/results/t2/) | Accepted Run14 with normalized mixed-rate query attribution |
+| BigQuery | [`bq-full-t2-20260810_152224/`](full-path-realtime/quotes/bigquery/results/bq-full-t2-20260810_152224/) | Accepted T2 with Capacity and On-demand alternatives |
+| Redshift Serverless | [`results/t2/`](full-path-realtime/quotes/redshift-serverless/results/t2/) | Accepted T2 with SUPER and typed read alternatives |
 
-CostBench is open so benchmark claims can be inspected, reproduced, and improved.
+The global score is:
 
-The repository publishes:
+```text
+(complete fresh-data-path cost + matched query cost) × accumulated query runtime
+```
 
-- workload definitions,
-- schema and table definitions,
-- ingest scripts,
-- query workloads,
-- system configurations,
-- pricing assumptions,
-- raw result files,
-- cost calculations,
-- and methodology notes for each benchmark.
+Lower is better. Each non-ClickHouse score is normalized within its own accepted pairwise
+row-progress window. The global chart combines those accepted pairwise ratios; it does not claim a
+single cross-provider iteration join. See the [quotes methodology](full-path-realtime/quotes/) and
+the generated provenance JSON beside every chart for the exact contract.
 
-If a result looks surprising, you can inspect the setup that produced it. If a configuration can be improved, it can be reviewed and corrected in the open.
+## Reproducibility and review
 
-Issues and pull requests are welcome.
+CostBench publishes the scripts and evidence needed to inspect benchmark claims:
 
-## Read more
+- workload, schema, and query definitions;
+- ingest and fixed-rate runner implementations;
+- provider configuration and pricing files;
+- raw runner results and row-progress reconciliation reports;
+- cost calculations and accepted summaries;
+- fail-closed visualization manifests and slide-ready outputs.
 
-### Current full-path methodology
+Generated summaries store repository-relative source paths and SHA-256 hashes. Credential files are
+local-only: the repository ignores every `*_credentials.txt` path, and CI rejects credential
+artifacts or high-confidence secret material if either is staged accidentally.
 
-- **[Introducing CostBench: an open benchmark for data warehouse cost-performance](https://clickhouse.com/blog/costbench-data-warehouse-cost-performance)** — what CostBench is and why cost-performance matters in the agentic era.
-- **[The end-to-end cost-performance of real-time analytics: Snowflake vs. ClickHouse Cloud](https://clickhouse.com/blog/real-time-analytics-cost-performance-snowflake-vs-clickhouse)** — full-path real-time analytics cost-performance: continuous ingest, query-ready data, pre-aggregation freshness, and continuous query serving.
+## Methodology history
 
-### Legacy component benchmarks
+The full-path methodology builds on two earlier CostBench studies:
 
-- **[How the 5 major cloud data warehouses compare on cost-performance](https://clickhouse.com/blog/cloud-data-warehouses-cost-performance-comparison)** — read-side cost-performance at 1B / 10B / 100B rows, including the interactive explorer.
-- **[Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse](https://clickhouse.com/blog/write-side-cost-performance-snowflake-clickhouse)** — the cost of keeping continuously ingested raw data query-ready.
+- [How the 5 major cloud data warehouses compare on cost-performance](https://clickhouse.com/blog/cloud-data-warehouses-cost-performance-comparison) — read-side cost-performance at 1B, 10B, and 100B rows.
+- [Agentic analytics starts with query-ready data](https://clickhouse.com/blog/write-side-cost-performance-snowflake-clickhouse) — the write-side cost of keeping raw data query-ready.
 
-### Billing model background
+Current methodology and background:
 
-- **[How the 5 major cloud data warehouses really bill you: a unified, engineer-friendly guide](https://clickhouse.com/blog/how-cloud-data-warehouses-bill-you)** — credits, DBUs, compute units, slot-seconds, and RPUs explained on equal footing.
+- [Introducing CostBench](https://clickhouse.com/blog/costbench-data-warehouse-cost-performance)
+- [The end-to-end cost-performance of real-time analytics](https://clickhouse.com/blog/real-time-analytics-cost-performance-snowflake-vs-clickhouse)
+- [How the 5 major cloud data warehouses really bill you](https://clickhouse.com/blog/how-cloud-data-warehouses-bill-you)
 
 ## Contributing
 
-CostBench is open because cost-performance claims should be reviewable. If you spot a setup that can be improved, a pricing detail that should be updated, or a vendor configuration worth adding, please contact us.
+Cost-performance claims should be reviewable. Pull requests that improve a configuration, pricing
+assumption, cost boundary, reconciliation rule, or disclosure are welcome. Keep secrets outside the
+repository and run `python3 scripts/check_repository_hygiene.py` before committing.
 
 ## License
 
-See [LICENSE](LICENSE) in this repository.
+See [LICENSE](LICENSE).

full-path-realtime/README.md

@@ -1,17 +1,51 @@
+# Full-path real-time benchmarks
 
-# Full-path real-time cost-performance benchmarks
+This directory contains CostBench workloads that measure the complete operating path of a real-time
+analytics system: continuous ingest, query-ready raw data, maintained aggregates, freshness, query
+latency under active ingestion, and the cost of keeping the path live.
 
-This directory contains benchmark workloads for measuring end-to-end real-time analytics cost and performance.
+## Workloads
 
-Unlike query-only benchmarks, these workloads cover the full path from data ingestion to query-ready analytical results, including:
+| Workload | Status | Scope |
+|---|---|---|
+| [Quotes](quotes/) | Current accepted multi-provider study | ClickHouse Cloud, Snowflake, BigQuery, and Redshift Serverless at roughly 1M events/s and 100B+ rows |
+| [Hits](hits/) | Workload implementation | Web analytics data; not part of the current accepted global quotes synthesis |
 
-- ingesting source data
-- maintaining derived or serving structures
-- refreshing or updating query-ready data
-- running dashboard and drilldown queries
-- collecting cost and performance results across systems
+## Common benchmark contract
 
-## Workloads
+- Ingest is rate-controlled and progress is recorded continuously.
+- Raw and aggregate query suites run on fixed schedules while ingest remains active.
+- Query observations are reconciled by base-table row progress, not by assuming iteration numbers
+  align across providers.
+- Fresh-data-path cost covers the provider-specific components required to ingest and maintain the
+  query-ready state for the complete run.
+- Query cost is reported for the accepted active-ingestion comparison window.
+- Persisted materialized-view lag is kept distinct from query-time freshness correction.
+- Source data, cost inputs, filters, smoothing, exclusions, and chart geometry are disclosed in
+  machine-readable summaries.
+
+The shared matcher is documented in [`utils/README.md`](utils/README.md). It writes matched JSONL,
+iteration lists, and a report containing source hashes and selection details. Provider-specific
+READMEs document any additional accepted rule.
+
+## Evidence flow
+
+```text
+provider runner JSONL
+        ↓
+row-progress reconciliation
+        ↓
+provider-native cost summaries
+        ↓
+pairwise charts and provenance
+        ↓
+fail-closed global manifest and charts
+```
+
+For the complete commands and accepted evidence roots, continue with the
+[quotes benchmark README](quotes/README.md).
+
+## Repository safety
 
-- `quotes/` — stock quotes benchmark using bid/ask market data.
-- `hits/` — web analytics data.
+Never commit credentials or provider exports containing secrets. Every `*_credentials.txt` path is
+ignored, and `scripts/check_repository_hygiene.py` enforces the same rule for tracked content.