Group commit, gRPC roles, /metrics, beastbench and security policy

AGENTS.md

@@ -0,0 +1,26 @@
+# BeastDB: instructions for coding agents
+
+Read by Codex and other agents that use `AGENTS.md`. Claude Code reads it through `CLAUDE.md`. Antigravity also reads `GEMINI.md`, which carries the same rules; keep the two in step.
+
+## File modularity (guideline, not a hard limit)
+
+- One responsibility per file. About **400 lines** is the point to stop and ask "does this file do more than one thing?" If yes, split along that seam (for example sync vs. lifecycle, load vs. run). If no, leave it alone.
+- Never split a cohesive file, or merge small files, just to satisfy a line count.
+- Exempt: generated code (`*.pb.go`), test files, JSON and raw data.
+- Do not add CI or lint checks that fail on line count. A non-exempt file past about 500 lines should say why in the change description.
+
+## Engine principles
+
+- **WAL before page:** a data page must never reach the data file before the WAL records that describe it are durable. The buffer pool enforces this with the WAL barrier (`WAL.EnsureDurable`); keep every dirty-page write on that path.
+- **Zero-allocation hot paths:** query, hashing, parsing and buffer-pinning paths avoid heap allocations; prefer byte slices and `sync.Pool`.
+- **Mechanical sympathy:** align structs to avoid false sharing, prefer flat memory layouts, respect 4 KB page alignment.
+
+## Scope and honesty
+
+- BeastDB is a general-purpose, domain-agnostic database. Do not hard-code schema or domain rules from downstream applications; client-specific partition metadata belongs in `partitions.json` or `-partition-config`.
+- Do not claim performance relative to other databases without an equivalent-workload benchmark (see `docs/benchmarking.md`). Describe planned features as planned.
+- Report vulnerabilities privately; see `SECURITY.md`.
+
+## Before you finish
+
+Run `go build ./... && go vet ./... && go test ./... -race`. `TestEngineSystemWorkloadBenchmark` has a fixed 1000 ops/s floor and can fail when the machine is loaded; re-run it alone before treating a failure as a regression.

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

CONTRIBUTING.md

@@ -6,9 +6,11 @@ Thank you for your interest in contributing to BeastDB! We welcome contributions
 
 ## 🏛️ Engineering Standards
 
-### 1. Strict File Modularity (< 200 LoC)
-- **Constraint:** Every source file (excluding auto-generated `.pb.go` code) must strictly remain readable and **under 200 Lines of Code (LoC)**.
-- **Decomposition:** Large subsystems (B+ Tree, Buffer Pool, Hash Table, WAL) must be decomposed into focused, single-responsibility files and companion helpers.
+### 1. File Modularity (guideline, not a hard limit)
+- **Principle:** One responsibility per file. Treat about **400 lines** as a prompt to ask "does this file do more than one thing?" If it does, split along that seam. If it doesn't, leave it as is. Never split, or merge, files only to hit a number.
+- **Exempt:** generated code (`*.pb.go`), test files, and data files (JSON and similar).
+- **Review, not CI:** there is no automated line-count gate. A non-exempt file past about 500 lines should say in the pull request why it stays one file.
+- **Decomposition:** Large subsystems (B+ Tree, Buffer Pool, Hash Table, WAL) should be organized into focused, single-responsibility files and companion helpers.
 
 ### 2. Zero-Allocation Hot Paths
 - Query execution, hash calculation, parsing, and buffer pinning paths must not trigger unnecessary heap allocations.
@@ -21,6 +23,12 @@ Thank you for your interest in contributing to BeastDB! We welcome contributions
 
 ---
 
+## 🔒 Security Issues
+
+Do not report vulnerabilities in public issues or pull requests. Follow [SECURITY.md](SECURITY.md).
+
+---
+
 ## 🧪 Verification & Testing
 
 Before submitting a pull request, ensure all validations pass:
@@ -32,9 +40,9 @@ go test -v -count=1 ./...
 # 2. Run benchmarks to verify zero allocations on hot paths
 go test -bench=. -benchmem ./...
 
-# 3. Enforce the < 200 LoC modularity limit
-Get-ChildItem -Recurse -Filter "*.go" | Where-Object { $_.FullName -notmatch "\.pb\.go" } | ForEach-Object {
+# 3. (Optional) List large non-test, non-generated files to review. Informational only.
+Get-ChildItem -Recurse -Filter "*.go" | Where-Object { $_.Name -notmatch "\.pb\.go$|_test\.go$" } | ForEach-Object {
     $lines = (Get-Content $_.FullName).Count
-    if ($lines -ge 200) { Write-Error "$lines lines in $($_.Name)" }
+    if ($lines -ge 400) { Write-Host "$lines lines in $($_.FullName): does it do more than one thing?" }
 }
 ```

GEMINI.md

@@ -4,9 +4,12 @@ These rules govern all interactions, coding, and architecture in this project.
 
 ---
 
-## 1. File Modularity & Readability (< 200 LoC)
-* **Strict File Length Limit:** In any project, if a file is not JSON or raw data, it must strictly remain readable and **< 200 Lines of Code (LoC)**.
-* **Component Decomposition:** Decompose complex systems (B+ trees, buffer pools, hash tables, page managers) into small, single-responsibility files, structs, and reusable components.
+## 1. File Modularity & Readability (guideline, not a hard limit)
+* **One responsibility per file.** Organize files by what they do. File length is a smell, not a rule: roughly **400 lines** is the point to stop and ask "does this file do more than one thing?" If yes, split it along that seam (for example sync vs. lifecycle, load vs. run). If no, leave it alone.
+* **Never split just to hit a number.** Do not cut a cohesive file into arbitrary pieces, move code without a clear responsibility boundary, or merge small files back together to satisfy a count.
+* **Exemptions:** generated code (`*.pb.go`), test files, and JSON/raw data have no size expectation.
+* **No automated gate.** Do not add CI or lint checks that fail on line count. Past about 500 lines, mention the reason in the change description or review.
+* **Component Decomposition:** Decompose complex systems (B+ trees, buffer pools, hash tables, page managers) into focused structs and reusable components.
 
 ---
 

README.md

@@ -159,6 +159,9 @@ BeastDB is optimized for **read-heavy OLTP** — deterministic point reads and o
 
 - [**V1 Semantics & Guarantees**](docs/v1-semantics.md) — Durability acknowledgment, limits, and replica reads.
 - [**System Benchmarks**](docs/benchmarks.md) — Reproducible throughput, tail latencies, and cache-eviction profiles.
+- [**Benchmarking Guide**](docs/benchmarking.md) — `beastbench` YCSB-style workloads and how to compare fairly with other databases.
+- [**Security Policy**](SECURITY.md) — How to report a vulnerability privately, scope, and safe deployment checklist.
+- [**Upgrade Spec**](docs/upgrade-spec.md) — Security, usability and performance work packages and what is still planned.
 - [**Operator Guide**](docs/runbooks/operator-guide.md) — TLS configuration, cluster roles, manual failover, and `/healthz` probes.
 - [**Disaster Recovery**](docs/runbooks/disaster-recovery.md) — Physical backup drills, checksum verification, and RTO/RPO SLA.
 - [**Client Integration**](docs/clients.md) — Protobuf contracts for Go, Python, TypeScript, and Rust.

ROADMAP.md

@@ -122,7 +122,7 @@ Order is dependency-based, not a calendar promise. Initial planning range for on
 
 ## Beyond V1 gates
 
-See the [roadmap plan](docs/roadmap-plan.md) for post-gate phases. Shipped so far: conditional writes (`ABSENT`/`PRESENT`/`VALUE_EQUALS` on Put, Delete and BatchWrite), limit-based scan pagination, and values up to 2 MiB with 8 MiB batches ([ADR 0001](docs/adr/0001-large-values.md)).
+See the [roadmap plan](docs/roadmap-plan.md) for post-gate phases and the [upgrade spec](docs/upgrade-spec.md) for the current security, usability and performance work. Shipped so far: conditional writes (`ABSENT`/`PRESENT`/`VALUE_EQUALS` on Put, Delete and BatchWrite), limit-based scan pagination, values up to 2 MiB with 8 MiB batches ([ADR 0001](docs/adr/0001-large-values.md)), WAL group commit with a write-ahead buffer-pool barrier, per-method gRPC role authorization with token expiry, a Prometheus `/metrics` endpoint, and the `beastbench` YCSB-style load generator ([benchmarking guide](docs/benchmarking.md)).
 
 ## References
 

SECURITY.md

@@ -0,0 +1,62 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+**Please do not open a public issue, pull request or discussion for a security problem.**
+
+Report it privately through GitHub:
+
+1. Go to the repository's **Security** tab.
+2. Choose **Report a vulnerability** (GitHub private vulnerability reporting).
+
+Include what you can of:
+
+- the affected version or commit and how you run BeastDB (flags, TLS, roles, replication);
+- steps to reproduce, or a proof of concept;
+- the impact you observed (for example unauthorised read or write, data corruption, denial of service);
+- any suggested fix.
+
+If the **Report a vulnerability** button is missing, the maintainers have not enabled it yet. Open a public issue that says only "security contact needed" and contains no technical details, and a maintainer will arrange a private channel.
+
+### What to expect
+
+BeastDB is maintained by a small team, so these are goals, not guarantees:
+
+- acknowledgement of your report within about 5 working days;
+- an assessment and a plan within about 14 days;
+- a fix or documented mitigation as quickly as the severity allows, with coordinated public disclosure after a fix is available. We will credit you if you want to be credited.
+
+## Supported versions
+
+BeastDB has not reached a tagged V1 release (see [ROADMAP.md](ROADMAP.md)). Until then, security fixes are made on the `main` branch only. Once V1 is released, this section will list the supported release lines.
+
+## Scope
+
+In scope:
+
+- authentication and authorization (gRPC bearer tokens and roles, Studio sessions and cookies, the `/metrics` endpoint);
+- transport security (TLS and mTLS for clients and replicas);
+- the storage engine, WAL, recovery, backup and restore paths, where a flaw could lose or corrupt acknowledged data or expose data to the wrong caller;
+- the replication stream;
+- the container image and default configuration shipped in this repository.
+
+Out of scope:
+
+- findings that require the attacker to already have admin credentials, local root access, or write access to the data directory;
+- running with `--dev` or `--insecure-auth` on a reachable network (these flags are documented as for local, non-production use);
+- denial of service that needs a very large volume of traffic;
+- missing hardening that is already listed as planned in [docs/upgrade-spec.md](docs/upgrade-spec.md) (for example encryption at rest). Reports on how a planned item could be exploited today are still welcome.
+
+## Deploying safely
+
+The [Operator Guide](docs/runbooks/operator-guide.md) is the reference. In short:
+
+- set a strong `BEASTDB_ADMIN_PASSWORD`; the server refuses the default password on non-loopback listeners and when `BEASTDB_ENV=production`;
+- enable TLS (and mTLS for replicas) and keep gRPC and Studio off public interfaces;
+- give each application its own API token with the smallest role it needs (`viewer`, `writer`, `admin`, `replication`) and set an expiry where you can;
+- protect `/metrics` with a dedicated `BEASTDB_METRICS_TOKEN` and scrape it over TLS;
+- keep backups off the host and test restores.
+
+## Safe harbour
+
+We will not pursue action against researchers who make a good-faith effort to follow this policy: test only against systems you own or have permission to test, avoid accessing or modifying other people's data, do not degrade service for others, and give us reasonable time to fix an issue before disclosing it.

cmd/beastbench/beastbench_test.go

@@ -0,0 +1,185 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"math/rand"
+	"sort"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+func TestHistogramQuantiles(t *testing.T) {
+	h := NewHistogram()
+	var vals []int
+	for i := 1; i <= 100000; i++ {
+		d := time.Duration(i) * time.Microsecond
+		h.Record(d)
+		vals = append(vals, int(d))
+	}
+	sort.Ints(vals)
+	for _, q := range []float64{.5, .95, .99} {
+		want := float64(vals[int(q*float64(len(vals)))-1])
+		got := float64(h.Quantile(q))
+		if got < want || got > want*1.02 {
+			t.Errorf("q=%v got %v want ~%v", q, got, want)
+		}
+	}
+	if h.Max() != 100000*time.Microsecond || h.Min() != time.Microsecond {
+		t.Errorf("min/max wrong: %v %v", h.Min(), h.Max())
+	}
+	if h.Count() != 100000 {
+		t.Errorf("count %d", h.Count())
+	}
+}
+
+func TestHistogramBucketsRoundTrip(t *testing.T) {
+	for _, v := range []uint64{0, 1, 127, 128, 255, 256, 257, 1000, 1 << 20, 1<<40 + 12345, 1<<63 + 5} {
+		lo, hi := bucketBounds(bucketIndex(v))
+		if v < lo || v > hi {
+			t.Errorf("v=%d not in [%d,%d]", v, lo, hi)
+		}
+	}
+}
+
+func TestHistogramMergeAndEmpty(t *testing.T) {
+	a, b := NewHistogram(), NewHistogram()
+	if a.Quantile(.5) != 0 || a.Mean() != 0 || a.Min() != 0 {
+		t.Error("empty histogram should report zeros")
+	}
+	a.Record(time.Millisecond)
+	b.Record(3 * time.Millisecond)
+	a.Merge(b)
+	if a.Count() != 2 || a.Max() != 3*time.Millisecond || a.Min() != time.Millisecond {
+		t.Errorf("merge wrong: %d %v %v", a.Count(), a.Min(), a.Max())
+	}
+}
+
+func TestZipfianSkewAndRange(t *testing.T) {
+	const n = 1000
+	z := NewZipfian(n, DefaultZipfTheta)
+	r := rand.New(rand.NewSource(1))
+	counts := make([]int, n)
+	for i := 0; i < 200000; i++ {
+		v := z.Next(r)
+		if v >= n {
+			t.Fatalf("out of range: %d", v)
+		}
+		counts[v]++
+	}
+	if counts[0] < counts[10] || counts[10] < counts[500] {
+		t.Errorf("not monotonically skewed: %d %d %d", counts[0], counts[10], counts[500])
+	}
+	if counts[0] < 200000/20 {
+		t.Errorf("item 0 too cold: %d", counts[0])
+	}
+}
+
+func TestUniformAndDeterminism(t *testing.T) {
+	u := Uniform{N: 10}
+	r := rand.New(rand.NewSource(7))
+	seen := map[uint64]bool{}
+	for i := 0; i < 1000; i++ {
+		seen[u.Next(r)] = true
+	}
+	if len(seen) != 10 {
+		t.Errorf("uniform missed buckets: %d", len(seen))
+	}
+	z := NewZipfian(100, .9)
+	a, b := rand.New(rand.NewSource(3)), rand.New(rand.NewSource(3))
+	for i := 0; i < 100; i++ {
+		if z.Next(a) != z.Next(b) {
+			t.Fatal("same seed must give same sequence")
+		}
+	}
+}
+
+func TestLatestFavoursNewest(t *testing.T) {
+	f := &atomic.Uint64{}
+	f.Store(1000)
+	l := Latest{Zipf: NewZipfian(1000, DefaultZipfTheta), Frontier: f}
+	r := rand.New(rand.NewSource(2))
+	newest := 0
+	for i := 0; i < 10000; i++ {
+		v := l.Next(r)
+		if v >= 1000 {
+			t.Fatalf("out of range %d", v)
+		}
+		if v >= 990 {
+			newest++
+		}
+	}
+	if newest < 2000 {
+		t.Errorf("latest not skewed to newest: %d", newest)
+	}
+}
+
+func TestWorkloadMixes(t *testing.T) {
+	for name, w := range workloads {
+		var sum float64
+		for _, p := range w.Mix {
+			sum += p
+		}
+		if sum < .999 || sum > 1.001 {
+			t.Errorf("workload %s mix sums to %v", name, sum)
+		}
+	}
+	w := workloads["A"]
+	if w.pick(0.1) != OpRead || w.pick(0.9) != OpUpdate {
+		t.Error("A pick wrong")
+	}
+	if _, err := lookupWorkload("Z"); err == nil {
+		t.Error("expected error for unknown workload")
+	}
+}
+
+func TestSmokeAllWorkloads(t *testing.T) {
+	for _, wl := range []string{"A", "B", "C", "D", "E", "F"} {
+		wl := wl
+		t.Run(wl, func(t *testing.T) {
+			var out, errb bytes.Buffer
+			err := realMain([]string{"--self-contained", "--json", "--workload", wl, "--records", "200",
+				"--ops", "300", "--warmup", "20", "--threads", "4", "--value-size", "64", "--seed", "5", "--max-scan-len", "10"}, &out, &errb)
+			if err != nil {
+				t.Fatalf("%v (%s)", err, errb.String())
+			}
+			var rep Report
+			if err := json.Unmarshal(out.Bytes(), &rep); err != nil {
+				t.Fatal(err)
+			}
+			if len(rep.Phases) != 2 || rep.Phases[1].Ops != 300 {
+				t.Fatalf("unexpected phases: %+v", rep.Phases)
+			}
+			for _, p := range rep.Phases {
+				for _, o := range p.PerOp {
+					if o.Errors != 0 {
+						t.Errorf("%s/%s: %d errors", p.Phase, o.Op, o.Errors)
+					}
+					if o.P50Us > o.P99Us || o.P99Us > o.MaxUs {
+						t.Errorf("%s/%s percentiles not monotonic: %+v", p.Phase, o.Op, o)
+					}
+				}
+			}
+			if rep.Phases[0].Ops != 200 {
+				t.Errorf("load ops %d", rep.Phases[0].Ops)
+			}
+		})
+	}
+}
+
+func TestTextOutputAndBadFlags(t *testing.T) {
+	var out, errb bytes.Buffer
+	if err := realMain([]string{"--self-contained", "--records", "50", "--ops", "50", "--threads", "2"}, &out, &errb); err != nil {
+		t.Fatal(err)
+	}
+	if !bytes.Contains(out.Bytes(), []byte("not a competitive benchmark")) || !bytes.Contains(out.Bytes(), []byte("p99(us)")) {
+		t.Errorf("text output missing pieces:\n%s", out.String())
+	}
+	if err := realMain([]string{"--workload", "Q"}, &out, &errb); err == nil {
+		t.Error("expected error")
+	}
+	if err := realMain([]string{"--value-size", "1"}, &out, &errb); err == nil {
+		t.Error("expected error")
+	}
+}