Add tests for types, composoite pkey, tables names, ddl

.github/workflows/ci.yml

@@ -71,7 +71,8 @@ jobs:
           sudo apt-get install -y --no-install-recommends \
             postgresql-${{ matrix.pg-major }} \
             postgresql-client-${{ matrix.pg-major }} \
-            postgresql-server-dev-${{ matrix.pg-major }}
+            postgresql-server-dev-${{ matrix.pg-major }} \
+            postgresql-${{ matrix.pg-major }}-pgvector
           sudo systemctl stop postgresql 2>/dev/null || true
           PG_BIN=/usr/lib/postgresql/${{ matrix.pg-major }}/bin
           "$PG_BIN/postgres" -V
@@ -125,10 +126,26 @@ jobs:
           fixtures/wal/xlog_switch/capture.sh
           fixtures/wal/vacuum_full_pg_depend/capture.sh
 
+      # Install the walshadow extension before the e2e suite so each shadow PG
+      # can CREATE EXTENSION walshadow for the oracle decode path.
+      - name: Build + install walshadow extension (e2e oracle path)
+        working-directory: walshadow/pgext
+        run: |
+          PG_CONFIG=/usr/lib/postgresql/${{ matrix.pg-major }}/bin/pg_config
+          make PG_CONFIG=$PG_CONFIG
+          sudo make PG_CONFIG=$PG_CONFIG install
+
       - name: Build (workspace + tests)
         working-directory: walshadow
         run: cargo build --workspace --all-targets --locked
 
+      # Each e2e test stages a source + shadow PG cluster on disk; the
+      # runner's root fs is small, so point TMPDIR at the larger /mnt volume.
+      - name: Set TMPDIR to /mnt
+        run: |
+          sudo mkdir -p /mnt/tmp && sudo chmod 1777 /mnt/tmp
+          echo "TMPDIR=/mnt/tmp" >> "$GITHUB_ENV"
+
       # All e2e tests run: shadow_lifecycle, shadow_catalog, catalog_seed,
       # wal_stream_e2e (need initdb), classify_fixture + filter_round_trip
       # + wal_stream_chunk_boundary (use the freshly regenerated fixtures),
@@ -241,7 +258,8 @@ jobs:
             | sudo tee /etc/apt/sources.list.d/pgdg.list
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends \
-            postgresql-17 postgresql-client-17
+            postgresql-17 postgresql-client-17 \
+            postgresql-server-dev-17 postgresql-17-pgvector
           sudo systemctl stop postgresql 2>/dev/null || true
           echo "/usr/lib/postgresql/17/bin" >> "$GITHUB_PATH"
 
@@ -281,6 +299,22 @@ jobs:
           fixtures/wal/xlog_switch/capture.sh
           fixtures/wal/vacuum_full_pg_depend/capture.sh
 
+      # Install the walshadow extension before the instrumented suite so the
+      # oracle-path tests run instead of skipping.
+      - name: Build + install walshadow extension (e2e oracle path)
+        working-directory: walshadow/pgext
+        run: |
+          PG_CONFIG=/usr/lib/postgresql/17/bin/pg_config
+          make PG_CONFIG=$PG_CONFIG
+          sudo make PG_CONFIG=$PG_CONFIG install
+
+      # Each e2e test stages a source + shadow PG cluster on disk; the
+      # runner's root fs is small, so point TMPDIR at the larger /mnt volume.
+      - name: Set TMPDIR to /mnt
+        run: |
+          sudo mkdir -p /mnt/tmp && sudo chmod 1777 /mnt/tmp
+          echo "TMPDIR=/mnt/tmp" >> "$GITHUB_ENV"
+
       # Single `--no-report` test pass so cargo only instruments &
       # executes the suite once; subsequent `report` invocations re-read
       # the cached profdata to emit lcov + html + summary.

plans/INDEX.md

@@ -21,6 +21,9 @@ components. Future-work proposals live under [future/](future/INDEX.md)
   read-time defaults
 - [xact.md](xact.md) — `XactBuffer`, `SubxactTracker`, TOAST
   reassembly, local-disk spill, `DrainEntry` ordering
+- [TOAST.md](TOAST.md) — full TOAST chunk store (`disabled`/`disk`/
+  `clickhouse`), bootstrap tap + defer-resolve, pre-window miss→fetch;
+  deferred R1 JOIN mode, chunk GC, streaming reassembly
 - [emitter.md](emitter.md) — parallel decode+insert pipeline
   (reorder → decode ×M → batcher → inserter ×N → ack watermark),
   `type_bridge`, synthetic columns, `DdlApplicator`, barrier fence

plans/TOAST.md

@@ -0,0 +1,79 @@
+# TOAST support — pg_toast chunks stored off the WAL window
+
+Externally-toasted column values are reconstructable in every path, including
+values toasted *before* the replication window. Chunks land in a pluggable
+store of record (`ToastResolver` / `ChunkStore`, `src/toast.rs`), selected by
+`[toast] mode`, so reassembly no longer depends on a value's chunks coinciding
+with the referring tuple in WAL. In-xact WAL reassembly is the fast path — see
+[xact.md](xact.md).
+
+## Shipped
+
+- **Stores.** `disabled` (default; NULL/default-fill on miss, counted
+  `toast_values_filled_default`, never an error), `disk` (`DiskChunkStore`,
+  append-only file per value, miss is a hard error), `clickhouse`
+  (`ClickHouseChunkStore`, chunks as rows in a `pg_toast_<relid>`
+  `ReplacingMergeTree(_lsn)` table, minimal `chunk_id`/`chunk_seq`/`chunk_data`/
+  `_lsn` form, `ORDER BY (chunk_id, chunk_seq)`). All `src/toast.rs`.
+- **WAL path.** Same-xact values reassemble inline from the buffered chunk map
+  (`reassemble`, `src/xact_buffer.rs`), unchanged fast path; chunks also `put`
+  to the store for future re-emit. A `MissingToastChunk` miss (pre-window
+  re-emit) falls back to `fetch_into` + `try_reassemble`.
+- **Bootstrap.** Page walk decodes `pg_toast_*` tuples into chunks instead of
+  counting-and-dropping; the drain defers any main-table tuple carrying a mapped
+  `ExternalToast` (`Deferred`), `put`s all chunks durable, then resolves via
+  `resolve_or_fill_toast` (`src/pipeline/bootstrap.rs`). One miss→fetch codepath
+  covers bootstrap and pre-window alike (option (b), not the two-pass (a)).
+- **Decode shape (R2).** Value reassembled before the main-table INSERT, stored
+  inline `Bytea`/`Text`; `encode_value` (`src/ch_emitter.rs`) unchanged. Tier 3
+  detoast routing: `detoasted_value` runs reassembled bytes back through
+  `varlena_to_value` (`src/heap_decoder.rs`), so a detoasted jsonb/array/numeric
+  resolves like an inline one (`PgPending` → oracle).
+- **Compression.** `chunk_data` holds PG's compressed bytes; the reassembler
+  decompresses at ingest from the pointer it already holds, via the shared
+  `decompress_varlena` (`src/heap_decoder.rs`, pglz/lz4).
+- **Convergence.** Toast tables are `ReplacingMergeTree(_lsn)`; chunk rows are
+  immutable per `va_valueid`, so re-shipped chunks are byte-identical and `_lsn`
+  dedup is purely a dedup, never a value change
+  ([[project_walshadow_eventual_consistency]]).
+
+## Deferred
+
+- **R1 query-time-JOIN mode.** Per-table opt-in: store the `ToastPointer` in the
+  main column and reassemble via a CH JOIN on `chunk_id = va_valueid` instead of
+  inline at ingest. Wins dedup + defers reassembly cost off ingest, costs a
+  CH-side concat + PGLZ path (materialized view / UDF / client-side) and a
+  pointer column carrying `va_extinfo` + `va_rawsize`. Behind demand; R2 inline
+  stays the default.
+- **Chunk GC / vacuum reclaim.** PG drops superseded chunks when a value is
+  deleted or updated to a new `va_valueid`. The shipped CH schema has no `_op`
+  column and the toast relation's replica identity is `nothing` (delete WAL
+  carries no key — same blind spot as system catalogs,
+  [[feedback_pg_version_wal_skew]]), so a delete marker has nowhere to land.
+  Dead chunk rows leak; dedup keeps the live `va_valueid`'s chunks correct.
+- **Bounded-memory streaming reassembly.** A multi-MB value is thousands of
+  chunks. `fetch` streams the SELECT block-by-block (no unbounded buffered
+  result read), but the reassembled value is still fully materialised in memory
+  (the `BTreeMap` supplement, then `try_reassemble`'s concat) — R2-inherent,
+  same as inline `reassemble`. Streaming reassembly of huge values unaddressed.
+- **Torn-fetch distinction.** `fetch` is one SELECT, its result taken as final,
+  no retry. The planned in-flight-vs-truncated distinction (compare `va_rawsize`
+  to summed chunk length, retry the in-flight case) is not implemented. Benign
+  while a completed `put` makes a value's chunks atomically visible (single-node
+  CH, synchronous INSERT ack, chunks immutable per `va_valueid`); reopen if a
+  partial or racing `put` can surface a torn set.
+
+## Rejected alternatives
+
+- **Inline reassembly only.** Correct for same-xact WAL, wrong for bootstrap
+  (errors at the emitter) and pre-window values (`MissingToastChunk`). The
+  pre-`[toast]`-store status quo.
+- **NULL / raw-marker fallback as the resolution.** Lossy: the WAL re-emit of
+  the referring tuple does not carry the chunks (PG reuses the old
+  `va_valueid`), so the value never resolves. Kept only as the explicit,
+  surfaced `disabled`-mode fill, never silent loss.
+- **pg_toast in the shadow PG catalog.** Would promote the catalog shadow to a
+  full data replica, reintroducing the cross-seg missing-page PANIC class the
+  NOOP rewrite exists to avoid ([[reference_walshadow_cross_seg_records]]) and
+  coupling every detoast to a replay-LSN wait + the catalog mutex. The disk/CH
+  stores are append-only, walshadow-owned, lifecycle-independent.

plans/bootstrap.md

@@ -279,12 +279,15 @@ V1 limits:
   captured mid-write walk as-shipped. WAL in `[start_lsn, end_lsn]`
   updating same tuples re-emits at higher `_lsn` &
   `ReplacingMergeTree(_lsn)` collapses duplicate
-- **TOAST-spilled columns fail fast.** Inline varlena decodes through
-  heap decoder; external pointers surface as
-  `ColumnValue::ExternalToast` and the CH drain rejects the row with
-  relation + column named (no reassembly path exists here).
-  `pg_toast_<relid>` tar entries are observed but not decoded; full
-  chunk-storage design in [future/TOAST.md](future/TOAST.md)
+- **TOAST-spilled columns resolve when a chunk store is configured.**
+  Inline varlena decodes through the heap decoder; external pointers
+  surface as `ColumnValue::ExternalToast`. With `[toast] mode != disabled`
+  the page walk decodes `pg_toast_<relid>` pages into chunks, `put`s them
+  to the store, defers the referring tuples, and reassembles after the
+  walk (`resolve_or_fill_toast`, `src/pipeline/bootstrap.rs`). With the
+  default `mode = disabled` an unresolved value NULL/default-fills and is
+  counted, no longer a hard reject. Full chunk-storage design in
+  [TOAST.md](TOAST.md)
 - **2C CH-side COPY load NOT shipped.** See
   [What is NOT 2C](#what-is-not-2c-ch-side-copy-load) below
 
@@ -347,8 +350,9 @@ shipped because it is only shape with bounded memory at scale
   `BatcherMsg::Row` into the shared tail; one ack seq per rfn flip.
   Returns `BootstrapDrainOutcome { next_seq, rows_routed }`; caller
   runs `tail.finish(msg_tx, ack, next_seq, fatal)` to seal + wait
-  durable. Fails fast on `ColumnValue::ExternalToast` (page walk does
-  no TOAST reassembly — [future/TOAST.md](future/TOAST.md))
+  durable. `ColumnValue::ExternalToast` is resolved from the configured
+  chunk store (deferred past the walk, then `resolve_or_fill_toast`), or
+  NULL/default-filled under `[toast] mode = disabled` — [TOAST.md](TOAST.md)
 - `drain_backfill` — metrics-only path (no `--ch-config`). Hands
   synthetic `CommittedTuple`s to a `TupleObserver`; `on_xact_end`
   fires on every rfn flip & once after channel close

plans/future/INDEX.md

@@ -15,6 +15,5 @@ here as they surface; promote into `plans/` when they land
 * [FUZZ.md](FUZZ.md) — continuous coverage-guided fuzzing (cargo-fuzz/libFuzzer) across wal-rus + walshadow + clickhouse-c-rs: tiered targets, round-trip/differential oracles, C-boundary ASan, unattended-VM supervisor
 * [pipeline_backpressure_and_scaling.md](pipeline_backpressure_and_scaling.md) — parallel decode+insert pipeline design record (core landed, current-state doc is [emitter.md](../emitter.md)); remaining work: WAL-pump backpressure via wire/record split, decode/insert scaling (bootstrap Option B, hot-table sharding, N/M sizing)
 * [dependencies.md](dependencies.md) — crates.io replacement candidates for generic object storage, MPMC, retry, throttling, and metrics code
-* [TOAST.md](TOAST.md) — full TOAST support: `pg_toast_<relid>` chunk tables on CH, bootstrap tap, pre-window value fallback (in-xact WAL reassembly already ships — see [xact.md](../xact.md))
 * [risks.md](risks.md) — measurement-deferred risks and open questions
 * [parked.md](parked.md) — small operational polish + cross-major fixtures + skipped-test drive