|
| 1 | +# TOAST support — pg_toast chunks stored off the WAL window |
| 2 | + |
| 3 | +Externally-toasted column values are reconstructable in every path, including |
| 4 | +values toasted *before* the replication window. Chunks land in a pluggable |
| 5 | +store of record (`ToastResolver` / `ChunkStore`, `src/toast.rs`), selected by |
| 6 | +`[toast] mode`, so reassembly no longer depends on a value's chunks coinciding |
| 7 | +with the referring tuple in WAL. In-xact WAL reassembly is the fast path — see |
| 8 | +[xact.md](xact.md). |
| 9 | + |
| 10 | +## Shipped |
| 11 | + |
| 12 | +- **Stores.** `disabled` (default; NULL/default-fill on miss, counted |
| 13 | + `toast_values_filled_default`, never an error), `disk` (`DiskChunkStore`, |
| 14 | + append-only file per value, miss is a hard error), `clickhouse` |
| 15 | + (`ClickHouseChunkStore`, chunks as rows in a `pg_toast_<relid>` |
| 16 | + `ReplacingMergeTree(_lsn)` table, minimal `chunk_id`/`chunk_seq`/`chunk_data`/ |
| 17 | + `_lsn` form, `ORDER BY (chunk_id, chunk_seq)`). All `src/toast.rs`. |
| 18 | +- **WAL path.** Same-xact values reassemble inline from the buffered chunk map |
| 19 | + (`reassemble`, `src/xact_buffer.rs`), unchanged fast path; chunks also `put` |
| 20 | + to the store for future re-emit. A `MissingToastChunk` miss (pre-window |
| 21 | + re-emit) falls back to `fetch_into` + `try_reassemble`. |
| 22 | +- **Bootstrap.** Page walk decodes `pg_toast_*` tuples into chunks instead of |
| 23 | + counting-and-dropping; the drain defers any main-table tuple carrying a mapped |
| 24 | + `ExternalToast` (`Deferred`), `put`s all chunks durable, then resolves via |
| 25 | + `resolve_or_fill_toast` (`src/pipeline/bootstrap.rs`). One miss→fetch codepath |
| 26 | + covers bootstrap and pre-window alike (option (b), not the two-pass (a)). |
| 27 | +- **Decode shape (R2).** Value reassembled before the main-table INSERT, stored |
| 28 | + inline `Bytea`/`Text`; `encode_value` (`src/ch_emitter.rs`) unchanged. Tier 3 |
| 29 | + detoast routing: `detoasted_value` runs reassembled bytes back through |
| 30 | + `varlena_to_value` (`src/heap_decoder.rs`), so a detoasted jsonb/array/numeric |
| 31 | + resolves like an inline one (`PgPending` → oracle). |
| 32 | +- **Compression.** `chunk_data` holds PG's compressed bytes; the reassembler |
| 33 | + decompresses at ingest from the pointer it already holds, via the shared |
| 34 | + `decompress_varlena` (`src/heap_decoder.rs`, pglz/lz4). |
| 35 | +- **Convergence.** Toast tables are `ReplacingMergeTree(_lsn)`; chunk rows are |
| 36 | + immutable per `va_valueid`, so re-shipped chunks are byte-identical and `_lsn` |
| 37 | + dedup is purely a dedup, never a value change |
| 38 | + ([[project_walshadow_eventual_consistency]]). |
| 39 | + |
| 40 | +## Deferred |
| 41 | + |
| 42 | +- **R1 query-time-JOIN mode.** Per-table opt-in: store the `ToastPointer` in the |
| 43 | + main column and reassemble via a CH JOIN on `chunk_id = va_valueid` instead of |
| 44 | + inline at ingest. Wins dedup + defers reassembly cost off ingest, costs a |
| 45 | + CH-side concat + PGLZ path (materialized view / UDF / client-side) and a |
| 46 | + pointer column carrying `va_extinfo` + `va_rawsize`. Behind demand; R2 inline |
| 47 | + stays the default. |
| 48 | +- **Chunk GC / vacuum reclaim.** PG drops superseded chunks when a value is |
| 49 | + deleted or updated to a new `va_valueid`. The shipped CH schema has no `_op` |
| 50 | + column and the toast relation's replica identity is `nothing` (delete WAL |
| 51 | + carries no key — same blind spot as system catalogs, |
| 52 | + [[feedback_pg_version_wal_skew]]), so a delete marker has nowhere to land. |
| 53 | + Dead chunk rows leak; dedup keeps the live `va_valueid`'s chunks correct. |
| 54 | +- **Bounded-memory streaming reassembly.** A multi-MB value is thousands of |
| 55 | + chunks. `fetch` streams the SELECT block-by-block (no unbounded buffered |
| 56 | + result read), but the reassembled value is still fully materialised in memory |
| 57 | + (the `BTreeMap` supplement, then `try_reassemble`'s concat) — R2-inherent, |
| 58 | + same as inline `reassemble`. Streaming reassembly of huge values unaddressed. |
| 59 | +- **Torn-fetch distinction.** `fetch` is one SELECT, its result taken as final, |
| 60 | + no retry. The planned in-flight-vs-truncated distinction (compare `va_rawsize` |
| 61 | + to summed chunk length, retry the in-flight case) is not implemented. Benign |
| 62 | + while a completed `put` makes a value's chunks atomically visible (single-node |
| 63 | + CH, synchronous INSERT ack, chunks immutable per `va_valueid`); reopen if a |
| 64 | + partial or racing `put` can surface a torn set. |
| 65 | + |
| 66 | +## Rejected alternatives |
| 67 | + |
| 68 | +- **Inline reassembly only.** Correct for same-xact WAL, wrong for bootstrap |
| 69 | + (errors at the emitter) and pre-window values (`MissingToastChunk`). The |
| 70 | + pre-`[toast]`-store status quo. |
| 71 | +- **NULL / raw-marker fallback as the resolution.** Lossy: the WAL re-emit of |
| 72 | + the referring tuple does not carry the chunks (PG reuses the old |
| 73 | + `va_valueid`), so the value never resolves. Kept only as the explicit, |
| 74 | + surfaced `disabled`-mode fill, never silent loss. |
| 75 | +- **pg_toast in the shadow PG catalog.** Would promote the catalog shadow to a |
| 76 | + full data replica, reintroducing the cross-seg missing-page PANIC class the |
| 77 | + NOOP rewrite exists to avoid ([[reference_walshadow_cross_seg_records]]) and |
| 78 | + coupling every detoast to a replay-LSN wait + the catalog mutex. The disk/CH |
| 79 | + stores are append-only, walshadow-owned, lifecycle-independent. |
0 commit comments