feat(postgres): add pgsodium, gzip, and pgzstd extensions (#60) (#61)

# Squash Merge Notes: pgsodium, gzip, and pgzstd Extensions

**PR:** [#61](#61)
**Issue:** [#60](#60)
**Branch:** `paudley/pgsodium-gzip-pgzstd-extensions`
**Base:** `main` @ `93258e4`
**Squash commit:** `feat(postgres): add pgsodium, gzip, and pgzstd extensions (#60)`

---

## Overview

Adds three new PostgreSQL extensions to the core_data platform, all built from source since none are available as PGDG apt packages for PostgreSQL 17. Also introduces a startup enforcement mechanism for `shared_preload_libraries` to prevent configuration drift.

**Total diff:** 6 files changed, +130 insertions, -3 deletions

---

## Extensions Added

### pgsodium (v3.1.9)

- **Source:** [github.com/michelp/pgsodium](https://github.com/michelp/pgsodium)
- **Purpose:** Modern cryptographic functions wrapping libsodium — encryption, decryption, hashing, key derivation, digital signatures, and Transparent Column Encryption (TCE) with server-side key management.
- **Build dependency:** `libsodium-dev`
- **Runtime requirement:** Must be in `shared_preload_libraries` for key management and TCE features.
- **Key management:** Uses `pgsodium.getkey_script` pointing to `/opt/core_data/tools/pgsodium_getkey.sh`, which reads the root encryption key from `/opt/core_data/secrets/pgsodium.key` (configurable via `PGSODIUM_KEY_FILE` env var). The key must be exactly 32 bytes (256-bit) of raw key material.
- **Complements:** Existing `pgcrypto` extension. pgsodium provides libsodium's modern algorithms (XChaCha20-Poly1305, Argon2id, Ed25519) versus pgcrypto's OpenSSL-based primitives.

### pgsql-gzip (extension name: `gzip`)

- **Source:** [github.com/pramsey/pgsql-gzip](https://github.com/pramsey/pgsql-gzip)
- **Purpose:** SQL-native gzip compression and decompression of `bytea` data. Provides `gzip()` and `gunzip()` functions for compressing large text/binary payloads and interoperating with gzip-compressed data from external systems.
- **Build dependency:** `zlib1g-dev`
- **Runtime requirement:** None (no `shared_preload_libraries` entry needed).

### pgzstd

- **Source:** [github.com/grahamedgecombe/pgzstd](https://github.com/grahamedgecombe/pgzstd)
- **Purpose:** Zstandard (zstd) compression and decompression in SQL. Provides `zstd_compress()` and `zstd_decompress()` functions. Zstandard achieves better compression ratios than gzip at comparable speeds and significantly faster decompression — ideal for high-throughput data pipelines.
- **Build dependency:** `libzstd-dev`
- **Runtime requirement:** None (no `shared_preload_libraries` entry needed).

---

## File-by-File Changes

### `postgres/Dockerfile`

**Build dependencies added to apt-get install (alphabetically ordered):**
- `libsodium-dev` — required by pgsodium for libsodium C library headers
- `libzstd-dev` — required by pgzstd for Zstandard C library headers
- `zlib1g-dev` — required by pgsql-gzip for zlib C library headers (may already be present as transitive dependency, but made explicit)

**Three new `RUN` blocks added after the existing pg_squeeze build**, following the identical source-build pattern established by Apache AGE and pg_squeeze:

1. **pgsodium** — Clones tag `v3.1.9` with `--depth 1 --single-branch`, builds with `make`/`make install` using `PG_CONFIG`, cleans up temp directory. Version is parameterized via `ARG PGSODIUM_VERSION=3.1.9` for build-time configurability.

2. **pgsql-gzip** — Clones latest `--depth 1`, standard `make`/`make install`, cleanup. No version tag pinning (upstream does not tag releases consistently).

3. **pgzstd** — Clones latest `--depth 1`, standard `make`/`make install`, cleanup. No version tag pinning.

### `scripts/lib/extensions_list.sh`

**`CORE_EXTENSION_LIST` array — 3 entries added (lines 31-33):**
- `pgsodium` — placed after `pgcrypto` (logical grouping: crypto-related extensions)
- `gzip` — placed after `pgsodium`
- `pgzstd` — placed after `gzip` (compression extensions grouped together)

This array drives `02-enable-extensions.sh`, which iterates it to run `CREATE EXTENSION IF NOT EXISTS` in every non-template database plus `template1`. Adding entries here is sufficient for automatic creation in all databases at startup — no changes to `02-enable-extensions.sh` were needed.

**New `REQUIRED_PRELOAD_LIBRARIES` array added (lines 50-62):**
```
age, pgaudit, pg_stat_statements, pg_cron, pg_squeeze,
auto_explain, pg_buffercache, pg_partman_bgw, pgsodium
```
This is the canonical single source of truth for which libraries must be present in `shared_preload_libraries`. It is consumed by the new `enforce_shared_preload_libraries()` function in `00-render-config.sh`.

### `postgres/conf/postgresql.conf.tpl`

**Line 13 — `shared_preload_libraries` updated:**
- Before: `'age,pgaudit,pg_stat_statements,pg_cron,pg_squeeze,auto_explain,pg_buffercache,pg_partman_bgw'`
- After: `'age,pgaudit,pg_stat_statements,pg_cron,pg_squeeze,auto_explain,pg_buffercache,pg_partman_bgw,pgsodium'`

**Line 90 — new pgsodium configuration added:**
- `pgsodium.getkey_script = '/opt/core_data/tools/pgsodium_getkey.sh'` — tells pgsodium how to retrieve the root encryption key for TCE. Placed before `datestyle` at the end of the config file.

### `postgres/initdb/00-render-config.sh`

**New function `enforce_shared_preload_libraries()` (lines 18-49):**

Sources `extensions_list.sh` to get the `REQUIRED_PRELOAD_LIBRARIES` array, then:
1. Reads the current `shared_preload_libraries` value from `postgresql.conf`
2. Checks each required library against the current value using comma-delimited matching
3. If any are missing: appends them to the value, patches the config file in-place with `sed -i`, logs a WARNING with the missing entries, logs the corrected value, and returns exit code 1 (signaling restart needed)
4. If all present: returns 0 (no action needed)

**Early-exit path updated (lines 135-149):**

Previously, when the config sentinel existed and `FORCE_RENDER_CONFIG!=1`, the script only refreshed network allow entries and issued a `pg_ctl reload`. Now it also calls `enforce_shared_preload_libraries()`. If the function corrects the config (returns 1), a `pg_ctl -m fast -w restart` is issued instead of a reload — because `shared_preload_libraries` is a `postmaster`-context GUC that requires a full restart to take effect. A simple reload would silently leave the old libraries loaded.

**Post-render safety belt (lines 233-236):**

After a fresh template render (both first-time and `FORCE_RENDER_CONFIG=1`), `enforce_shared_preload_libraries()` runs as a safety belt to catch any drift between the template's hardcoded value and the canonical `REQUIRED_PRELOAD_LIBRARIES` array. This guards against the template and the list getting out of sync during development.

### `postgres/tools/pgsodium_getkey.sh` (NEW FILE)

Server key retrieval script for pgsodium's Transparent Column Encryption:
- Uses `set -euo pipefail` for strict error handling
- Reads key from `PGSODIUM_KEY_FILE` env var (default: `/opt/core_data/secrets/pgsodium.key`)
- If the key file is missing or not readable, prints an actionable error message with the exact command to generate a key, then exits with code 1
- On success, outputs the raw key to stdout (consumed by pgsodium internally)

### `scripts/lib/extensions.sh`

**`exercise_extensions()` — 5 new smoke queries added (lines 38-48):**

| Extension | Smoke Test | Validates |
|-----------|-----------|-----------|
| pgsodium | `pgsodium.crypto_pwhash_str('test_password')` | Argon2id password hashing works |
| gzip | `gzip('hello world')` + `gunzip(gzip(...))` | Compression produces output, roundtrip preserves data |
| pgzstd | `zstd_compress('hello world'::bytea)` + `zstd_decompress(...)` | Compression produces output, roundtrip preserves data |

Note: gzip roundtrip uses `convert_from(gunzip(...), 'UTF8')` because `gunzip()` returns `bytea`. pgzstd roundtrip compares `bytea` directly.

**`run_pgtap_smoke()` — plan count and assertions updated:**
- Plan count: `39` → `42` (3 new assertions)
- Added `has_extension('pgsodium', ...)`, `has_extension('gzip', ...)`, `has_extension('pgzstd', ...)` after the existing `pgcrypto` assertion (logical grouping)

---

## Architecture Decisions

### Why source builds (not apt packages)

None of these three extensions are available in the PGDG apt repository for PostgreSQL 17. All three follow the identical build pattern already established by Apache AGE and pg_squeeze: shallow git clone → `make` with `PG_CONFIG` → `make install` → cleanup.

### Why `REQUIRED_PRELOAD_LIBRARIES` is separate from `CORE_EXTENSION_LIST`

Not all extensions in `CORE_EXTENSION_LIST` require preloading (most don't — only 9 out of 49 do). A separate canonical list prevents accidentally adding non-preloadable extensions to `shared_preload_libraries`, which would cause postgres to fail to start.

### Why restart instead of reload for preload corrections

`shared_preload_libraries` is a `postmaster`-context GUC in PostgreSQL. It can only take effect when the server process starts. A `pg_ctl reload` (SIGHUP) updates `SIGHUP`-context parameters but silently ignores `postmaster`-context changes. Using restart ensures the correction actually takes effect.

### Why enforce on every startup

The config rendering has an early-exit path: once the sentinel file exists, the template is not re-rendered (unless `FORCE_RENDER_CONFIG=1`). This means manual edits to `postgresql.conf` that remove required preload libraries would persist across container restarts. The enforcement function closes this gap by validating config state independently of whether templates are rendered.

### Why pgsodium needs `shared_preload_libraries` but gzip/pgzstd do not

pgsodium's key management system (which powers TCE) requires a background worker process that must be started at server boot. The `pgsodium.getkey_script` is invoked during this early initialization phase to obtain the root key. gzip and pgzstd are pure SQL function libraries with no background workers or boot-time requirements.

---

## Operational Notes

### Generating a pgsodium key

Before using pgsodium's TCE features, generate a root key:
```bash
head -c 32 /dev/urandom | od -A n -t x1 | tr -d ' \n' > secrets/pgsodium.key
```

The key file must be readable by the postgres user inside the container. It is accessed via the shared secrets volume at `/opt/core_data/secrets/pgsodium.key`.

### Extension count

- **CORE_EXTENSION_LIST:** 46 → 49 extensions
- **REQUIRED_PRELOAD_LIBRARIES:** 9 entries (new array)
- **pgTap assertions:** 39 → 42

### Docker image size impact

Three additional source builds and three new shared library dependencies (`libsodium`, `zlib`, `libzstd`) will increase the Docker image size. The runtime libraries are small (libsodium ~500KB, zlib ~100KB, libzstd ~600KB). Build tools are already present in the image for AGE/pg_squeeze compilation.

Author: paudley

postgres/Dockerfile

@@ -45,10 +45,13 @@ RUN set -euxo pipefail; \
       gettext-base \
       bison \
       flex \
+      libsodium-dev \
       libssl-dev \
       libxml2-dev \
       libxslt1-dev \
       libreadline-dev \
+      libzstd-dev \
+      zlib1g-dev \
       "postgresql-server-dev-${PG_MAJOR}" \
       "postgresql-${PG_MAJOR}-pgaudit" \
       "postgresql-${PG_MAJOR}-pgvector" \
@@ -93,6 +96,31 @@ RUN set -eux; \
     make install PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
     rm -rf /tmp/pg_squeeze
 
+# Build pgsodium from source
+ARG PGSODIUM_VERSION=3.1.9
+RUN set -eux; \
+    git clone --depth 1 --branch "v${PGSODIUM_VERSION}" --single-branch https://github.com/michelp/pgsodium.git /tmp/pgsodium; \
+    cd /tmp/pgsodium; \
+    make PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
+    make install PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
+    rm -rf /tmp/pgsodium
+
+# Build pgsql-gzip from source
+RUN set -eux; \
+    git clone --depth 1 https://github.com/pramsey/pgsql-gzip.git /tmp/pgsql-gzip; \
+    cd /tmp/pgsql-gzip; \
+    make PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
+    make install PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
+    rm -rf /tmp/pgsql-gzip
+
+# Build pgzstd from source
+RUN set -eux; \
+    git clone --depth 1 https://github.com/grahamedgecombe/pgzstd.git /tmp/pgzstd; \
+    cd /tmp/pgzstd; \
+    make PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
+    make install PG_CONFIG="/usr/lib/postgresql/${PG_MAJOR}/bin/pg_config"; \
+    rm -rf /tmp/pgzstd
+
 # Copy configuration templates and initialization scripts
 COPY postgres/conf /opt/core_data/conf
 COPY postgres/initdb /docker-entrypoint-initdb.d

postgres/conf/postgresql.conf.tpl

@@ -10,7 +10,7 @@ shared_buffers = ${PG_SHARED_BUFFERS}
 work_mem = ${PG_WORK_MEM}
 maintenance_work_mem = ${PG_MAINTENANCE_WORK_MEM}
 effective_cache_size = ${PG_EFFECTIVE_CACHE_SIZE}
-shared_preload_libraries = 'age,pgaudit,pg_stat_statements,pg_cron,pg_squeeze,auto_explain,pg_buffercache,pg_partman_bgw'
+shared_preload_libraries = 'age,pgaudit,pg_stat_statements,pg_cron,pg_squeeze,auto_explain,pg_buffercache,pg_partman_bgw,pgsodium'
 wal_level = logical
 archive_mode = on
 archive_command = 'pgbackrest --config=/var/lib/postgresql/data/pgbackrest.conf --stanza=main archive-push %p'
@@ -87,5 +87,6 @@ ssl = ${POSTGRES_SSL_ENABLED}
 ssl_cert_file = '${POSTGRES_SSL_CERT_FILE}'
 ssl_key_file = '${POSTGRES_SSL_KEY_FILE}'
 ssl_prefer_server_ciphers = on
+pgsodium.getkey_script = '/opt/core_data/tools/pgsodium_getkey.sh'
 datestyle = 'iso, mdy'
 timezone = '${TZ}'

postgres/initdb/00-render-config.sh

@@ -15,6 +15,40 @@ NETWORK_ACCESS_DIR=${NETWORK_ACCESS_DIR:-/opt/core_data/network_access}
 NETWORK_ALLOW_FILE=${NETWORK_ALLOW_FILE:-${NETWORK_ACCESS_DIR}/allow.list}
 FORCE_RENDER_CONFIG=${FORCE_RENDER_CONFIG:-0}
 
+# shellcheck disable=SC1091
+# shellcheck source=/opt/core_data/scripts/lib/extensions_list.sh
+source /opt/core_data/scripts/lib/extensions_list.sh
+
+# Enforce that all required libraries are present in shared_preload_libraries.
+# Called on every startup to prevent config drift from manual edits.
+# Returns 0 if no changes needed, 1 if config was corrected (reload required).
+enforce_shared_preload_libraries() {
+	local conf_file="${PGDATA}/postgresql.conf"
+	[[ -f "${conf_file}" ]] || return 0
+
+	local current
+	current=$(grep -E "^shared_preload_libraries" "${conf_file}" | sed "s/shared_preload_libraries *= *'\\(.*\\)'/\\1/")
+
+	local missing=()
+	for lib in "${REQUIRED_PRELOAD_LIBRARIES[@]}"; do
+		if ! echo ",${current}," | grep -q ",${lib},"; then
+			missing+=("${lib}")
+		fi
+	done
+
+	if [[ ${#missing[@]} -gt 0 ]]; then
+		echo "[core_data] WARNING: shared_preload_libraries missing required entries: ${missing[*]}" >&2
+		local new_value="${current}"
+		for lib in "${missing[@]}"; do
+			new_value="${new_value},${lib}"
+		done
+		sed -i "s|^shared_preload_libraries *= *'.*'|shared_preload_libraries = '${new_value}'|" "${conf_file}"
+		echo "[core_data] Corrected shared_preload_libraries to: ${new_value}" >&2
+		return 1
+	fi
+	return 0
+}
+
 apply_network_allow_entries() {
 	local hba_path="${PGDATA}/pg_hba.conf"
 	if [[ ! -f "${hba_path}" ]]; then
@@ -101,8 +135,17 @@ if [[ -f "${SENTINEL}" ]]; then
 	if [[ "${FORCE_RENDER_CONFIG}" != "1" ]]; then
 		echo "[core_data] Configuration already rendered; refreshing network allow entries." >&2
 		apply_network_allow_entries
+		local needs_reload=0
+		if ! enforce_shared_preload_libraries; then
+			needs_reload=1
+		fi
 		if pg_ctl -D "${PGDATA}" status >/dev/null 2>&1; then
-			if ! pg_ctl -D "${PGDATA}" reload >/dev/null 2>&1; then
+			if [[ "${needs_reload}" -eq 1 ]]; then
+				echo "[core_data] Restarting PostgreSQL to apply corrected shared_preload_libraries." >&2
+				if ! pg_ctl -D "${PGDATA}" -m fast -w restart >/dev/null 2>&1; then
+					echo "[core_data] WARNING: pg_ctl restart failed after shared_preload_libraries correction." >&2
+				fi
+			elif ! pg_ctl -D "${PGDATA}" reload >/dev/null 2>&1; then
 				echo "[core_data] WARNING: pg_ctl reload failed while refreshing network allow entries." >&2
 			fi
 		fi
@@ -190,6 +233,10 @@ pg1-path=${PGDATA}
 pg1-port=5433
 CONF
 
+# Safety belt: enforce shared_preload_libraries even after fresh render
+# to catch any drift between the template and the canonical list.
+enforce_shared_preload_libraries || true
+
 echo "[core_data] Rendered PostgreSQL configs and pgBackRest configuration." >&2
 
 # Start postgres with our rendered config (which includes shared_preload_libraries).

postgres/tools/pgsodium_getkey.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: 2025 Blackcat Informatics® Inc.
+# SPDX-License-Identifier: MIT
+
+# pgsodium server key retrieval script.
+# Called by pgsodium to obtain the root encryption key for Transparent Column Encryption.
+# The key must be exactly 32 bytes (256 bits) of raw key material.
+
+set -euo pipefail
+
+PGSODIUM_KEY_FILE="${PGSODIUM_KEY_FILE:-/opt/core_data/secrets/pgsodium.key}"
+
+if [[ ! -r "${PGSODIUM_KEY_FILE}" ]]; then
+	echo "[core_data] ERROR: pgsodium key file not found or not readable: ${PGSODIUM_KEY_FILE}" >&2
+	echo "[core_data] Generate a key with: head -c 32 /dev/urandom | od -A n -t x1 | tr -d ' \\n' > ${PGSODIUM_KEY_FILE}" >&2
+	exit 1
+fi
+
+cat "${PGSODIUM_KEY_FILE}"

scripts/lib/extensions.sh

@@ -35,6 +35,17 @@ RESET search_path;
 -- pgcrypto smoke
 SELECT encode(digest('core_data', 'sha256'), 'hex');
 
+-- pgsodium smoke
+SELECT length(pgsodium.crypto_pwhash_str('test_password')) > 0 AS pgsodium_ok;
+
+-- gzip smoke
+SELECT length(gzip('hello world')) > 0 AS gzip_compress_ok;
+SELECT convert_from(gunzip(gzip('hello world')), 'UTF8') = 'hello world' AS gzip_roundtrip_ok;
+
+-- pgzstd smoke
+SELECT length(zstd_compress('hello world'::bytea)) > 0 AS zstd_compress_ok;
+SELECT zstd_decompress(zstd_compress('hello world'::bytea)) = 'hello world'::bytea AS zstd_roundtrip_ok;
+
 -- uuid-ossp smoke
 SELECT uuid_generate_v4();
 
@@ -122,7 +133,7 @@ run_pgtap_smoke() {
 		psql --set ON_ERROR_STOP=on --username "${POSTGRES_SUPERUSER:-postgres}" --dbname "${database}" <<'SQL'
 CREATE SCHEMA IF NOT EXISTS test_core_data;
 SET search_path = test_core_data, public;
-SELECT plan(39);
+SELECT plan(42);
 SELECT ok(current_schema = 'test_core_data', 'search_path set to test schema');
 SELECT has_extension('vector', 'vector extension installed');
 SELECT has_extension('postgis', 'postgis extension installed');
@@ -134,6 +145,9 @@ SELECT has_extension('pg_stat_statements', 'pg_stat_statements extension install
 SELECT ok(position('auto_explain' in current_setting('shared_preload_libraries')) > 0, 'auto_explain registered in shared_preload_libraries');
 SELECT has_extension('pg_buffercache', 'pg_buffercache extension installed');
 SELECT has_extension('pgcrypto', 'pgcrypto extension installed');
+SELECT has_extension('pgsodium', 'pgsodium extension installed');
+SELECT has_extension('gzip', 'gzip extension installed');
+SELECT has_extension('pgzstd', 'pgzstd extension installed');
 SELECT has_extension('citext', 'citext extension installed');
 SELECT has_extension('cube', 'cube extension installed');
 SELECT has_extension('hstore', 'hstore extension installed');

scripts/lib/extensions_list.sh

@@ -28,6 +28,9 @@ CORE_EXTENSION_LIST=(
 	pg_stat_statements
 	pg_trgm
 	pgcrypto
+	pgsodium
+	gzip
+	pgzstd
 	pgstattuple
 	pgtap
 	pgaudit
@@ -44,3 +47,18 @@ CORE_EXTENSION_LIST=(
 	uuid-ossp
 	vector
 )
+
+# Canonical list of libraries that must be in shared_preload_libraries.
+# Enforced on every container startup regardless of config state.
+# shellcheck disable=SC2034
+REQUIRED_PRELOAD_LIBRARIES=(
+	age
+	pgaudit
+	pg_stat_statements
+	pg_cron
+	pg_squeeze
+	auto_explain
+	pg_buffercache
+	pg_partman_bgw
+	pgsodium
+)