docs: architecture overview -- data flow, wire protocol, BSON type system

[richardsimmonds]

Summary

Adds docs/architecture.md -- a comprehensive architecture overview explaining how DocumentDB's three components interact, how a MongoDB request travels from the driver to PostgreSQL storage and back, how the wire protocol is translated, and how BSON types map to PostgreSQL.

Also adds two links to the README's Helpful Links section.

What is included

docs/architecture.md (~500 lines)

• High-Level Overview: ASCII stack diagram
• Components: per-component breakdown of all four packages (pg_documentdb_core, pg_documentdb, pg_documentdb_gw, pg_documentdb_gw_host)
• Data Flow: step-by-step annotated walkthrough of a find and an insertOne, naming the exact source files and SQL functions at each stage
• Wire Protocol Translation: message header, OpCode table, OP_MSG structure, command dispatch, SCRAM-SHA-256 auth flow
• BSON to PostgreSQL Type System: pgbson varlena layout, BSON type code table, Decimal128/Intel libbid details, index types, storage schema DDL
• Deployment Topologies: Docker background worker, external binary, managed service

README.md

Added links to docs/architecture.md and docs/build-from-source.md in the Helpful Links section.

Testing

Docs-only change. No code changes, no test impact.

[richardsimmonds]

Docs Review — NEEDS CHANGES

architecture.md is a solid foundation — the component breakdown, data flow diagrams, and BSON type table are accurate and genuinely useful for a newcomer. A few issues need fixing before merge, one of which is a factual error. configuration-reference.md is also accurate, but it's not mentioned in the PR description at all.

---

Blocking issues

1. Storage schema table name is wrong (architecture.md, Storage schema section)

The DDL example shows:

CREATE TABLE documentdb_data.<db>_<collection> (...)

Source code says otherwise. Tables are named documents_<collectionId> (a numeric uint64), not <dbname>_<collectionname>. From pg_documentdb/src/commands/insert.c:

appendStringInfo(&query, "documents_" UINT64_FORMAT, collectionId);

And from create_indexes.c:

CREATE INDEX documents_rum_index_<id> ON documentdb_data.documents_<collectionId>

A newcomer who runs \dt documentdb_data.* in psql will see documents_12345, not mydb_orders. Fix the DDL to show documents_<collection_id> (numeric). You may also want to explain that collection metadata (the db/collection name ↔ ID mapping) lives in documentdb_api_catalog.collections.

2. QueryCatalog is not loaded from SetupConfiguration.json (architecture.md, Command dispatch section)

The doc states:

"The SQL templates are loaded from SetupConfiguration.json at startup into a QueryCatalog struct, keeping the gateway code free of hardcoded SQL strings."

This is incorrect on two counts. QueryCatalog is a plain Rust struct populated by Default::default() with hardcoded string literals compiled into the binary (see documentdb_gateway_core/src/postgres/query_catalog.rs, the impl Default for QueryCatalog block). SetupConfiguration.json carries connection configuration (ports, TLS config, database name) — not SQL query strings. The "free of hardcoded SQL strings" characterization is also incorrect: the SQL strings are hardcoded in the Rust source. Please remove or correct this paragraph.

3. Auth section omits MONGODB-OIDC (architecture.md, Authentication section)

The auth description says "Client sends saslStart with mechanism SCRAM-SHA-256." The gateway supports two saslStart mechanisms: SCRAM-SHA-256 and MONGODB-OIDC. From documentdb_gateway_core/src/auth.rs:

if mechanism != "SCRAM-SHA-256" && mechanism != "MONGODB-OIDC" {
    return Err(...);
}

Either extend the auth flow description or add a note that OIDC is also supported.

4. OpCode table is incomplete (architecture.md, Wire Protocol section)

The table shows 4 opcodes (OP_MSG, OP_QUERY, OP_INSERT, OP_COMPRESSED). protocol/opcode.rs also defines and parses: OP_REPLY (1), OP_UPDATE (2001), OP_DELETE (2006), OP_GET_MORE (2005), OP_KILL_CURSORS (2007). They're #[deprecated] in the Rust source but still parsed from the wire. Add them to the table or add a note that additional deprecated opcodes are handled for backward compatibility.

5. PR description doesn't mention configuration-reference.md

The PR description describes docs/architecture.md and README changes only. This PR also adds docs/configuration-reference.md (325 lines, 91+ GUC entries, separate commit b7752abb). Update the PR description to include this file. Or if it was intentionally a separate piece of work, consider splitting it out to keep history clean.

6. README links to docs/build-from-source.md which doesn't exist

The README now contains:

See [docs/build-from-source.md](docs/build-from-source.md)

That file doesn't exist in this branch or in main. Either create it, remove this link, or link to documentdb-local/README.md which covers local setup and build.

---

Non-blocking issues

7. "Three PostgreSQL extensions" doesn't match the four-component diagram

The intro says "layers three PostgreSQL extensions" but documents four components. The gateway components (pg_documentdb_gw, pg_documentdb_gw_host) aren't PostgreSQL extensions in the CREATE EXTENSION sense — only pg_documentdb_core and pg_documentdb are. pg_documentdb_gw_host is a pgrx background worker (technically a shared library loaded by PostgreSQL), distinct from the other two. Clarify the distinction.

8. PR description says ~500 lines, file is 317 lines

Minor, but the PR description claims "~500 lines" for architecture.md. Actual line count is 317.

---

What's good

• Component breakdown is accurate and matches source layout (protocol/, requests/, processor/, postgres/, auth/, context/, responses/ all verified)
• pgbson struct definition matches pg_documentdb_core/include/io/pgbson.h exactly
• InstallBsonMemVTablesLocal described correctly — confirmed in pg_documentdb_core/include/bson_init.h
• BSON type table type codes are correct
• find_cursor_first_page SQL call verified against query_catalog.rs
• Deployment topologies (background worker via SPI vs standalone binary) accurately describe the two configurations
• pgrx reference for pg_documentdb_gw_host is accurate (confirmed bgworker.rs uses BackgroundWorker::connect_worker_to_spi)
• All commits have DCO Signed-off-by lines ✓
• configuration-reference.md GUC names and defaults spot-checked against feature_flag_configs.c, system_configs.c, background_job_configs.c — accurate

Fix items 1–6 and this will be in good shape.

[richardsimmonds]

Factual error — table name is wrong.

The DDL shows documentdb_data.<db>_<collection> but the actual table name is documentdb_data.documents_<collectionId> where <collectionId> is a numeric uint64, not a human-readable db/collection name. Confirmed in pg_documentdb/src/commands/insert.c:

appendStringInfo(&query, "documents_" UINT64_FORMAT, collectionId);

A newcomer running \dt documentdb_data.* in psql will see documents_12345, not mydb_orders. Fix the DDL and note that the db/collection name ↔ ID mapping lives in documentdb_api_catalog.collections.

[richardsimmonds]

Factual error — QueryCatalog is not loaded from SetupConfiguration.json.

'The SQL templates are loaded from SetupConfiguration.json at startup into a QueryCatalog struct, keeping the gateway code free of hardcoded SQL strings.'

This is wrong. QueryCatalog is a Rust struct populated by Default::default() with SQL string literals compiled into the binary — not read from any file. See documentdb_gateway_core/src/postgres/query_catalog.rs (the impl Default for QueryCatalog block). SetupConfiguration.json carries connection config (ports, TLS, DB name), not query strings. Please remove or correct this paragraph.

[richardsimmonds]

Missing auth mechanism: MONGODB-OIDC.

The gateway supports two saslStart mechanisms. From auth.rs:

if mechanism != "SCRAM-SHA-256" && mechanism != "MONGODB-OIDC" {
    return Err(...);
}

Add a note that OIDC is also supported (handled by handle_oidc() in auth.rs), or extend the flow description to cover both paths.

[richardsimmonds]

Incomplete opcode table.

The table shows 4 opcodes but protocol/opcode.rs also defines and parses: OP_REPLY (1), OP_UPDATE (2001), OP_DELETE (2006), OP_GET_MORE (2005), OP_KILL_CURSORS (2007). They're marked #[deprecated] but still handled on the wire. Add them to the table or note: 'Additional deprecated opcodes (OP_REPLY, OP_UPDATE, OP_DELETE, OP_GET_MORE, OP_KILL_CURSORS) are parsed for backward compatibility.'

[richardsimmonds]

Minor — 'three PostgreSQL extensions' doesn't match the four-component diagram.

The intro says 'layers three PostgreSQL extensions' but documents four components. pg_documentdb_core and pg_documentdb are CREATE EXTENSION extensions. pg_documentdb_gw is a standalone Rust binary (not a PG extension), and pg_documentdb_gw_host is a pgrx shared library that PG loads as a background worker. Clarify the distinction here so readers aren't confused when they look at the actual CREATE EXTENSION calls in the setup SQL.

[richardsimmonds]

Broken link — docs/build-from-source.md doesn't exist.

This line links to docs/build-from-source.md but that file doesn't exist in this branch or main. Either create it in this PR, remove this link, or replace it with documentdb-local/README.md which covers local setup.