You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* chore: ignore .worktrees/ directory
* docs: add MongoDB adapter deep-dive and unified strategy docs
* docs(spec): add MongoDB adapter spec for adapters/mongodb
Single adapter targeting Atlas (GA) and self-hosted Community 8.2+ via
a unified $vectorSearch API. Documents public API, data layout, method
semantics, WHERE clause pre/post-filter split, index lifecycle, dev/CI
environment (mongodb-atlas-local Docker image), test plan, and
acceptance criteria.
Reviewed by spec-document-reviewer subagent (status: Approved); five
advisory recommendations folded in.
* docs(spec): require mongodb adapter in changeset fixed array
* docs(spec): add explicit root package.json scripts for mongodb (test:setup:mongodb, test:adapters:mongodb, build:adapters:mongodb), placed adjacent to existing PG/CF siblings
* docs(spec): move test:setup/test:teardown to adapters/mongodb/package.json (local to the adapter, not root)
* docs(plan): add MongoDB adapter implementation plan
20 bite-sized TDD tasks covering: package skeleton, escapeRegExp,
types, lazy MongoClient, convertWhereToMongo (pre/post-filter split,
and/or recursion, id→_id ObjectId casting, evaluatePostFilter),
ensureSearchIndex with definition-mismatch detection, storeChunk,
search via \$vectorSearch aggregation, deleteChunks/hasEmbeddingVersion,
docker-compose, compliance/where/integration suites, README,
root build+test wiring, changeset registration, CI job, end-to-end
verification.
* feat(mongodb): scaffold adapter package skeleton
* feat(mongodb): add escapeRegExp utility
* feat(mongodb): add public types and config helpers
* feat(mongodb): add lazy singleton MongoClient with test close helper
* feat(mongodb): convertWhereToMongo handles pre-filter leaf operators
* feat(mongodb): convertWhereToMongo routes like/contains/all to post-filter
* feat(mongodb): convertWhereToMongo handles and/or composition with pre/post split
* feat(mongodb): restore explanatory comments dropped from convertWhere
* feat(mongodb): add evaluatePostFilter for runtime post-filter matching
* feat(mongodb): ensureSearchIndex with definition-mismatch detection and READY polling
* docs(plan): simplify Task 10 embedding coercion (remove redundant ternary)
* feat(mongodb): add storeChunk that ensures index then inserts document
* fix(adapters): spread extensionFields before reserved fields in storeChunk
Both pg and mongodb adapters previously spread data.extensionFields after
sourceCollection/docId/chunkIndex/chunkText/embeddingVersion, allowing
user-supplied extension data to silently overwrite the canonical reserved
values. Move the spread to the front so reserved fields always win at
runtime — defense-in-depth on top of the existing config-time field-name
collision check in createEmbeddingsCollection.
* feat(mongodb): implement search via $vectorSearch with pre/post split
* fix(mongodb): throw on missing search score instead of silently producing NaN
A search result without a numeric score is meaningless — sort order is
load-bearing on it. Replace the silent Number() coercion with an
explicit guard that throws if doc.score is not a number, with an
actionable message pointing at the $project stage.
* feat(mongodb): map Payload id → Mongo _id with ObjectId casting
* feat(mongodb): add createMongoVectorIntegration factory and adapter wiring
* feat(mongodb): add docker-compose for local mongodb-atlas-local stack
* fix(mongodb): ensure collection exists before createSearchIndex; add compliance suite
Atlas Local rejects createSearchIndex against a non-existent collection
("Collection 'X' does not exist"). Materialize the collection idempotently
via db.createCollection in indexes.ts before the first search-index
creation. Atlas Cloud is more lenient, but the adapter must work in both.
Also lands the compliance suite (12 tests across getConfigExtension,
storeChunk, search, deleteChunks, hasEmbeddingVersion) that surfaced
this gap, plus shared dev/specs/{constants,utils}.ts helpers.
* test(mongodb): add WHERE-clause + integration suites against live Mongo
* docs(plan): record Task 16 fixes — limit/numCandidates, Atlas Local visibility lag, PENDING-state poll
Three plan-side bugs surfaced when the WHERE + integration suites ran
against the live Atlas Local container:
1. performVectorSearch helper default limit=100 violates Atlas's
limit <= numCandidates (pool is numCandidates: 50). Lowered to 10.
2. ~1s eventual-consistency lag between insertOne and $vectorSearch
visibility even after index READY — added 1200ms waits in WHERE
beforeAll and integration immediate-search test.
3. Manually-created search index returns in PENDING; ensureSearchIndex
treats non-READY/BUILDING as unexpected — added poll loop in the
conflicting-definition test before triggering the conflict.
* test(mongodb): tighten Task 16 — try/finally on ad-hoc client, add length guards before forEach
* docs(mongodb): add README walking from install through Atlas + self-hosted
* feat(mongodb): wire mongodb adapter into root build/test scripts and changesets
* ci(mongodb): add test_adapters_mongodb job using mongodb-atlas-local service
* docs(plan): record Task 19 gate wiring — test_adapters_mongodb in aggregate test job's needs array
* chore(changeset): add MongoDB adapter (minor)
* chore: gitignore adapters/mongodb/dist for parity with pg and cf
* fix(mongodb): drop rejected MongoClient promises from cache so next call retries
* test(mongodb): assert MongoClient.connect call count to gate the rejected-cache regression
* fix(mongodb): promise-singleton ensureSearchIndex; propagate listSearchIndexes errors
* test(mongodb): mock listSearchIndexes consistently across concurrent callers to actually gate the cache race
* fix(mongodb): default numCandidates to 10× limit, drop arbitrary 100 floor
* refactor(mongodb): capture URI in adapter closure; drop credential from payload custom
* test(mongodb): lock URI-not-on-config invariant in compliance spec
Task 4 stripped uri from _mongoConfig but compliance.spec.ts:29 still
asserted ext.custom._mongoConfig.uri === MONGO_URI, breaking the test
suite. Replace with a positive `not.toHaveProperty('uri')` assertion
so the security invariant is locked: re-introducing uri on the persisted
config would now fail this test with a clear "expected ... to not have
property 'uri'" message.
Manual revert proof performed (uri added back in src/index.ts → test
fails with expected message → restored).
* test(mongodb): boot real PayloadCMS in adapter specs; remove makeFakePayload shim
* chore(mongodb-tests): tighten injectDbName + drop unused import/cast
Code-review follow-ups from Task 5:
- injectDbName now throws on URIs with a path component (e.g. SRV
cluster strings carrying a default DB), instead of silently producing
invalid double-path URIs like mongodb+srv://.../myapp/test.
- Drop unused dropTestDb import in integration.spec.ts.
- Drop the (payload as any).destroy() escape hatch in teardownDbs;
BasePayload.destroy() is on the public type.
* test(mongodb): assert convertWhereToMongo rejects undeclared filter fields
* test(mongodb): add extensionFields spec covering index decl + persistence
* test(mongodb): multipools spec — collection+index isolation, cross-pool query isolation
* ci: verify @payloadcms-vectorize/mongodb dist/index.d.ts is emitted
* docs(mongodb): drop experimental framing — adapter is GA on Atlas, supported on 8.2+
* docs(mongodb): drop remaining experimental wording in README lede + npm keyword
* feat(mongodb): validate limit at search entrypoint with clear error
Throws "limit must be a positive integer" before reaching $vectorSearch
when limit is 0, negative, NaN, or non-integer. Without this guard the
driver returns an opaque "Executor error during aggregate command"
which is hard to trace back to caller input.
* test(mongodb): cover BUILDING -> READY polling transition
ensureSearchIndex polls listSearchIndexes every 1s until status === 'READY'
after creating a new index. Adds a fake-timers test that drives the mock
through []/BUILDING/BUILDING/READY and asserts createSearchIndex is called
once and listSearchIndexes is called 4 times.
Manual revert proof: relaxing the polling guard to also early-return on
BUILDING makes the test fail with "expected spy to be called 4 times,
but got 2 times".
* fix(mongodb): make definitionsEqual insensitive to mongot field/key ordering
mongot is free to return latestDefinition with the fields array in any
order and with object property keys in any order. The previous
JSON.stringify-based comparison flagged these as 'different definition'
and threw, forcing users to drop a perfectly valid index.
Canonicalize before compare:
- sort all object keys
- sort the fields array by canonical-JSON of each field
Other arrays (in case the schema grows later) keep their order so a real
ordered list can't be silently swallowed.
Tests:
- positive: a reordered-but-equivalent definition is accepted, no
createSearchIndex call.
- negative: a definition with a different similarity still throws
/different definition/.
Manual revert proofs:
- restoring JSON.stringify makes the positive test fail with
/different definition/.
- stubbing definitionsEqual to always-true makes the negative test fail
('promise rejected' vs resolved undefined).
* docs(mongodb): expand README — prerequisites, verify-it-works snippet, tuning, contributing, sibling parity
Addresses /judge-readme punch list:
- Add npm/CI/license/Payload badges and TOC
- Add Prerequisites section (Atlas / self-hosted / driver / Payload / Node)
- Quick Start now wires plugin + collections and includes a "Verify it works" curl block (ingest + search)
- Soften "GA on Atlas" claim — status callout names mongodb-atlas-local CI; surfaced again under Limitations
- Reconcile post-filter info: mechanism canonical in "WHERE clause behavior", Limitations links to it
- Add "How it works", "Who is this for?", and Tuning numCandidates/forceExact sections
- Add Multiple Knowledge Pools worked example
- Add Contributing section with test setup/teardown commands and src layout map
- Link CHANGELOG.md
* docs: surface mongodb adapter in root README and adapters README
- Available Adapters table: add mongodb row
- Installation snippet: add mongodb pnpm add line
- Quick Start adapter docs links: add mongodb pointer
- Migrations: note mongodb auto-ensures the $vectorSearch index
- Adapter Configuration: link mongodb API Reference
- Adapter parity callout: describe mongodb pre/post filter split
- Roadmap: move mongodb from "Help wanted" to "Already shipped"
- adapters/README.md Available Adapters: add mongodb row
Add `@payloadcms-vectorize/mongodb` adapter (Atlas + self-hosted Community 8.2+) backed by `$vectorSearch`, with pre/post filter splitting and full WHERE-clause parity across operators (equals, not_equals, in, notIn, like, contains, gt/gte/lt/lte, exists, and/or). Search indexes are auto-ensured on first use.
@@ -84,6 +86,7 @@ First, configure your database adapter. See the adapter-specific documentation:
84
86
85
87
-**PostgreSQL**: [@payloadcms-vectorize/pg README](./adapters/pg/README.md) — pgvector setup, schema initialization, and migrations.
86
88
-**Cloudflare Vectorize**: [@payloadcms-vectorize/cf README](./adapters/cf/README.md) — index creation, bindings, and known limitations.
89
+
-**MongoDB**: [@payloadcms-vectorize/mongodb README](./adapters/mongodb/README.md) — Atlas / self-hosted 8.2+, `filterableFields`, and the `$vectorSearch` index lifecycle.
87
90
88
91
### 2. Configure the Plugin
89
92
@@ -222,6 +225,7 @@ Migration steps depend on your database adapter:
- **Cloudflare Vectorize**: index creation is a one-time setup step — see [@payloadcms-vectorize/cf README](./adapters/cf/README.md#1-create-vectorize-index).
228
+
- **MongoDB**: no manual migration — the `$vectorSearch` index is auto-ensured on first write. See [@payloadcms-vectorize/mongodb README → Index lifecycle](./adapters/mongodb/README.md#index-lifecycle).
225
229
226
230
### 4. Search Your Content
227
231
@@ -298,6 +302,7 @@ Each adapter has its own configuration shape — this is where index parameters,
The embeddings collection name in Payload will be the same as the knowledge pool name.
303
308
@@ -374,7 +379,7 @@ You can filter on:
374
379
375
380
References to fields that don't exist on the embeddings table are silently dropped (the rest of the clause still applies).
376
381
377
-
>**Adapter parity.** All operators are implemented in`@payloadcms-vectorize/pg`. The Cloudflare Vectorize adapter has narrower native filtering — see [@payloadcms-vectorize/cf → Known Limitations](./adapters/cf/README.md#metadata-filtering) for what is and isn't supported there.
382
+
>**Adapter parity.** All operators are implemented in`@payloadcms-vectorize/pg`. The Cloudflare Vectorize adapter has narrower native filtering — see [@payloadcms-vectorize/cf → Known Limitations](./adapters/cf/README.md#metadata-filtering) for what is and isn't supported there. The MongoDB adapter splits the clause into a native `$vectorSearch` pre-filter and a JS post-filter — `like`/`contains`/`all` and any mixed-pre/post `or` are post-filtered, so they may return fewer than `limit` rows. See [@payloadcms-vectorize/mongodb → WHERE clause behavior](./adapters/mongodb/README.md#where-clause-behavior).
378
383
379
384
## Chunkers
380
385
@@ -1006,15 +1011,15 @@ Common scripts:
1006
1011
**Already shipped:**
1007
1012
1008
1013
- **Multiple Knowledge Pools** — independent configurations and embedding functions per pool.
- **Serverless-friendly job model** — bulk runs are split into small, requeueable units (`prepare-bulk-embedding` and `poll-or-complete-single-batch`) so individual jobs stay well under typical serverless time limits. The `batchLimit` option (see [CollectionVectorizeOption](#collectionvectorizeoption)) lets you cap docs-per-job to fit your platform. Tested locally and on Node-style hosts; deeper Vercel-specific integration testing is on the help-wanted list.
0 commit comments