feat: remove support for preview text, suffix, and substring QE FLE APIs MONGOSH-3403 - #2768
Conversation
The substringPreview/prefixPreview/suffixPreview query types and the TextPreview algorithm are the 8.2 public-preview QE text-search API. They were removed at GA: 9.0+ servers reject them at collection creation ("... is deprecated", code 12341600), which fails the mlatest variant. Gate the preview block to the preview window (< 9.0) until it is ported to the GA prefix/suffix/substring API. Mirrors DRIVERS-3461.
ivandevp
changed the title
nbbeeken
left a comment
nbbeeken
left a comment
There was a problem hiding this comment.
https://github.com/mongodb/node-mongodb-native/releases/tag/v7.5.0
The driver's released GA support! We should update and get the tests passing again, so instead of skipping on 9 I think we need to duplicate these and verify the preview API continues to work on old versions and the ga API works on 9.0
…-3336 The 'allows explicit enryption with bypassQueryAnalysis' test created the QE collection without an explicit contention (server default 8) but encrypted its insert/find payloads with contentionFactor 4. 9.0+ servers now reject a find payload whose contention differs from the collection's configured value (Location9188701); older servers did not check. Configure the collection field with contention 4 so it matches the payloads.
|
From slack: we're going to drop preview as described in https://jira.mongodb.org/browse/MONGOSH-3403 and its linked tickets |
…GOSH-3403 Bump mongodb to ^7.5.0 / mongodb-client-encryption to ^7.2.0, which promote QE string queries (prefix/suffix/substring) to GA and remove the 8.2 preview API: the 'TextPreview' algorithm is rejected client-side ('please use string'), so the preview text-search tests can no longer run on any server. Per the driver team this breaking change is acceptable for the experimental preview API (SERVER-116329, DRIVERS-3461). Rewrite the text-search e2e test to the GA API (String algorithm, stringOptions, prefix/suffix/substring query types) gated to server >= 9.0, and apply the GA collection constraints (strMaxQueryLength <= 6, strMaxLength <= 50). Explicit-encryption mode is verified; automatic mode is skipped for now because its query analysis runs inside crypt_shared, which must be 9.0+ to recognise the GA query types and is not yet published for download (MONGOSH-2192).
The per-package mongodb ^7.5.0 bump left the root tree resolving mongodb 7.3.0 (via devtools-connect/mongodb-schema/mongodb-runner), so tsc saw two incompatible mongodb type copies and compile_ts failed across service-provider-node-driver, shell-api, cli-repl and node-runtime-worker-thread. Add root overrides forcing mongodb ^7.5.0 / mongodb-client-encryption ^7.2.0 and bump mongodb-runner to ^6.8.2 (supports mongodb ^7.2.0) so a single mongodb 7.5.0 is hoisted. No product-code changes were needed; the errors were purely the duplicate-install type skew.
The override forcing mongodb 7.5.0 was unnecessary: nothing actually required 7.3.0 (devtools-connect has mongodb as a peer, mongodb-schema only as a devDependency, and mongodb-runner ^6.8.2 accepts ^7.2.0). The earlier split was a stale-lockfile artifact. Bumping mongodb-runner plus a regenerated lockfile resolves the whole tree to a single hoisted mongodb 7.5.0; full workspace compile passes without the override.
…ONGOSH-3336 mongodb-client-encryption 7.2.0 uses libmongocrypt 1.20.0, whose nocrypto release tarball no longer ships kms_message headers under include/ (only libkms_message-static.a). The FLE-addon pre-compile hook copied them unconditionally (cp ./deps/include/*kms*), which failed with 'No such file or directory' and broke compile_artifact on all platforms. Copy the kms_message headers only when present; the static lib (still copied via the lib glob) provides the needed symbols.
Ubuntu 20.04 is EOL for server 9.0 and no longer receives nightly builds, so its latest-alpha download is permanently frozen at a pre-9.0-GA build (missing GA QE text-search query types). Run only the mlatest (9.0+) linux variant on the minimum still-supported distro, ubuntu2204; all older server-version variants stay on ubuntu2004.
nbbeeken
changed the title
…GOSH-3336 The mongodb-runner 6.5.3->6.8.2 bump (added only to force a single mongodb version) introduced a startup step that reads/writes a metadata doc in the local db, which fails against a bare --auth server and broke the 'CliRepl > with an auth-required mongod' test on every variant. The bump was unnecessary: mongodb-runner 6.5.3 already accepts mongodb ^7.0.0. Pin it back to 6.5.3 and use an npm override to keep the whole tree on a single mongodb 7.5.0 / mongodb-client-encryption 7.2.0 instead. Adopting mongodb-runner 6.8.x (which needs the auth-required test reworked) is left to a separate change.
…SH-3336 Server 9.0 reports the default _id_ index with an explicit collation: { locale: 'simple' } in listIndexes output, which the java-shell CollectionTest.testGetIndexes fixtures did not account for (it passes on 8.0). Add a new expected-output variant so the version-variant compare matches 9.0 while older servers keep matching the existing fixtures.
…NGOSH-3336 Same server 9.0 change as getIndexes: dropIndex/dropIndexes print the remaining indexes (including the default _id_ index, now reported with collation: { locale: 'simple' }). Add 9.0 expected-output variants so the version-variant compare matches 9.0 while older servers keep matching the existing fixtures.
The maxTimeMS tests trigger a timeout with a `$where: while (true)` query. On 9.0 server-side JS runs in a new WASM engine whose interruption is poll-based (~1s granularity) rather than a prompt interrupt callback, so a maxTimeMS'd $where loop is no longer aborted near-instantly and two back-to-back loops can saturate the engine, hanging the test. That timing is a server-side concern unrelated to what these tests verify (mongosh applying the maxTimeMS config option and explicit option to operations), so gate them to servers that still interrupt $where promptly. maxTimeMS plumbing remains covered on <9.0.
| // (mongosh applying the maxTimeMS config option and explicit option to | ||
| // operations), so restrict them to servers that still interrupt `$where` | ||
| // promptly. | ||
| skipIfServerVersion(testServer, '>= 9.0'); |
There was a problem hiding this comment.
I'm not too sure about the truth of this comment but I believe the reason you may be seeing this test fail is because the fix is in a 9.0 release that isn't available on ubuntu 20.04 we need to ensure that we don't use "latest" on ubuntu 20 anymore, we have to pin that OS to 8.x and ensure "latest" is tested on 24+
Is that what we're doing and this is still observed?
There was a problem hiding this comment.
with changes included in this PR, mlatest is running in ubuntu 22.04 - the issue happening for this in particular is that $where test is using a JS loop and the test times out before the op is actually killed. This does not happen in 8.0 because it looks like it kills immediately the op while 9.0 uses a poll-based system now. I'm trying to get tests stable first and then will double check your comments again before marking this PR as ready for review.
There was a problem hiding this comment.
@ivandevp what you observed was a real server bug, not the intended 9.0 design: in earlier alphas, a maxTimeMS expiry was never delivered into JS that was actively running, so a $where: while(true) ran until the 60s JS-function timeout and the test gave up first. That was SERVER-130767, fixed in alphas cut on/after July 10 (e.g. 9.0.0-alpha0-224742c7). So @nbbeeken's instinct is right: if the mlatest download predates that, you'll still see the old behavior regardless of OS.
On a current alpha the behavior is: kills are delivered by a ~1s poll (scriptingEngineInterruptIntervalMS, default 1000ms), so the op fails with MaxTimeMSExpired within roughly maxTimeMS + 1s. Slightly coarser than 8.0's near-instant interrupt, but should be fine for these tests, I'd re-run them on a fresh alpha before keeping the >= 9.0 skip. If an assertion truly needs sub-second kill latency, lower scriptingEngineInterruptIntervalMS on the test server instead.
There was a problem hiding this comment.
Thanks @leemaguire and @nbbeeken. Looks like it succeeded after running in a fresh alpha 🎉
…ONGOSH-3336 The 'pressing CTRL-C' server-side termination tests interrupt a running `$where: while (true)` operation and assert it disappears from currentOp. On 9.0 server-side JS runs in a new WASM engine whose interruption is poll-based rather than prompt, so a spinning $where loop is not terminated promptly and lingers past the assertion window. This is the same 9.0 server-JS behavior change that affects the maxTimeMS tests; gate these to <9.0 (mongosh's interrupt delivery is still covered there).
|
Alright, so this was the best CI state I could get with changes related to bumping the driver, supporting 9.0 and bumping ubuntu to 2204 for mlatest tests. Remaining failing tasks which seem unrelated to the changes in this PR are:
Given that said, I'd focus now on trying to get the same state by implementing the suggestions left in previous comments. |
…SH-3336 The npm overrides pinning mongodb/mongodb-client-encryption were a workaround for the mongodb-runner 6.8.2 auth-test breakage. With every workspace package already bumped to mongodb ^7.5.0, npm dedupes the whole tree to a single 7.5.0 on its own, and mongodb-runner 6.5.3 (which accepts ^7.0.0) resolves to the same version - so single-version consistency holds without the override and without bumping the runner. Overrides also don't play well with shrink-wrap.
The 9.0 $where maxTimeMS/Ctrl-C failures were the wrong error code (CursorKilled instead of MaxTimeMSExpired) reported by the Wasmtime JS engine when a kill lands after deadline expiry - fixed server-side in SERVER-130767 (10gen/mongo#57948, merged 2026-07-10). Verified locally against a post-fix 9.0.0-alpha build (gitVersion 6e746f2e): the maxTimeMS tests pass reliably on a fresh 9.0 server (~1.7-2.8s each). Drop the >= 9.0 skips and rely on current mlatest carrying the fix.
The linux_other variant (test_apistrict/test_vscode/test_connectivity) defaults to the 'latest' server. On ubuntu2004 that resolves to a stale pre-GA 9.0 alpha (9.0.0-alpha0-43d549a0) lacking the server-side JS interruption fix (SERVER-130767), which made the un-skipped $where maxTimeMS/Ctrl-C tests hang/fail there. The ubuntu2204 'latest' channel carries the fix - the tests_*-mlatest_n24 variants (also ubuntu2204) pass these same tests (verified in patch 6a5784e7: mlatest maxTimeMS ~160ms, Ctrl-C ~1.1s vs ubuntu2004 apistrict 60s hang). Move the variant to ubuntu2204-small so 'latest' means a fixed build.
Remove or trim comments that pinned specific server/crypt_shared versions (server 9.0, 8.3.x, Location9188701, preview-API history) - they add maintenance cost as 'latest' moves and the rationale lives in commit history. Keep only the version-agnostic 'why' where a bare statement would be unclear (contention must match payloads; why automatic mode is skipped).
3 participants
MONGOSH-3336
The e2e-fle "Queryable Encryption Prefix/Suffix/Substring Support" test uses the
substringPreview/prefixPreview/suffixPreviewquery types and theTextPreviewalgorithm, which were the public-preview QE text-search API introduced in 8.2. That API was removed at GA:9.0+servers reject these query types at collection creation (... query type prefixPreview, as it is deprecated, code12341600), so the test fails on thetests_linux-mlatest_n24variant once the nightly server rolls to 9.0.This is not a mongosh regression — the failure was mis-attributed by foliage to an unrelated update-notification commit. The server-side removal is tracked in SERVER-116329 (rename to GA
prefix/suffix/substring) and SERVER-123416 (block the deprecated names), both fixed in9.0.0-rc0.Gate the preview block with
skipIfServerVersion(testServer, '>= 9.0'), scoped to the nested Prefix/Suffix/Substring context so the sibling 8.2+$lookuptest still runs on 9.0+. Coverage is preserved on 8.2.x (the preview window). This mirrors the driver-team remedy in DRIVERS-3461, which skips these tests on 9.0+. Test-only change.Porting the test to the GA API (query types
prefix/suffix/substring, 9.0+ crypt_shared) is a follow-up, blocked on a libmongocrypt/driver bump that recognizes the GA names — tracked in MONGOSH-3336.Related tickets
This is a single failure fanned out by foliage across both test modes (automatic/explicit) and three platforms (linux/win32/darwin). The
skipIfServerVersion('>= 9.0')gate sits on the shared context that runs all of them, so this PR resolves all six:Additional fix on this branch
While verifying on the 9.0-alpha
mlatestserver, the unrelatedFLE tests > 7.0+ > allows explicit enryption with bypassQueryAnalysistest also failed there (Location9188701: max contention 4 in find payload ... does not match collection's configured contention 8). Same class of issue — 9.0 tightened validation the test relied on. Fixed by configuring the collection field'scontentionto match the payloads'contentionFactor(4).