Skip to content

feat(testkit): expand UTF adapter coverage and real-mongod parity path #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
midagedev merged 1 commit into from
Feb 24, 2026
Merged

feat(testkit): expand UTF adapter coverage and real-mongod parity path #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 47 additions & 62 deletions docs/COMPATIBILITY_SCORECARD.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,100 +2,85 @@

Status date: 2026-02-24

This document is the final R3 certification snapshot used for release decisions.
It is an integration-test compatibility report, not a production MongoDB parity claim.
Scope: release baseline for `v0.1.1` (`f4a8bbb`), not the latest `main` head.
This scorecard tracks integration-test compatibility against MongoDB official specs.
It is not a production MongoDB parity claim.

## Evidence Sources
## Scope

| Evidence | Source run / artifact | Result |
| --- | --- | --- |
| Official UTF sharded differential (baseline) | GitHub Actions `Official Suite Sharded` run `22332998372` | PASS |
| Official UTF sharded differential (rerun/flake gate) | GitHub Actions `Official Suite Sharded` run `22332998372` | PASS |
| R3 failure ledger | GitHub Actions `R3 Failure Ledger` run `22332937657` | PASS |
| External Spring canary certification | GitHub Actions `R3 External Canary Certification` run `22332937633` | PASS |
| Support manifest scorecard | `build/reports/r2-compatibility/r2-compatibility-scorecard.json` | PASS |
- Baseline target: `v0.1.2` and current `main` evidence.
- Primary objective: increase imported differential coverage while keeping behavior deterministic.

## Final Gate Summary
## Evidence Sources

| Gate | Status | Metrics |
| Evidence | Source run / artifact | Result |
| --- | --- | --- |
| UTF sharded baseline | PASS | imported=146, match=146, mismatch=0, error=0 |
| UTF sharded rerun | PASS | imported=146, match=146, mismatch=0, error=0 |
| UTF flake consistency | PASS | shard-0/1/2 baseline == rerun |
| R3 failure ledger | PASS | suiteCount=3, failureCount=0 |
| External canary certification | PASS | projectCount=3, canaryPass=3, rollbackSuccess=3, maxRecoverySeconds=38 |
| Spring compatibility matrix | PASS | totalCells=20, pass=20, fail=0, passRate=1.0 |
| Compatibility scorecard gates | PASS | pass=2, fail=0, missing=0 |
| Official UTF sharded differential | GitHub Actions `Official Suite Sharded` run `22339640229` | PASS |
| R3 failure ledger | GitHub Actions `R3 Failure Ledger` run `22339640205` | FAIL (gate reports remaining mismatches) |
| Real mongod baseline | GitHub Actions `Real Mongod Baseline` run `22339640211` | PASS |

## Post-0.1.1 Main Deltas (Not Included in This Snapshot)
## Frozen Baseline (Run `22339640229`)

The following `main` changes landed after `v0.1.1` and require the next release-cycle certification snapshot:

- `#86`: single-hook Spring test annotation (`@JongodbMongoTest`)
- `#87`: `countDocuments`, `replaceOne`, `findOneAndUpdate`, `findOneAndReplace` command support
- `#88`: ordered `bulkWrite` core subset
- `#91`: projection subset expansion for `findOneAndUpdate` / `findOneAndReplace`

## Official UTF Coverage Snapshot

Totals from the baseline shard artifacts in run `22332998372`:
Aggregated baseline metrics from shard artifacts:

| Metric | Value |
| --- | --- |
| imported | 146 |
| imported | 200 |
| skipped | 567 |
| unsupported | 868 |
| total differential cases | 146 |
| unsupported | 814 |
| total differential cases | 200 |
| match | 146 |
| mismatch | 0 |
| mismatch | 54 |
| error | 0 |

Top unsupported reasons in baseline artifacts:
## R3 Ledger Snapshot (Run `22339640205`)

| Suite | Imported | Unsupported | Mismatch | Error |
| --- | --- | --- | --- | --- |
| `crud-unified` | 188 | 362 | 54 | 0 |
| `transactions-unified` | 0 | 396 | 0 | 0 |
| `sessions` | 12 | 56 | 0 | 0 |

Current ledger gate status:

- `failureCount=54` (all `MISMATCH`, no `ERROR`)
- primary mismatch bucket: `bulkWrite` protocol scenarios

## Top Unsupported Reasons (Baseline)

| Reason | Count |
| --- | --- |
| `unsupported UTF operation: startTransaction` | 234 |
| `unsupported UTF operation: failPoint` | 134 |
| `unsupported UTF operation: clientBulkWrite` | 86 |
| `unsupported UTF operation: bulkWrite` | 84 |
| `unsupported UTF operation: findOneAndUpdate` | 48 |
| `unsupported UTF operation: createEntities` | 40 |
| `unsupported UTF operation: findOneAndReplace` | 36 |
| `unsupported UTF aggregate stage in pipeline` | 28 |
| `unsupported UTF operation: countDocuments` | 22 |
| `unsupported UTF update option: arrayFilters` | 26 |
| `unsupported UTF operation: countDocuments` | 24 |
| `unsupported UTF operation: replaceOne` | 22 |

## Artifact Digest Snapshot
## Policy Exclusions

SHA-256 digests captured during certification assembly:
- `failPoint` is currently treated as a policy exclusion for deterministic in-process execution.
- Scorecard accounting distinguishes this as `unsupported-by-policy UTF operation: failPoint` in importer output.

| Artifact | SHA-256 |
| --- | --- |
| `utf-shard-summary.md` | `93a8fefa915b48e6799f9f3f4004c53bf4d678a7f473b41e75ca5e3ee974fd08` |
| `shard-0 baseline utf-differential-report.json` | `483cd8cfbd8acd3ea590849bfe063b8abf31b4bbe37800e6c6ba5742b99b92af` |
| `shard-1 baseline utf-differential-report.json` | `cf226ab24bdddce9c6748bf1393cbaf776c704358788137f9875f78dbb6856eb` |
| `shard-2 baseline utf-differential-report.json` | `6dabf85f22be0e60bf275ab0fdc896a7fbc35413ce455c678ada5b0e32fe68e3` |
| `r3-failure-ledger.json` | `0dfe884ed25be8be46c2d7937f8386075ec00e90e4d5c5e83ccdbbe2b2079d08` |
| `r2-canary-certification.json` (R3 canary output artifact name) | `8ffd103c55f0cb3ef95c7fc52bfc347454984fcc057419c50e607acf4f565103` |
| `r2-compatibility-scorecard.json` | `3ba3469366a19453786e631841a633645082d6cd80fd1a22a50ed7ba4dead6a0` |
| `r2-support-manifest.json` | `84fd659f98a332134209b02d1302321aceaf56125fd691e0a7d38e564f4f913f` |
## Gap-to-Issue Mapping

- `#100`: transaction operation adapter coverage (`startTransaction` and lifecycle wiring)
- `#101`: unified CRUD adapter coverage (`findOneAndUpdate`, `findOneAndReplace`, `countDocuments`, `replaceOne`)
- `#102`: `createEntities` subset for official suites
- `#103`: explicit `failPoint` policy for deterministic in-memory backend
- `#104`: aggregate-stage unsupported reduction

## Reproduction

Run the same gates locally:
Run the same ledger locally:

```bash
./.tooling/gradle-8.10.2/bin/gradle --no-daemon springCompatibilityMatrixEvidence

./.tooling/gradle-8.10.2/bin/gradle --no-daemon \
r2CompatibilityEvidence \
-Pr2CompatibilityUtfReport=/tmp/official-utf-summary.json \
-Pr2CompatibilitySpringMatrixJson=build/reports/spring-matrix/spring-compatibility-matrix.json \
-Pr2CompatibilityFailOnGate=true

./.tooling/gradle-8.10.2/bin/gradle --no-daemon \
r3CanaryCertificationEvidence \
-Pr3CanaryInputJson=testkit/canary/r3/projects.sample.json \
-Pr3CanaryFailOnGate=true
-Pr3SpecRepoRoot="<path-to-mongodb-specifications>" \
-Pr3FailureLedgerMongoUri="<replica-set-uri>" \
-Pr3FailureLedgerFailOnFailures=true \
r3FailureLedger
```
2 changes: 1 addition & 1 deletion src/main/java/org/jongodb/testkit/DifferentialHarness.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ private static void compareMap(
}

private static boolean isEphemeralPath(final String path) {
return path.endsWith(".cursor.ns");
return path.endsWith(".cursor.ns") || path.endsWith(".cursor.id");
}

private static void compareList(
Expand Down
142 changes: 114 additions & 28 deletions src/main/java/org/jongodb/testkit/RealMongodBackend.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,15 @@
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoDatabase;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import org.bson.BsonArray;
import org.bson.BsonDocument;
import org.bson.BsonInt64;
import org.bson.BsonValue;

/**
* Differential backend adapter that executes scenario commands against real mongod.
Expand Down Expand Up @@ -52,39 +56,48 @@ public ScenarioOutcome execute(Scenario scenario) {
Objects.requireNonNull(scenario, "scenario");
String databaseName = scenarioDatabaseName(scenario.id());

try (MongoClient client = clientFactory.create(connectionUri);
ClientSession session = client.startSession()) {
try (MongoClient client = clientFactory.create(connectionUri)) {
final Map<String, ClientSession> sessionPool = new HashMap<>();
MongoDatabase database = client.getDatabase(databaseName);
resetDatabase(database);

List<Map<String, Object>> commandResults = new ArrayList<>(scenario.commands().size());
for (int i = 0; i < scenario.commands().size(); i++) {
ScenarioCommand command = scenario.commands().get(i);
MongoDatabase commandDatabase = resolveCommandDatabase(client, database, command.commandName());
BsonDocument commandDocument;
try {
commandDocument = ScenarioBsonCodec.toRealMongodCommandDocument(command, databaseName);
} catch (RuntimeException exception) {
return ScenarioOutcome.failure(
"invalid command payload for " + command.commandName() + ": " + exception.getMessage()
);
}

BsonDocument responseBody;
try {
responseBody = commandDatabase.runCommand(session, commandDocument, BsonDocument.class);
} catch (MongoCommandException commandException) {
responseBody = commandException.getResponse();
} catch (RuntimeException exception) {
return ScenarioOutcome.failure(
"real mongod execution failed for " + command.commandName() + ": " + exception.getMessage()
);
}

commandResults.add(ScenarioBsonCodec.toJavaMap(responseBody));
if (!ScenarioBsonCodec.isSuccess(responseBody)) {
return ScenarioOutcome.failure(ScenarioBsonCodec.formatFailure(command.commandName(), i, responseBody));
try {
for (int i = 0; i < scenario.commands().size(); i++) {
ScenarioCommand command = scenario.commands().get(i);
MongoDatabase commandDatabase = resolveCommandDatabase(client, database, command.commandName());
BsonDocument commandDocument;
try {
commandDocument = ScenarioBsonCodec.toRealMongodCommandDocument(command, databaseName);
} catch (RuntimeException exception) {
return ScenarioOutcome.failure(
"invalid command payload for " + command.commandName() + ": " + exception.getMessage()
);
}

final ClientSession session = resolveSession(client, command, sessionPool);

BsonDocument responseBody;
try {
responseBody = session == null
? commandDatabase.runCommand(commandDocument, BsonDocument.class)
: commandDatabase.runCommand(session, commandDocument, BsonDocument.class);
} catch (MongoCommandException commandException) {
responseBody = commandException.getResponse();
} catch (RuntimeException exception) {
return ScenarioOutcome.failure(
"real mongod execution failed for " + command.commandName() + ": " + exception.getMessage()
);
}
responseBody = normalizeResponseForComparison(command, responseBody);

commandResults.add(ScenarioBsonCodec.toJavaMap(responseBody));
if (!ScenarioBsonCodec.isSuccess(responseBody)) {
return ScenarioOutcome.failure(ScenarioBsonCodec.formatFailure(command.commandName(), i, responseBody));
}
}
} finally {
closeSessions(sessionPool);
}
return ScenarioOutcome.success(commandResults);
} catch (RuntimeException exception) {
Expand Down Expand Up @@ -123,6 +136,79 @@ private static MongoDatabase resolveCommandDatabase(
return defaultDatabase;
}

private static ClientSession resolveSession(
final MongoClient client,
final ScenarioCommand command,
final Map<String, ClientSession> sessionPool
) {
final String sessionId = readSessionId(command);
if (sessionId == null) {
return null;
}
return sessionPool.computeIfAbsent(sessionId, ignored -> client.startSession());
}

private static String readSessionId(final ScenarioCommand command) {
final Object lsidValue = command.payload().get("lsid");
if (!(lsidValue instanceof Map<?, ?> lsid)) {
return null;
}
final Object idValue = lsid.get("id");
if (idValue == null) {
return null;
}
final String normalized = String.valueOf(idValue).trim();
if (normalized.isEmpty()) {
return null;
}
return normalized;
}

private static void closeSessions(final Map<String, ClientSession> sessions) {
for (final ClientSession session : sessions.values()) {
try {
session.close();
} catch (RuntimeException ignored) {
// Best-effort cleanup for test harness sessions.
}
}
}

private static BsonDocument normalizeResponseForComparison(
final ScenarioCommand command,
final BsonDocument responseBody
) {
if (!"countDocuments".equals(command.commandName())) {
return responseBody;
}
if (!responseBody.containsKey("n") || responseBody.containsKey("count")) {
final BsonValue cursorValue = responseBody.get("cursor");
if (cursorValue == null || !cursorValue.isDocument()) {
return responseBody;
}
final BsonValue firstBatchValue = cursorValue.asDocument().get("firstBatch");
if (firstBatchValue == null || !firstBatchValue.isArray()) {
return responseBody;
}
final BsonArray firstBatch = firstBatchValue.asArray();
long count = 0L;
if (!firstBatch.isEmpty()) {
final BsonValue first = firstBatch.get(0);
if (first != null && first.isDocument()) {
final BsonValue nValue = first.asDocument().get("n");
if (nValue != null && nValue.isNumber()) {
count = nValue.asNumber().longValue();
}
}
}
return new BsonDocument()
.append("n", new BsonInt64(count))
.append("count", new BsonInt64(count))
.append("ok", responseBody.get("ok"));
}
return responseBody.clone().append("count", responseBody.get("n"));
}

private static void resetDatabase(MongoDatabase database) {
Objects.requireNonNull(database, "database");
database.drop();
Expand Down
Loading