Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
/.gitattributes text eol=lf
benchmark-evidence/** -text diff whitespace=cr-at-eol
benchmark-evidence/**/*.log whitespace=-blank-at-eol,-blank-at-eof,cr-at-eol
benchmark-evidence/**/*.gz -text -diff
benchmark-evidence/**/*.pyc -text -diff
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@
# macOS AppleDouble in-progress turds
.!*!*
.gstack/
__pycache__/
*.pyc
6 changes: 6 additions & 0 deletions benchmark-evidence/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Credentials and account-specific cloud state never enter the public fork.
*.env
*.key
*.token
cloud/credentials/
cloud/session/
77 changes: 77 additions & 0 deletions benchmark-evidence/BASELINE_README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Independent RocketRide Benchmark Study

This bundle contains three deliberately separate tracks at upstream commit
`43be41acb58558dfae8e2e3deb86d8a00cb1b1c8`.

| Track | Evidence status | Promotion status |
|---|---|---|
| A: unchanged upstream reproduction | Complete on supported Linux; Windows appendix incomplete | Independent, unsubmitted |
| B: Node application extension | Complete, 90 results and 30 RocketRide traces | Blocked by the fixed 10 s protocol deadline |
| C: RocketRide Cloud appendix | Control complete and audited; exact upstream pipe blocked | Operational, unsubmitted |

## Track A

The primary reproduction ran the unchanged benchmark source, fixtures,
competitors, pool sizes, retries, and timeouts on Ubuntu 22.04 under WSL2. All
10 fault-isolation, 10 concurrent-processing, 10 data-isolation, and one
authoring-effort cells completed. The earlier Windows run is retained as an
incomplete exploratory appendix rather than substituted for the supported Linux
result.

This supports only an independent reproduction claim. It does not support a
general framework-superiority claim or an official RocketRide claim without an
external acceptance receipt.

## Track B

Track B compares native worker threads, RocketRide 1.2.0 with local engine
3.2.1.30, and real `langchain-core==0.3.86`. All executors return candidate-only
results; application validation, version checks, CAS, proposals, review, and
final writes remain application-owned.

The N=3 matrix completed for five frozen fixtures in normal and injected
hard-failure variants. Its orchestration gate passed, but the post-run protocol
audit found that all 30 RocketRide envelopes exceeded the fixed 10,000 ms total
deadline. Evidence is complete; protocol admission and promotion are blocked.

## Track C

The `JULY2026BENCHMARK` promotion was redeemed for Starter at 100 percent off.
The subscription readback reports active Starter, USD 0 due at checkout, and
renewal disabled at period end. A pre-registered Cloud-native control passed 48
normal requests, 9 unaffected failure-phase requests, 3 expected requests to a
terminated task, and 0 cross-task leaks across 12 resident tasks.

The unchanged upstream fault-isolation pipeline did not run on Cloud because
the hosted catalog lacks its custom `workload` service. Track C is therefore an
operational control, not an upstream benchmark score and not a comparison with
LangChain. The failed first receipt and passing rerun are both retained.

## Frozen Applications

| App | Frozen base | Adapter commit |
|---|---|---|
| NodeRoom | `ca25e347dc467bc37f06918e1a18656f7336ee28` | `353ddcce5b4ee791c4a0d70713a47ab302a643b9` |
| NodeBenchAI | `6ed0a58eeda993ff2a937ea4bacc2856756dd521` | `651020807ed13ea3f36cdf6be601f5a874705b29` |
| NodeSlide | `dd67e4c642c40e6bb414af617a67a31dbed507c5` | `702020d5cdd9785bfb4e6b79a30246a3c88a23b5` |
| NodeVideo | `bb79bc385de93c90cee89b160fc801d18372d89e` | `cf2c696e1c0ccea6ebb43617dc81a23f05e785da` |

## Evidence Map

- Track A primary summary: `baseline/linux/linux-20260715T122439Z/aggregate/summary.json`
- Track A Windows appendix: `baseline/concurrent-work/aggregate/summary.json`
- Track B pre-registration: `extensions/node-suite/PRE_REGISTRATION.md`
- App verifier: `extensions/node-suite/app-verification.json`
- Post-run app verifier: `extensions/node-suite/app-verification-post-run.json`
- Track B original scorecard: `extensions/node-suite/runs/node-suite-20260715T150250Z/scorecard.json`
- Track B superseding protocol audit: `extensions/node-suite/runs/node-suite-20260715T150250Z/audit.json`
- Track C pre-registration: `cloud/PRE_REGISTRATION.md`
- Track C failed first run: `cloud/runs/cloud-20260715T225614Z/audit.json`
- Track C passing control audit: `cloud/runs/cloud-20260715T230356Z/audit.json`
- Track C refreshed billing UI receipt: `cloud/ui-verification.json`
- Rerun history: `failures.jsonl` and `deviations.json`

No model API was called. RocketRide Cloud was called only in Track C under the
redeemed promotion. Paid model cost and Cloud checkout charge are both USD 0;
the passing control consumed 15.3 promotional compute tokens. Raw credentials,
task tokens, account identifiers, and local environment files are not included.
50 changes: 50 additions & 0 deletions benchmark-evidence/LIMITATIONS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Limitations

- No RocketRide team acceptance, official publication, or external scorer receipt
exists. Every result is independent and unsubmitted.
- The primary Linux host is Ubuntu 22.04 under WSL2 on x86-64, not the Apple
Silicon host used for upstream committed results. Timing remains host-dependent.
- The Windows appendix is incomplete: data isolation never completed its first
repetition within the unchanged retry and timeout contract.
- Track B covers five deterministic fixtures with frozen candidate artifacts. It
measures binding, scheduling, failure isolation, and admission behavior, not
model quality or framework-specific artifact generation.
- Broader cancellation, checkpoint/resume, duplicate callback, and malformed
candidate scenarios remain outside the scored V1 matrix even where app verifier
tests cover related invariants.
- Track B V1's `totalMs` includes RocketRide pool warmup and remains protocol
blocked. V2 is the separately pre-registered always-warm study and cannot
replace that unfavorable result.
- V2 excludes RocketRide engine and pool readiness from the request clock while
retaining those values as standing and recovery metrics. Native and LangChain
retain subprocess startup in their totals, so cross-framework speed rankings
are not symmetric and are not claimed.
- V2 pool warmup remained substantial: 6,976.188-11,940.908 ms across the 20
provisioned pools. Meeting the request deadline therefore does not erase
deployment and failed-worker replacement cost.
- Native controls are additive deterministic workers, not a claim about every
production deployment topology in the four applications.
- Application validation, version checks, CAS, proposals, and review are
application guarantees. No framework receives credit for them, and Track B
attempted no final writes.
- Local compute and engineer time are not monetized. Only model and cloud API spend
are stated as USD 0.
- RocketRide Cloud was run only for the separately pre-registered deterministic
Track C control. It used no model provider and cannot replace the pinned local
baseline or supply a LangChain comparison.
- The exact upstream Cloud pipeline remains blocked because its custom
`workload` service is absent from the hosted catalog. Cloud validation did not
expose that incompatibility before execution.
- Track C injects failure by terminating a hosted task through the API. It does
not prove isolation from an operating-system-level hard crash inside a hosted
worker.
- Hosted task startup was 25.867-44.476 seconds in Track C. Request latency
excludes that standing cost but reports it separately.
- The Starter promotion is 100 percent off for the checkout period only. Renewal
was disabled and verified with `cancelAtPeriodEnd=true`; future account state
remains external to this repository.
- Physical and non-blank line counts are descriptive authoring evidence, not a
semantic complexity score.
- Failed provisioning and harness attempts are preserved. Later successful runs
do not erase the missing-provider, unbounded-send, or connection-lifecycle
failures recorded in `failures.jsonl` and `deviations.json`.
147 changes: 147 additions & 0 deletions benchmark-evidence/RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Results

## Claim Status

- **Track A:** evidence complete, independent unsubmitted reproduction.
- **Track B V1:** evidence complete, orchestration gate passed, protocol
admission failed, promotion blocked.
- **Track B V2:** evidence complete, orchestration and protocol admission
passed, eligible for external submission.
- **Track C:** Cloud-native operational control passed and independently
audited; exact upstream Cloud execution blocked by missing `workload` service.
- **Official status:** no external RocketRide acceptance or publication receipt.
- **Paid spend:** USD 0 model cost and USD 0 cloud cost.

## Track A: Supported Linux

The unchanged upstream suite completed at commit
`43be41acb58558dfae8e2e3deb86d8a00cb1b1c8` with no aggregate gate issues.

| Dimension | RocketRide | LangChain control |
|---|---|---|
| Hard-failure isolation, N=10 | Held 10/10 | Lost all work 10/10 |
| Concurrent work, pool 8 | 10/10 clean; wall median 1.285 s, p95 1.548 s | `batch_shared` crashed; `abatch` median 7.522 s; sequential median 7.530 s |
| Concurrent work, pool 16 | 10/10 clean; wall median 0.867 s, p95 1.040 s | Same controls as above |
| Data isolation, N=10 | 10/10 clean; 0 lost, duplicated, or leaked; wall median 5.508 s | Not a direct equivalent result in this cell |
| Warm deployment cost | Pool 8 median 9.160 s; pool 16 median 15.590 s; data median 30.085 s | Reported separately from measured work |
| Authoring descriptor | 0 imperative lines in the authored pipe | 14-17 imperative lines; 5 decision points for the correct idiom |

The thread-count appendix emitted a median 31 node errors and is not used as a
headline result.

## Track A: Windows Appendix

The unchanged Windows path is evidence-incomplete. Fault and concurrency cells
completed, but data-isolation repetition 1 timed out on all five unchanged
360-second attempts.

- RocketRide pool 8 was clean 10/10 but had wall median 45.514 s versus
LangChain `abatch` 7.292 s.
- RocketRide pool 16 was clean 0/10, retained 152 rows where 80 were expected,
and had wall median 60.821 s. Open SQLite cleanup on Windows is a recorded
inference, not a patched or promoted result.
- This appendix cannot replace the supported Linux reproduction.

## Track B V1: Node Applications

The app verifier passed all four immutable adapter commits, base ancestry,
canonical fixture parity, application tests, and shared protocol hash
`b9b8b9f293c486fcd68f532f177e40f86c07c9310f31a64aa316ba469c1b1d59`.

The completed matrix contains five fixtures, two variants, three repetitions,
three frameworks, 30 requests, 90 result envelopes, 30 RocketRide traces, and
30 aggregate rows.

| Dimension | Native | RocketRide | LangChain |
|---|---|---|---|
| Normal correctness | 4/4 units in all 15 runs | 4/4 units in all 15 runs | 4/4 units in all 15 runs |
| Injected hard failure | 3 unaffected units in all 15 runs | 3 unaffected units in all 15 runs; engine healthy after every run | Shared interpreter completed 0/4 in all 15 runs |
| Normal total median range | 190.836-203.875 ms | 25,966.952-31,814.868 ms | 1,220.388-1,390.712 ms |
| Hard-failure total median range | 176.464-222.259 ms | 31,934.275-37,909.386 ms | 1,057.141-1,139.453 ms before process loss was recorded |
| Candidate digest | Frozen candidate parity held | Frozen candidate parity held | Frozen candidate parity held |
| Model/cloud cost | USD 0 / USD 0 | USD 0 / USD 0 | USD 0 / USD 0 |

### Promotion Blocker

The original scorecard's completion/isolation gate reports `passed`. The later
audit mirrors the shared app protocol and supersedes that status for promotion:
all 30 RocketRide envelopes had `totalMs > deadlineMs` (10,000 ms). They would be
rejected before candidate admission. The audit therefore reports:

```text
evidenceStatus: complete
protocolAdmissionStatus: failed
promotionStatus: blocked
officialStatus: separate_application_study_unsubmitted
```

This is still a useful integration result: RocketRide isolated worker failure,
but the tested local resident-pool lifecycle is too slow for the pre-registered
application contract. The appropriate next experiment is a newly pre-registered
warm-service topology or deadline budget study, not retroactively changing this
run.

## Track B V2: Resident Runtime

V2 was separately pre-registered and committed before implementation. It keeps
the five frozen candidates and 10,000 ms request deadline, adds a fixed 2,000 ms
per-unit failure-detection budget, and binds every request and result to one
resolved-definition digest containing the merged app commit plus protocol,
adapter, domain-tool, validator, and fixture hashes.

The completed V2 matrix again contains 30 requests, 90 results, 30 RocketRide
traces, and 30 aggregate rows. The independent audit reports complete evidence,
passed protocol admission, and `eligible_for_external_submission`.

| Dimension | Native | RocketRide | LangChain |
|---|---|---|---|
| Normal correctness | 4/4 in all 15 runs | 4/4 in all 15 runs | 4/4 in all 15 runs |
| Injected hard failure | 3 unaffected units in all 15 runs | 3 unaffected units in all 15 runs; engine healthy after every run | Shared interpreter completed 0/4 in all 15 runs |
| Normal total median range | 136.618-156.006 ms | 1,043.073-1,090.228 ms | 790.117-897.225 ms |
| Hard-failure total median range | 137.180-161.455 ms | 3,008.731-3,025.511 ms | 733.217-930.532 ms before process loss was recorded |
| Deadline overruns | 0/30 | 0/30 | 0/30 |
| Model/cloud API cost | USD 0 / USD 0 | USD 0 / USD 0 | USD 0 / USD 0 |

RocketRide standing cost remains material and is not hidden: engine cold start
was 1,569.703 ms, normal pool warmup median was 7,721.659 ms, and replacement
pool warmup after destructive fault median was 10,109.896 ms. Native and
LangChain totals conservatively retain subprocess startup while RocketRide V2
uses the pre-registered prewarmed request clock, so V2 supports protocol
admission and fault-isolation claims, not a raw speed-winner claim.

All four adapter PRs are merged, their post-merge checks passed, and the exact
main revisions are deployed to production. Deployment IDs, immutable URLs,
aliases, and HTTP checks are recorded in
`extensions/node-suite/production-deployment-v2.json`.

## Track C: RocketRide Cloud

Track C was pre-registered before its measured hosted run. It uses the official
Cloud endpoint and a deterministic built-in `webhook -> parse -> response_text`
pipeline with no model provider. Each of three repetitions created four
resident tasks, warmed them, processed 16 unique documents, terminated task 0,
and then sent one unique failure-phase request to every task.

| Check | Audited result |
|---|---:|
| Normal requests correct | 48/48 |
| Unaffected failure-phase requests correct | 9/9 |
| Terminated-task requests failed as expected | 3/3 |
| Cross-task leaks | 0 |
| Successful termination calls | 12/12 |
| Normal response latency | 827.990 ms median; 1,109.528 ms p95 |
| Unaffected failure response latency | 520.824 ms median; 844.998 ms p95 |
| Hosted task start time | 25.867-44.476 s; 35.319 s median |
| Warm-up response time | 1.539-2.414 s; 1.920 s median |

Hosted startup is therefore material and is not hidden by the request-latency
figures. The unchanged upstream fault-isolation `.pipe` was also attempted and
failed after 7.823 seconds with `The service workload was not found`. Hosted
validation had accepted the pipeline during setup, so this is a catalog/runtime
portability issue rather than a successful official benchmark cell.

Billing readback after the passing run reports `JULY2026BENCHMARK` valid at 100
percent off, zero cents due, Starter active, and `cancelAtPeriodEnd=true`. The
passing control consumed 15.3 promotional compute tokens, leaving 9,969.3. Its
audit reports complete evidence and passed control admission, while retaining
`cloud_operational_appendix_unsubmitted` as the official status.
9 changes: 9 additions & 0 deletions benchmark-evidence/baseline/application-shas.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"capturedAt": "2026-07-15T09:26:22.398Z",
"repositories": {
"HomenShum/NodeRoom": "ca25e347dc467bc37f06918e1a18656f7336ee28",
"HomenShum/NodeBenchAI": "6ed0a58eeda993ff2a937ea4bacc2856756dd521",
"HomenShum/NodeSlide": "dd67e4c642c40e6bb414af617a67a31dbed507c5",
"HomenShum/NodeVideo": "bb79bc385de93c90cee89b160fc801d18372d89e"
}
}
30 changes: 30 additions & 0 deletions benchmark-evidence/baseline/concurrent-work/aggregate/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Independent Windows Baseline

Evidence status: **evidence_incomplete**. External status: **independent_unsubmitted_reproduction**.
This is an unchanged local reproduction, not a RocketRide-accepted official result.

## Completeness

| Benchmark | Expected | Observed | Complete |
|---|---:|---:|---|
| fault-isolation | 10 | 10 | true |
| concurrent-processing | 10 | 10 | true |
| data-isolation | 10 | 0 | false |
| authoring-effort | 1 | 1 | true |

## Dimensional Results

- Fault isolation: RocketRide held in 10/10 runs; the in-process LangChain probe lost all work in 10/10 runs.
- RocketRide pool M=8: warm execution median 45.514 s, p95 80.232 s; warm-up median 26.775 s; clean 10/10.
- RocketRide pool M=16: warm execution median 60.821 s, p95 81.444 s; warm-up median 63.655 s; clean 0/10.
- LangChain batch_shared: statuses crash; wall median 0.042 s, p95 0.056 s.
- LangChain abatch_blocking: statuses ok; wall median 7.292 s, p95 7.344 s.
- LangChain seq: statuses ok; wall median 7.435 s, p95 7.612 s.
- Data isolation: RocketRide clean 0/0; lost-doc median None; duplicated/leaked median None.
- Preserved retry signals: {"concurrent-processing": 9, "data-isolation": 5}.

Cold start, warm-up, and warm execution remain separate. Resource fields are reported only where the upstream result emits them.

## Evidence Gaps

- data-isolation: expected 10 results, observed 0
Loading