Skip to content

Add independent reproduction and Node application study#5

Open
HomenShum wants to merge 11 commits into
rocketride-org:mainfrom
HomenShum:codex/rocketride-node-study
Open

Add independent reproduction and Node application study#5
HomenShum wants to merge 11 commits into
rocketride-org:mainfrom
HomenShum:codex/rocketride-node-study

Conversation

@HomenShum

@HomenShum HomenShum commented Jul 15, 2026

Copy link
Copy Markdown

Summary

  • reproduces the frozen upstream benchmark at 43be41acb58558dfae8e2e3deb86d8a00cb1b1c8 without changing its source, fixtures, pools, retries, or timeouts
  • preserves the complete supported-Linux run, incomplete Windows appendix, failed attempts, environment records, logs, traces, manifests, and dimensional reports
  • adds a separately pre-registered Node application V1 study and preserves its unfavorable protocol result
  • adds a second, separately pre-registered resident-runtime V2 study bound to merged production app commits and resolved source-definition digests
  • includes independent app-verifier, post-run audit, deployment, cost, and source-integrity receipts

Findings

Track A: unchanged upstream reproduction

This remains an independent, unsubmitted reproduction. On supported Linux, RocketRide held fault isolation in 10/10 runs and completed the concurrency and data-isolation lanes cleanly. The retained Windows appendix includes unfavorable RocketRide correctness and latency results and remains evidence-incomplete.

Track B V1: application study

V1 completed 30 request cells, 90 result envelopes, and 30 RocketRide traces. Its completion/isolation scorecard passed, but the independent protocol audit found all 30 RocketRide results exceeded the pre-registered 10,000 ms request deadline because pool readiness was included in every request. Protocol admission failed and V1 remains promotion-blocked.

Track B V2: resident-runtime study

V2 was pre-registered in commit 3d1848f before implementation and does not replace V1. It keeps the same five candidates and 10,000 ms request deadline, adds a fixed 2,000 ms per-unit failure-detection bound, and binds each request/result to a resolved definition containing the merged app commit plus protocol, adapter, domain-tool, validator, and fixture hashes.

V2 completed 30 request cells, 90 result envelopes, 30 RocketRide traces, and 20 pool lifecycle receipts. The independent audit reports:

evidenceStatus: complete
protocolAdmissionStatus: passed
promotionStatus: eligible_for_external_submission
officialStatus: separate_application_study_unsubmitted

RocketRide normal request medians ranged from 1,043.073 to 1,090.228 ms. Hard-failure medians ranged from 3,008.731 to 3,025.511 ms; all three unaffected units completed and the engine remained healthy in every fault cell. There were zero request-deadline overruns.

Standing cost remains material: engine cold start was 1,569.703 ms, normal pool warmup median was 7,721.659 ms, and fault-replacement pool warmup median was 10,109.896 ms. These are reported separately and V2 makes no raw speed-winner claim.

Production proof

The governed execution-port PRs are merged into NodeRoom, NodeBenchAI, NodeSlide, and NodeVideo. All post-merge checks passed, all four exact main revisions are deployed to Vercel production, and all canonical URLs returned HTTP 200. Deployment IDs and immutable URLs are in benchmark-evidence/extensions/node-suite/production-deployment-v2.json.

Boundaries

  • This PR does not claim an official or accepted RocketRide result; maintainer acknowledgment is still required.
  • V2 is a new production-style topology study and never supersedes V1's failed result.
  • Node application candidates never receive mutation authority; app validation, CAS, proposals, review, and final writes remain app-owned.
  • The deterministic extension does not measure model quality.
  • No paid model or RocketRide Cloud API was called. Model and cloud benchmark spend are both USD 0.

Verification

  • supported-Linux summary recomputed with no manifest issues
  • original 518-artifact evidence bundle hash-verified
  • 13 V1/V2 harness and audit tests passed
  • V2: 30 requests, 90 results, 30 traces, 30 aggregate rows, 20 pool receipts
  • V2 manifest: 407 artifacts, all hashes and byte counts verified
  • V2 audit: evidence complete, protocol admission passed, eligible for external submission
  • all four merged-main app verifier receipts and deployment checks passed

@HomenShum
HomenShum marked this pull request as ready for review July 15, 2026 17:50
@HomenShum

Copy link
Copy Markdown
Author

The separately pre-registered resident-runtime V2 evidence is now complete and this PR is ready for maintainer review.

V1 remains preserved and protocol-blocked by 30/30 RocketRide deadline overruns. V2 does not replace it: V2 has 30 requests, 90 results, 30 RocketRide traces, zero deadline overruns, complete manifest/source/definition audit, and local status eligible_for_external_submission. Standing and recovery pool costs remain reported prominently, and no official/accepted claim is made.

All four governed app ports are merged and deployed. The PR head is also published on the fork's main; upstream merge still requires a RocketRide maintainer.

@HomenShum

Copy link
Copy Markdown
Author

RocketRide Cloud benchmark update

I redeemed JULY2026BENCHMARK and completed a separately pre-registered hosted appendix. The first gated run is retained as failed; the corrected run and independent audit are in head commit af7d3fe.

What passed

  • 3 repetitions, 4 resident Cloud tasks per repetition
  • 48/48 normal requests correct
  • 9/9 unaffected sibling requests correct after task 0 termination
  • 3/3 requests to the terminated task failed as expected
  • 0 cross-task marker leaks
  • 12/12 task termination calls succeeded
  • no model provider calls; USD 0 model cost and USD 0 checkout charge
  • passing run used 15.3 promotional compute tokens
  • Starter is active; promo validation reports 100% off and zero due; refreshed UI says Ends at period end

Where RocketRide lost / blocked

  • The unchanged upstream fault-isolation pipeline is not runnable on Cloud: execution fails after 7.823 s with The service workload was not found. Cloud validation did not surface this during setup.
  • Hosted task startup was 25.867-44.476 s (35.319 s median), before warm-up. Normal request latency was 827.990 ms median / 1,109.528 ms p95.
  • A Development-scoped key configured with control/data/monitor permissions returned Permission 'task.control' denied; an all-team key worked.
  • SDK 1.2.0 billing types do not match current Cloud payloads: documented organizations vs returned organization, and documented label/cents/priceId vs returned nickname/amountCents/stripePriceId.
  • Scheduling cancellation returned Internal error: 'str' object has no attribute 'tzinfo', although API readback and refreshed UI both confirm end-of-period cancellation.

Receipts:

This is labeled cloud_operational_appendix_unsubmitted, not an official score. The exact upstream Cloud lane remains blocked unless the benchmark's workload provider can be provisioned in the hosted catalog.

@HomenShum

Copy link
Copy Markdown
Author

Clarification on the Cloud finding

The repository README defines the requested benchmark reproduction against the pinned local RocketRide runtime (make provision, make start, make smoke, then the 10x runs). That unchanged supported-Linux suite did complete and is the primary submission evidence.

The JULY2026BENCHMARK Cloud work above is an additional operational appendix because the invitation said the promo could be used for the benchmark or exploration. The hosted catalog's missing custom workload service is therefore not a blocker to the requested local suite or this submission. It is only a Cloud portability/setup finding. The remaining official step is RocketRide maintainer acceptance/publication of the submitted aggregate results.

Primary unchanged-suite evidence: supported-Linux aggregate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant