Skip to content

ci: run execution spec tests - #662

Merged
Qumeric merged 13 commits into
mainfrom
codex/execution-spec-tests
Jul 28, 2026
Merged

ci: run execution spec tests#662
Qumeric merged 13 commits into
mainfrom
codex/execution-spec-tests

Conversation

@Qumeric

@Qumeric Qumeric commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Run the pinned tests-zkevm@v0.4.1 execution-spec suite on every pull request and on demand.

Key decisions:

  • Run on every PR instead of a cron because the cached job completes in roughly three minutes and catches regressions before merge.
  • Pin the fixture release and checksum, and fail if the standalone test lockfile drifts from the workspace OpenVM revision.
  • Treat the current Reth divergences as an exact baseline: both new failures and unexpectedly resolved divergences fail the test.
  • Use host k256 implementations in native CI because OpenVM k256 relies on guest-only intrinsics; preserve the existing guest recovery paths because sharing them through a helper regressed metered guest instructions.

Closes INT-8714

@socket-security

socket-security Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​serde_json@​1.0.1508210093100100
Addedcargo/​rayon@​1.12.09910093100100

View full report

@socket-security

socket-security Bot commented Jul 15, 2026

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn High
Obfuscated code: cargo writeable is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: ?cargo/writeable@0.6.3

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore cargo/writeable@0.6.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: cargo zerocopy is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: ?cargo/alloy-eips@1.5.2cargo/alloy-trie@0.9.4cargo/reqwest@0.13.3cargo/metrics@0.24.6cargo/alloy-primitives@1.6.0cargo/alloy-json-rpc@2.0.5cargo/alloy-rpc-types@2.0.5cargo/alloy-eips@2.0.5cargo/alloy-rpc-client@2.0.5cargo/alloy-consensus@2.0.5cargo/alloy-provider@2.0.5cargo/alloy@2.0.5cargo/revm@40.0.3cargo/reth-primitives-traits@0.4.1cargo/reqwest@0.12.20cargo/ark-bls12-381@0.5.0cargo/ark-bn254@0.5.0cargo/ark-ec@0.5.0cargo/ark-ff@0.5.0cargo/aurora-engine-modexp@1.2.0cargo/alloy-trie@0.8.1cargo/metrics@0.23.1cargo/revm@27.1.0cargo/bls12_381@0.8.0cargo/actix-web@4.12.1cargo/zerocopy@0.8.54

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore cargo/zerocopy@0.8.54. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@Qumeric
Qumeric force-pushed the codex/execution-spec-tests branch from 248b0dd to fd9ef50 Compare July 15, 2026 15:40
@Qumeric
Qumeric requested a review from shuklaayush July 22, 2026 09:06
@Qumeric
Qumeric marked this pull request as ready for review July 22, 2026 11:09

@shuklaayush shuklaayush left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! this should be really useful
one suggestion is to adopt a custom test harness like datatest-stable so individual fixtures appear as separate test cases. not sure if it's possible here but worth a try

Comment thread .github/workflows/execution-spec-tests.yml Outdated
Comment thread .github/workflows/execution-spec-tests.yml
Comment thread crates/revm-crypto/src/k256.rs Outdated
@Qumeric
Qumeric force-pushed the codex/execution-spec-tests branch 2 times, most recently from 7330925 to 4c60143 Compare July 22, 2026 14:05
@Qumeric

Qumeric commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

dataset-stable is a good suggestion but I don't think it's a fit for a huge amount of very small test -- it starts a separate nextest process for each test. We can do custom test harness but it seems like an overkill, so I would keep it as it is

@Qumeric
Qumeric requested a review from shuklaayush July 28, 2026 08:31
@Qumeric
Qumeric merged commit abb0585 into main Jul 28, 2026
8 checks passed
@Qumeric
Qumeric deleted the codex/execution-spec-tests branch July 28, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants