Skip to content

Fail fast on invalid benchmark output - #183

Merged
silverstein merged 1 commit into
masterfrom
codex/olmocr-output-preflight-20260903
Sep 4, 2026
Merged

Fail fast on invalid benchmark output#183
silverstein merged 1 commit into
masterfrom
codex/olmocr-output-preflight-20260903

Conversation

@silverstein

@silverstein silverstein commented Sep 3, 2026

Copy link
Copy Markdown
Member

Summary

  • validate the benchmark output path before reading the corpus or launching PDF Tools
  • reject missing or unsafe output parents and occupied targets immediately
  • repeat the no-overwrite checks during final atomic publication
  • make the layout determinism test reuse one exact PDF byte source rather than compare time-varying generated files

Regression coverage

  • proves an unavailable parent processes zero documents
  • proves an occupied output processes zero documents and remains untouched
  • reproduces and closes the CI-only fake-PDF hash flake under Node 20.19
  • affected suites pass on Linux under Node 20.19 and 22, and on macOS under the project Node 22 runtime

Scope

The product change is limited to the offline olmOCR benchmark runner. The additional test-only correction makes an existing layout determinism assertion compare two runs over the same source bytes. Shipped PDF behavior and benchmark claim policy are unchanged.

@silverstein
silverstein force-pushed the codex/olmocr-output-preflight-20260903 branch from 34b3f4f to 6e345b0 Compare September 3, 2026 22:32
@silverstein
silverstein merged commit 2da0623 into master Sep 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant