Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/scripts/test_determinism_workflow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
#
# Copyright 2026 The Ethos maintainers
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from __future__ import annotations

import re
import unittest
from pathlib import Path


ROOT = Path(__file__).resolve().parents[2]
WORKFLOW = ROOT / ".github/workflows/determinism.yml"
REQUIRED_MATRIX_OS = {"macos-14", "ubuntu-latest", "windows-latest"}


def workflow_text() -> str:
return WORKFLOW.read_text(encoding="utf-8")


def matrix_os_values(text: str) -> set[str]:
match = re.search(r"^\s*os:\s*\[([^\]]+)\]\s*$", text, flags=re.MULTILINE)
if match is None:
return set()
return {item.strip() for item in match.group(1).split(",")}


class DeterminismWorkflowTests(unittest.TestCase):
def test_matrix_includes_gate_zero_platforms_and_windows_preflight(self) -> None:
self.assertEqual(matrix_os_values(workflow_text()), REQUIRED_MATRIX_OS)

def test_windows_is_no_longer_left_as_a_todo(self) -> None:
text = workflow_text()

self.assertIn("Windows x64 runs a Milestone B", text)
self.assertNotIn("TODO", text)
self.assertNotIn("add windows-latest", text)

def test_contract_vectors_run_on_every_matrix_os(self) -> None:
text = workflow_text()

self.assertIn("runs-on: ${{ matrix.os }}", text)
self.assertIn("cargo test --locked -p ethos-core --all-features", text)

def test_pdfium_corpus_step_remains_explicitly_configured(self) -> None:
text = workflow_text()

self.assertIn("full-corpus fingerprint equality", text)
self.assertIn("shell: bash", text)
self.assertIn("ETHOS_PDFIUM_LIBRARY_PATH", text)
self.assertIn("skipped: pinned PDFium runtime is not configured", text)

def test_matrix_does_not_fail_fast(self) -> None:
self.assertIn("fail-fast: false", workflow_text())


if __name__ == "__main__":
unittest.main()
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ jobs:
run: python3 .github/scripts/test_readiness_gate.py
- name: Gate Zero evidence preflight tests
run: python3 .github/scripts/test_gate_zero_evidence_preflight.py
- name: determinism workflow tests
run: python3 .github/scripts/test_determinism_workflow.py
- name: Gate Zero harness tests
run: python3 benchmarks/harness/test_run_gate_zero.py
- name: same-platform double-parse byte-diff
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/determinism.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# determinism.yml — nightly + contract-change PRs (plan §9).
# Gate Zero platforms first (macOS arm64 + Linux x64); Windows x64 joins no later than
# Milestone B exit (week 8). Unresolved Windows divergence blocks or re-scopes Public Beta.
# Gate Zero platforms first (macOS arm64 + Linux x64); Windows x64 runs a Milestone B
# preflight lane. Unresolved Windows divergence blocks or re-scopes Public Beta.
# Determinism failures are NEVER retried into green — a flaky fingerprint IS the bug.
name: determinism

Expand All @@ -19,9 +19,10 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [macos-14, ubuntu-latest] # macOS arm64 + Linux x64 (Gate Zero platforms)
# TODO(week 8 / Milestone B exit): add windows-latest (risk R9; release-blocking
# for Public Beta if divergent)
# macOS arm64 + Linux x64 are the Gate Zero platforms. Windows x64 is a
# Milestone B preflight lane and only runs PDFium-backed corpus work when
# the pinned runtime is explicitly configured on that runner.
os: [macos-14, ubuntu-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
Expand Down
2 changes: 2 additions & 0 deletions docs/execution-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ The committed implementation now includes:
- A real PDFium-backed born-digital PDF extraction path in `ethos-pdf`, loaded only from an explicit `ETHOS_PDFIUM_LIBRARY_PATH`.
- A pinned Phase 1 PDFium profile in `docs/pdfium-profile.md` and `profiles/ethos-deterministic-v1.json`: `chromium/7881`, V8/XFA disabled, platform artifact hashes, runtime library hashes, and provenance are recorded.
- Runtime checks that reject missing or mismatched PDFium versions, release artifacts, and extracted libraries with stable errors before dynamic loading.
- The determinism workflow includes a Windows x64 preflight lane for core c14n/profile/fingerprint contract tests, while PDFium-backed corpus work remains explicitly skipped unless the pinned runtime is configured on that runner. A static workflow test guards that matrix wiring.
- `ethos doc parse` / `ethos fingerprint` PDF execution through a worker process with `max_parse_ms` timeout enforcement, stable error-envelope relay, diagnostics-gated worker stderr, and page-range validation/filtering.
- Quantized page/span extraction at the backend boundary, plus a basic deterministic layout pass that assembles paragraph `text_block` elements, fixture-backed alpha heading and flat list-item elements, and simple column reading order for the current born-digital fixtures. Fixture validation binds selected `fixture.json` expectations to committed extraction/layout goldens and binds current alpha text/Markdown exports to committed layout output so current read-order, element-type, heading-export, list-item, and export cases fail closed on drift.
- An internal layout evaluator scaffold exists at `fixtures/evaluate_layout_alpha.py` and `make layout-evaluator-alpha`. It reads committed `fixture.json` and `layout.json` files, summarizes alpha element-type and subset coverage, and fails closed on missing layout expectations or drift in fixture-backed reading order / heading / list-item cases.
Expand Down Expand Up @@ -58,6 +59,7 @@ Milestone A has an accepted internal Gate Zero decision for roadmap control, so
| Schema/example validation | Landed: schemas, examples, deterministic profile, referential integrity, and bbox sanity pass the `jsonschema` validation gate | Contract changes still require explicit versioning and compatibility review |
| Trust-layer implementation | Landed: `ethos verify` quote/value/presence/table-cell checks, explicit quote-containment labeling, normalized equality for value/table-cell checks, stale and unverifiable fingerprint handling, unsupported claim reporting, structured capability limits, native Ethos JSON path, ODL-style adapter path with synthetic table/cell mapping, pinned real OpenDataLoader 2.4.7 grounded/ungrounded fixtures, foreign fixture manifest hash validation, crop-ref evidence plumbing, stable logical native crop refs, native crop descriptor artifacts, raw BGRA crop rendering in `ethos-pdf`, CLI PNG crop artifact production for bound native source PDFs, same-host rendered crop repeatability check, rendered-crop run comparison helper, strict citation/config input validation, citation input schema, split-quote fixture coverage, explicit unsupported non-v1 claim reporting, OpenDataLoader-style structure diagnostics for malformed bbox and unknown-page references, verify-alpha case inventory checks, and demo fixtures | Still needed: real OpenDataLoader table-cell grounding, additional adapter hardening against broader real output shapes, future claim-kind expansion outside the current v1 alpha policy, and a decision on whether cross-platform rendered crop artifact equality is worth pursuing after the current macOS/Linux bbox drift finding |
| WS-HARNESS readiness | Partially landed: readiness path is green for frozen corpus/hardware and pinned competitors, Gate Zero evidence preflight validates the current `ethos-bench` handoff, and gates fail closed if those records regress | Public-safe comparison report flow, release/package approval, claim-wording approval, and future evidence-refresh workflow still need hardening |
| Determinism workflow | Landed: macOS arm64, Linux x64, and Windows x64 matrix entries run core contract tests; PDFium-backed corpus work stays gated on an explicitly configured pinned runtime; static workflow tests guard the matrix | Windows PDFium runtime provisioning and broader cross-platform corpus validation remain future work |

## PM Rule

Expand Down
Loading