From 8cd6cb821a61126a01fca1d5d04e5465b2673a56 Mon Sep 17 00:00:00 2001 From: Cristian Girlea Date: Thu, 15 Oct 2026 11:55:38 +0300 Subject: [PATCH] test: assert every action is hash-pinned and no checkout persists credentials docs/security.md promised that the checkout step is pinned by commit hash and does not persist credentials, but nothing enforced either. test_workflows.py now reads every `uses:` line of both workflows and requires `owner/repo@<40 hex>`, optionally followed by a `# vX.Y.Z` comment, and reads every step under a `steps:` list (indentation kept, `name:`-first and quoted forms included) to require `persist-credentials: false` in the `with:` block of each `actions/checkout` step. Two counts keep the readers honest: every spelling of a `uses` key YAML accepts is counted against the lines the strict pattern read, and the steps that carry an action against the `uses:` lines, so a step the readers did not see fails the test instead of slipping past it. Seen failing on tests.yml with a tag instead of a hash, a checkout without a `with:` block, `persist-credentials: true`, and a short hash with a bare comment; a `name:`-first, quoted, pinned checkout with the flag passes. The same shapes, with a two-job sample, are kept as a fixture test so the readers cannot loosen unnoticed. docs/security.md says the test enforces the promise; docs/testing.md counts 142 Python tests. Closes #34 --- docs/security.md | 2 +- docs/testing.md | 2 +- tests/python/test_workflows.py | 128 ++++++++++++++++++++++++++++++++- 3 files changed, 129 insertions(+), 3 deletions(-) diff --git a/docs/security.md b/docs/security.md index 8e8ab1f..f8071c3 100644 --- a/docs/security.md +++ b/docs/security.md @@ -7,7 +7,7 @@ This experimental service is designed for trusted application-to-service traffic - Only trusted task implementations are allowed. Python child processes inherit the service environment; they are not suitable for arbitrary user-provided scripts. - Container configuration uses a non-root user, read-only filesystem, dropped capabilities, no-new-privileges, CPU/memory/PID bounds, and an internal-only network for execution. It mounts only this checkout and the optional dependency/model cache, never a Docker socket or home directory. - Dependency acquisition is separate and internet-enabled. Execution installs hash-checked wheels from the local cache without internet access. The model downloader retrieves fixed files at a pinned revision and checks their hashes; it does not execute model-repository code. -- CI runs on GitHub's hosted runners only, because the repository is public. GitHub runs a pull request's workflows as the pull request has them, so a fork's pull request executes its own code, workflow edits included; on a disposable runner of GitHub's that holds nothing of the maintainer's, that is the intended boundary. No workflow names a self-hosted runner, and `tests/python/test_workflows.py` fails if one does again; a fork can edit that test along with the workflows, so while any self-hosted runner is still registered for the repository, keep requiring approval for outside contributors' workflow runs, and unregister it so that there is none a pull request could route its code onto. The `actions/checkout` step is pinned by commit hash and does not persist credentials. +- CI runs on GitHub's hosted runners only, because the repository is public. GitHub runs a pull request's workflows as the pull request has them, so a fork's pull request executes its own code, workflow edits included; on a disposable runner of GitHub's that holds nothing of the maintainer's, that is the intended boundary. No workflow names a self-hosted runner, and `tests/python/test_workflows.py` fails if one does again; a fork can edit that test along with the workflows, so while any self-hosted runner is still registered for the repository, keep requiring approval for outside contributors' workflow runs, and unregister it so that there is none a pull request could route its code onto. Every action a workflow uses is pinned by commit hash, and the `actions/checkout` step does not persist credentials; the same test fails if either changes. - Model files, native libraries and runtime dependencies still carry supply-chain risk. Review upgrades, regenerate locks deliberately and rerun tests before changing pins. - Requests and results may contain sensitive documents. They live in RAM, may be present in process/core dumps, and are not encrypted by this service. Minimize retention and use appropriate host/storage policies. Embedding vectors are derived from their input text and can reveal much of it; store and transmit them with the same care as the text. - Redaction is assistive, not a compliance control. The rules find only the structured identifiers they recognise, and the optional named-entity models were trained on English news, so their recall on names from other domains or languages is lower and unmeasured here; review redacted output before treating it as free of personal data. The model files are third-party ONNX conversions of the original weights, pinned by revision and hash but not signed by the original author. diff --git a/docs/testing.md b/docs/testing.md index 828a647..0d6adfd 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -48,7 +48,7 @@ PHP branch support was verified locally on 2026-09-17 on PHP 8.2.33, 8.3.33, 8.4 | Layer | Scope and counting | | --- | --- | -| Python worker | 139 tests: lifecycle, validation and HTTP, process crashes, cancellation, timeouts, retention, allocation/exit races, redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample and scoring, the workflows' runners, both Claude Code hooks (the post-edit rules, and the pre-call push guard through its own cases), progress bounds, and the model-name table every result reports from | +| Python worker | 142 tests: lifecycle, validation and HTTP, process crashes, cancellation, timeouts, retention, allocation/exit races, redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample and scoring, the workflows' runners, action pins and checkout credentials, both Claude Code hooks (the post-edit rules, and the pre-call push guard through its own cases), progress bounds, and the model-name table every result reports from | | MCP server | 119 tests, offline with hash-pinned wheels: the protocol client against an in-process worker, every tool through the SDK's in-memory client with test tasks enabled, the entry point as a subprocess, result validators on synthetic data, the index builder (chunking, the format and each corruption it refuses, building against the worker, caching, its command line) `bridge_search` (the model gate, path confinement, reloads, rerank on and off), the reachability of every answer in the LLM evaluation, that a default search sees only part of its corpus and that its two-step question takes two steps, that the demo worker reaches none of the semantic evaluation's answers, that a tool waits out a full worker, before each of a search's jobs and within one budget, and says so when it stays full, the evaluation runner's command, isolation checks, scoring and per-backend launcher and index check without starting `claude`, a check that the server's model names match the worker's, that every path parameter says where it resolves and every position in a result says what it counts, and a check that every timing claim in a tool description is a measured median | | PHP contracts + HTTP | 134 checks per PHP version; this **includes** the 101 contract checks, not 134 + 101 | | FrankenPHP | 12 jobs submitted by six concurrent test threads through a reused PHP worker, a 512-document `top_k` request, embedding and redaction round trips, and invalid-input, missing-job and multibyte boundary checks | diff --git a/tests/python/test_workflows.py b/tests/python/test_workflows.py index b3354c5..a7ab4ff 100644 --- a/tests/python/test_workflows.py +++ b/tests/python/test_workflows.py @@ -3,9 +3,11 @@ The repository is public: GitHub does not meter Actions minutes for it, and a fork's pull request runs its own code on a disposable runner of GitHub's, never on a machine of the maintainer's. Nothing here installs Docker Compose, since GitHub's images ship the plugin. Read as text, since the standard library has no YAML parser; the workflows -keep their jobs two spaces in under `jobs:`. +keep their jobs two spaces in under `jobs:`. Every action a step uses is pinned to a commit, and no checkout +leaves the job's token behind: `docs/security.md` promises both. """ +import itertools import re import unittest from pathlib import Path @@ -13,6 +15,13 @@ REPO = Path(__file__).resolve().parents[2] WORKFLOWS = sorted(path for ext in ("*.yml", "*.yaml") for path in (REPO / ".github" / "workflows").glob(ext)) RUNS_ON = "runs-on: ubuntu-latest" +# A step's `uses:` may follow its `name:` (so the dash is optional) and may quote its value. +USES = re.compile(r"^\s*(?:-\s+)?uses:\s*(.*?)\s*$") +# Every spelling of the key YAML accepts (quoted, a space before the colon, a flow-style step), so that a line +# USES cannot read is counted, not passed over. +ANY_USES = re.compile(r"""^\s*(?:-\s+)?(?:\{\s*)?['"]?uses['"]?\s*:""") +# `owner/repo@`, and the version that commit is, as a comment, for the human reading the workflow. +PINNED = re.compile(r"""^['"]?[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9_.-]+@[0-9a-f]{40}['"]?(?: +# v\d+\.\d+\.\d+)?$""") def jobs(text: str) -> dict: @@ -30,6 +39,52 @@ def jobs(text: str) -> dict: return found +def steps(text: str) -> list: + """Every step of every job, in file order, each the list of its lines with their indentation kept. + + A step is an item of a `steps:` list: its `- ` line and the lines below it until the next item at the same + indentation, or the end of the list. + """ + found, key_indent, item_indent = [], None, None + for line in text.splitlines(): + key = re.match(r"^(\s*)steps:\s*$", line) + if key: + key_indent, item_indent = len(key.group(1)), None + continue + if key_indent is None or not line.strip() or line.lstrip().startswith("#"): + continue + indent = len(line) - len(line.lstrip()) + if indent <= key_indent: + key_indent = None + elif line.lstrip().startswith("- ") and item_indent in (None, indent): + item_indent = indent + found.append([line]) + else: + found[-1].append(line) + return found + + +def action(step: list) -> str: + """The value of a step's `uses:` line as written, or an empty string for a step that runs a command.""" + return next((USES.match(line).group(1) for line in step if USES.match(line)), "") + + +def is_checkout(step: list) -> bool: + """Whether the step runs actions/checkout; GitHub reads owner and repository names case-insensitively.""" + return action(step).lstrip("'\"").lower().startswith("actions/checkout@") + + +def inputs(step: list) -> list: + """The stripped lines of a step's `with:` block.""" + lines = iter(step) + for line in lines: + key = re.match(r"^(\s*)with:\s*$", line) + if key: + deeper = itertools.takewhile(lambda below: len(below) - len(below.lstrip()) > len(key.group(1)), lines) + return [below.strip() for below in deeper] + return [] + + class WorkflowTests(unittest.TestCase): def test_the_workflows_are_found(self): self.assertEqual([path.name for path in WORKFLOWS], ["model-smoke.yml", "tests.yml"]) @@ -76,6 +131,77 @@ def test_every_job_that_runs_compose_checks_the_repository_out_first(self): checkout = next(i for i, line in enumerate(lines) if line.startswith("- uses: actions/checkout@")) self.assertLess(checkout, commands[0]) + def test_every_action_is_pinned_to_a_commit(self): + # A tag or a branch can be moved after the review that admitted it; a commit cannot. The version comment + # beside the hash is what a human reads, and what the next bump is compared against. + for path in WORKFLOWS: + text = path.read_text(encoding="utf-8") + lines = text.splitlines() + used = [USES.match(line).group(1) for line in lines if USES.match(line)] + self.assertTrue(used, path.name) + self.assertEqual(len(used), len([line for line in lines if ANY_USES.match(line)]), path.name) + for value in used: + with self.subTest(workflow=path.name, uses=value): + self.assertRegex(value, PINNED) + + def test_every_checkout_step_leaves_no_credentials_behind(self): + # By default the checkout writes the job's token into .git/config, where every later step can read it. + for path in WORKFLOWS: + text = path.read_text(encoding="utf-8") + found = steps(text) + with self.subTest(workflow=path.name): + # The step parser saw every `uses:` line, so no checkout step is missing from the check below. + seen = len([step for step in found if action(step)]) + self.assertEqual(seen, len([line for line in text.splitlines() if USES.match(line)])) + checkouts = [step for step in found if is_checkout(step)] + self.assertTrue(checkouts, path.name) + for step in checkouts: + with self.subTest(workflow=path.name, step=step[0].strip()): + self.assertIn("persist-credentials: false", inputs(step)) + + def test_the_pin_and_step_readers_on_the_shapes_they_must_tell_apart(self): + sha = "3d3c42e5aac5ba805825da76410c181273ba90b1" + for value in (f"actions/checkout@{sha}", f"'actions/checkout@{sha}' # v7.0.1", + f"owner-1/re.po_1@{sha} # v10.2.33"): + self.assertRegex(value, PINNED) + for value in ("actions/checkout@v7.0.1", "actions/checkout@3d3c42e # v7", f"actions/checkout@{sha} # v7", + f"actions/checkout@{sha} # v7.0.1 node24", f"./local@{sha}", f"docker://python@{sha}"): + self.assertNotRegex(value, PINNED) + text = f"""jobs: + one: + steps: + # a comment at the items' indentation + - name: Check out + uses: 'Actions/Checkout@{sha}' # v7.0.1 + with: + fetch-depth: 1 + persist-credentials: false + - run: | + echo one + + echo two + - uses: actions/checkout@{sha} + with: + persist-credentials: true + two: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@{sha} + env: + persist-credentials: false +""" + found = steps(text) + checkout = f"- uses: actions/checkout@{sha}" + self.assertEqual([step[0].strip() for step in found], ["- name: Check out", "- run: |", checkout, checkout]) + self.assertEqual([is_checkout(step) for step in found], [True, False, True, True]) + self.assertEqual(inputs(found[0]), ["fetch-depth: 1", "persist-credentials: false"]) + self.assertEqual(inputs(found[2]), ["persist-credentials: true"]) + self.assertEqual(inputs(found[3]), []) + self.assertEqual(len([line for line in text.splitlines() if ANY_USES.match(line)]), 3) + for line in ("- { uses: actions/checkout@v4 }", '- "uses": actions/checkout@v4', "uses : actions/checkout@v4"): + self.assertTrue(ANY_USES.match(line), line) + self.assertIsNone(USES.match(line), line) + def test_tests_run_for_pull_requests_and_for_pushes_to_main_only(self): # A push to a pull request's branch would otherwise run every job twice. text = (REPO / ".github" / "workflows" / "tests.yml").read_text(encoding="utf-8")