Add the watcher and the stamper

.github/workflows/checks.yml

@@ -0,0 +1,36 @@
+# The tooling's own tests. Nothing here reads a release file: the release and
+# amendment checks validate what is submitted.
+name: checks
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+  schedule:
+    # The examples job talks to real release hosts. It runs on its own clock,
+    # not in front of a pull request.
+    - cron: "37 6 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  tests:
+    name: tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - run: python3 -m unittest discover -s tools -t tools -v
+
+  examples:
+    name: examples
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      # Re-derives the design repository's hand-stamped examples/ from the real
+      # release hosts and diffs.
+      - env:
+          GITHUB_TOKEN: ${{ github.token }}
+        run: python3 tools/verify_examples.py --check-mirrors

.github/workflows/watcher.yml

@@ -0,0 +1,129 @@
+# Ask every listing's authority host for its releases and stamp every release
+# that has no file under releases/<id>/ yet (RFC 0033).
+#
+# The tick holds no state. What is stamped in the repository is the state, so a
+# dropped run costs latency and not data, and a re-run stamps nothing twice.
+name: watcher
+
+on:
+  schedule:
+    # Ten minutes is a tuning parameter and needs no RFC to change. GitHub's
+    # floor is five. The times sit off the hour, where runs are dropped worst.
+    - cron: "4,14,24,34,44,54 * * * *"
+  workflow_dispatch:
+    inputs:
+      listing:
+        description: "Only this listing id. Empty means every listing."
+        required: false
+      lookback_days:
+        description: "Ignore releases older than this many days. 0 scans the whole list."
+        required: false
+        default: "0"
+      dry_run:
+        description: "Derive everything, write and open nothing."
+        type: boolean
+        default: false
+
+permissions:
+  contents: read
+
+concurrency:
+  # Two ticks writing release files at once would race on the push, and the
+  # scan rule makes a queued tick as good as a parallel one.
+  group: watcher
+  cancel-in-progress: false
+
+jobs:
+  tick:
+    name: tick
+    runs-on: ubuntu-latest
+    steps:
+      # The one identity with access to both halves of the index, and the only
+      # actor the branch protection ruleset lets write here unattended.
+      - uses: actions/create-github-app-token@v2
+        id: app-token
+        with:
+          app-id: ${{ vars.INDEX_APP_ID }}
+          private-key: ${{ secrets.INDEX_APP_PRIVATE_KEY }}
+          owner: ${{ github.repository_owner }}
+          repositories: |
+            content-index-releases
+            content-index
+
+      - uses: actions/checkout@v7
+        with:
+          # Always the tip of main: an old checkout would decide "already
+          # stamped" against an outdated tree and then fail to push.
+          ref: main
+          token: ${{ steps.app-token.outputs.token }}
+
+      # The authored documents the tick reads. A checkout costs one clone
+      # instead of one API request per listing.
+      - uses: actions/checkout@v7
+        with:
+          repository: ${{ github.repository_owner }}/content-index
+          ref: main
+          path: .authored
+          token: ${{ steps.app-token.outputs.token }}
+
+      # Derived cache, never state: the per-listing ETags, the failure counts
+      # and the issue numbers. Losing it costs one expensive tick.
+      - name: Restore the derived cache
+        uses: actions/cache/restore@v4
+        with:
+          path: .watcher
+          key: watcher-${{ github.run_id }}
+          restore-keys: watcher-
+
+      - name: Tick
+        env:
+          GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}
+          LISTING: ${{ inputs.listing }}
+          LOOKBACK_DAYS: ${{ inputs.lookback_days || '0' }}
+          DRY_RUN: ${{ inputs.dry_run }}
+        run: |
+          set -euo pipefail
+          git config user.name "${{ steps.app-token.outputs.app-slug }}[bot]"
+          git config user.email "${{ steps.app-token.outputs.app-slug }}[bot]@users.noreply.github.com"
+
+          arguments=(
+            --authored .authored
+            --authored-repo "${{ github.repository_owner }}/content-index"
+            --cache .watcher/cache.json
+            --lookback-days "$LOOKBACK_DAYS"
+            # The workflow the sweep re-dispatches, and the check whose verdict
+            # it reads. Both belong to KSAModding/content-index#4, which has to
+            # accept a pull_request input and report under this check name.
+            --sweep-workflow checks.yml
+            --verdict-check validate
+          )
+          if [ -n "$LISTING" ]; then
+            arguments+=(--listing "$LISTING")
+          fi
+          if [ "$DRY_RUN" = "true" ]; then
+            arguments+=(--dry-run)
+          fi
+
+          python3 tools/watch.py "${arguments[@]}"
+
+      - name: Push
+        if: ${{ success() && inputs.dry_run != true }}
+        run: |
+          set -euo pipefail
+          for attempt in 1 2 3; do
+            if git push; then
+              exit 0
+            fi
+            # Something else landed on main, the hourly game versions job most
+            # likely. Release files never conflict with it.
+            git pull --rebase --autostash
+          done
+          echo "could not push after three attempts; the next tick rescans and re-stamps"
+          exit 1
+
+      - name: Save the derived cache
+        if: ${{ success() && inputs.dry_run != true }}
+        uses: actions/cache/save@v4
+        with:
+          path: .watcher
+          key: watcher-${{ github.run_id }}

.gitignore

@@ -0,0 +1,8 @@
+# The authored repository, checked out beside this one by the watcher.
+/.authored/
+
+# The watcher's derived cache: ETags, failure counts, issue numbers. Rebuildable
+# from the repository and the hosts, so it is never committed.
+/.watcher/
+
+__pycache__/

README.md

@@ -37,6 +37,34 @@ It is seeded from `Content/Versions/`, the dated history every installed copy of
 
 That poll only ever sees the build that is current when it runs, so a build superseded within the hour can be missing from it. The copy on your own disk stays the complete source.
 
+## The watcher
+
+`.github/workflows/watcher.yml` runs every ten minutes as the org App.
+Each tick asks every listing's authority host for its releases and stamps every release that has no file under `releases/<id>/` yet, so a release published out of version order is stamped too.
+
+There is no queue. What is stamped here is the whole of the watcher's state, which is why a tick GitHub delays, drops or cancels costs latency and not data, and why a re-run stamps nothing twice.
+
+| Tool | What it does |
+|---|---|
+| `tools/stamp_release.py` | Authored document plus release archive in, release file out. The one place a release file is derived, shared with the release pull request checks so the two paths cannot disagree. Needs no token. |
+| `tools/hosts.py` | The release hosts, GitHub and SpaceDock, behind one interface. GitHub is polled conditionally against a stored ETag, so an unchanged listing costs no rate limit at all. |
+| `tools/watch.py` | One tick: scan, stamp, commit, append a mirror that only appeared later, keep one error issue per listing current on the authored repository, and sweep its open pull requests. |
+| `tools/verify_examples.py` | Re-derives the design repository's hand-stamped `examples/` from their release hosts and diffs. |
+
+A release the watcher cannot stamp, a tag that does not parse or an archive whose install root is neither derivable nor authored, becomes one open issue per listing on the authored repository, kept current rather than reopened every tick.
+
+A version is stamped exactly once. A tag that reappears with different bytes is rejected and never overwritten, and both hashes are named in that issue.
+
+`download.mirrors` is the one field the watcher may append to after publish, and only after downloading the other host's archive and finding it byte-identical.
+
+An authored `game_max` naming a month that was still running at stamp time is stamped with no upper bound, and a later tick resolves and adds the bound once the month completes.
+
+To run a tick by hand, dispatch the workflow: `listing` narrows it to one id, and `dry_run` derives everything and writes nothing. Locally, against a checkout of the authored half:
+
+```text
+python3 tools/watch.py --authored ../content-index --dry-run
+```
+
 ## A published release is immutable
 
 Identity, the version, the download and the install data never change.