Scheduled snapshot refresh, offline tests, README kept in step

.github/workflows/refresh-snapshots.yml

@@ -0,0 +1,140 @@
+name: refresh-snapshots
+
+# Reruns the backfill scripts, rewrites src/data/*.json and the README figures, runs the same checks as CI
+# (type check and every *.check.ts) and opens a pull request with the result. It never pushes to main.
+# No secrets: every source is public and keyless (Tenderly's public gateway, DefiLlama, FRED, Blockscout, public RPCs).
+# Needs the repository setting Settings > Actions > General > "Allow GitHub Actions to create and approve pull requests".
+
+on:
+  schedule:
+    - cron: '0 6 * * 1' # Mondays 06:00 UTC, after the previous week has closed
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+  issues: write
+
+concurrency: refresh-snapshots
+
+env:
+  BRANCH: data/refresh-snapshots
+
+jobs:
+  refresh:
+    if: github.repository == 'RahilBhavan/Ondo'
+    runs-on: ubuntu-latest
+    timeout-minutes: 180
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 20
+          cache: npm
+      - run: npm ci
+
+      # About 25 minutes; the dense USDC, PYUSD and BUIDL series can take 40 and the public gateway may answer 429.
+      # The script writes its file only when every asset finished, so a failed attempt leaves the data as it was.
+      - name: backfill stablecoin event logs (src/data/issuance.json)
+        timeout-minutes: 100
+        run: node scripts/backfill-issuance.mjs || { echo "first attempt failed, retrying once"; node scripts/backfill-issuance.mjs; }
+
+      - name: backfill DefiLlama supply and FRED rates (src/data/supply-by-chain.json, rates.json)
+        timeout-minutes: 10
+        run: node scripts/backfill-context.mjs
+
+      # Keeps only parts that came back fully live. If a public source is down it exits 1 and leaves that part as it was,
+      # so this step may fail without blocking the rest; the pull request says which.
+      - name: refresh last good values (src/data/last-good.json)
+        id: lastgood
+        continue-on-error: true
+        timeout-minutes: 10
+        run: npx tsx scripts/snapshot-last-good.ts
+
+      - name: rewrite README figures and the study guide from the new snapshots
+        run: |
+          npm run sync:readme
+          npx tsx docs/textbook/build.ts
+
+      # A failing check must not hide the new data: the pull request still opens, as a draft, and the job then fails.
+      - name: type check
+        id: tsc
+        continue-on-error: true
+        run: npx tsc --noEmit
+      - name: self-checks (offline)
+        id: tests
+        continue-on-error: true
+        run: npm test
+
+      - name: open or update the pull request
+        env:
+          GH_TOKEN: ${{ github.token }}
+          LASTGOOD_OUTCOME: ${{ steps.lastgood.outcome }}
+          CHECKS_OUTCOME: ${{ steps.tsc.outcome == 'success' && steps.tests.outcome == 'success' && 'success' || 'failure' }}
+        run: |
+          if [ -z "$(git status --porcelain -- src/data README.md docs/textbook)" ]; then
+            echo "no change to the snapshots"
+            exit 0
+          fi
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git checkout -B "$BRANCH"
+          git add src/data README.md docs/textbook/index.html
+          git commit -m "data: refresh snapshots $(date -u +%F)"
+          git push --force origin "$BRANCH"
+
+          run_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+          {
+            echo "## Summary"
+            echo
+            echo "Scheduled refresh of the stablecoin flows snapshots ($run_url)."
+            echo
+            echo '```'
+            git diff --stat HEAD~1 HEAD
+            echo '```'
+            echo
+            if [ "$CHECKS_OUTCOME" = "success" ]; then
+              echo "- Type check and every \`*.check.ts\` passed in the run, offline."
+            else
+              echo "- **A check failed on the new data. Do not merge.** This pull request is a draft; the run log has the failing output."
+            fi
+            if [ "$LASTGOOD_OUTCOME" = "success" ]; then
+              echo "- \`last-good.json\`: every part came back live and was refreshed."
+            else
+              echo "- \`last-good.json\`: **not fully refreshed**, a source was not fully live. Parts that failed keep their earlier dated rows. See the run log."
+            fi
+            echo
+            echo "## Before merging"
+            echo
+            echo "- [ ] Reread the hand-written README bullet on the rate-cut episodes (the sigma values); the workflow does not rewrite it."
+            echo "- [ ] \`ci.yml\` does not start for pull requests opened with the workflow token. Close and reopen this pull request to run lint and build."
+            echo "- [ ] \`docs/img/usdc-share-card.*\` and \`docs/img/summary.png\` are not regenerated here."
+            echo
+            echo "🤖 Opened by .github/workflows/refresh-snapshots.yml"
+          } > "$RUNNER_TEMP/pr-body.md"
+
+          title="data: refresh snapshots $(date -u +%F)"
+          n=$(gh pr list --head "$BRANCH" --state open --json number -q '.[0].number')
+          draft=""
+          [ "$CHECKS_OUTCOME" = "success" ] || draft="--draft"
+          if [ -n "$n" ]; then
+            gh pr edit "$n" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md"
+            [ -z "$draft" ] || gh pr ready "$n" --undo
+          else
+            gh pr create --base main --head "$BRANCH" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md" $draft
+          fi
+
+      - name: fail the job if a check failed (the alert below then fires)
+        if: always() && (steps.tsc.outcome == 'failure' || steps.tests.outcome == 'failure')
+        run: exit 1
+
+      - name: alert (one open issue per failure streak; comment on it if already open)
+        if: failure()
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          title="Snapshot refresh failed"
+          url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+          n=$(gh issue list --state open --search "\"$title\" in:title" --json number -q '.[0].number')
+          if [ -n "$n" ]; then gh issue comment "$n" --body "Failed again: $url"
+          else gh issue create --title "$title" --body "Run: $url"; fi

README.md

@@ -11,7 +11,11 @@ A Next.js dashboard that decodes Ondo's OUSG and USDY InstantManager events on E
 
 ## Results
 
-From the dated snapshot in `src/data/issuance.json` (event logs read 2026-10-24, Ethereum only, weeks of 26 Dec 2022 through 12 Oct 2026):
+The figures in this section and the dates below come from the committed snapshots. `npm run sync:readme` rewrites them, and `npm test` fails if they drift. The bullet on the 2024 and 2025 rate-cut episodes is hand-written: reread it after every refresh.
+
+<!--snapshot:results-->
+
+From the dated snapshot in `src/data/issuance.json` (event logs read 2026-10-25 UTC, Ethereum only, weeks of 26 Dec 2022 through 12 Oct 2026):
 
 | Asset | Mints | Redemptions | Net |
 |-------|------:|------------:|----:|
@@ -23,10 +27,22 @@ From the dated snapshot in `src/data/issuance.json` (event logs read 2026-10-24,
 | USDY | $250.1M | $68.8M | $181.3M |
 | **Five issuers** | **$526.3B** | **$490.5B** | **$35.8B** |
 
-- The snapshot holds 3,212,656 decoded event logs (the `audit` counts in `src/data/issuance.json`). USDC is 3.17M of them.
+<!--/snapshot-->
+
+<!--snapshot:counts-->
+
+- The snapshot holds 3,217,426 decoded event logs (the `audit` counts in `src/data/issuance.json`). USDC is 3.18M of them.
 - The five issuers total about $1T of mints and redemptions: $1.02T after removing $192.1B of Circle CCTP transfers, $1.21T if they were counted (`grossUsd` and `grossWithCctpUsd` in `src/lib/issuance.ts`). The tracker excludes CCTP because moving USDC between chains is not new money. It also excludes Paxos's mistaken 300 trillion PYUSD mint and its burn on 2025-10-15.
+
+<!--/snapshot-->
+
 - In the 2024 easing episode (Sep to Dec 2024, 100 bp), USDC and USDT net issuance ran above their own norm (+0.5σ each), and BUIDL and OUSG ran below theirs (−0.2σ and −0.3σ). The 2025 episode (Sep to Dec 2025, 75 bp) only partly repeated it. With two episodes this is descriptive, not evidence of cause. `/summary` renders these figures from `src/lib/issuance.ts`.
-- Ethereum log-derived net issuance matched DefiLlama's Ethereum supply change within 17% for USDC and 7% for PYUSD (`src/lib/issuance.check.ts`, run 2026-10-24).
+
+<!--snapshot:reconcile-->
+
+- Ethereum log-derived net issuance matched DefiLlama's Ethereum supply change within 18% for USDC and 8% for PYUSD (`src/lib/issuance.check.ts`, checked against the snapshot read <!--asof:flows-->2026-10-25<!--/asof-->).
+
+<!--/snapshot-->
 
 ## How it works
 
@@ -47,15 +63,19 @@ npm test
 npm run dev        # serves on port 3000; the deployed copy is ondo-nexus.vercel.app
 ```
 
-`npm test` runs the seven self-check scripts in `src/lib/*.check.ts` with `tsx` (pinned in `package.json`). They need no network. All seven passed on 2026-10-26. `issuance.check.ts` reconciles the committed snapshots. CI runs lint, the type check, `npm test` and `npm run build`.
+`npm test` runs the self-check scripts in `src/lib/*.check.ts` with `tsx` (pinned in `package.json`). They work from the committed snapshots and need no network: each one is preloaded with `scripts/no-network.mjs`, which makes any `fetch`, DNS lookup or socket connection throw, so a check that reaches for the network fails. `issuance.check.ts` reconciles the committed snapshots and `readmeSync.check.ts` checks the README against them. CI runs lint, the type check, `npm test` and `npm run build`.
+
+The snapshots refresh themselves. `.github/workflows/refresh-snapshots.yml` runs every Monday at 06:00 UTC (and on demand from the Actions tab). It reruns the backfill scripts and `scripts/snapshot-last-good.ts`, rewrites the README figures, runs the type check and `npm test`, and opens a pull request with the changed `src/data/*.json`. It needs no secrets: every source is public and keyless. Nothing merges by itself.
 
-To refresh the stablecoin snapshots (about 25 minutes on the free public RPC), then rebuild the study guide:
+To refresh by hand (about 25 minutes on the free public RPC):
 
 ```bash
 node scripts/backfill-issuance.mjs
 node scripts/backfill-context.mjs
-npx tsx src/lib/issuance.check.ts
-npx tsx docs/textbook/build.ts
+npx tsx scripts/snapshot-last-good.ts
+npm run sync:readme
+npm test
+npx tsx docs/textbook/build.ts   # rebuild the study guide
 ```
 
 The USDC share card (`docs/img/usdc-share-card.svg` and `.png`, 1024px wide, navy and red) is built from the same snapshot. On posting day run `npm run card:usdc-share:fresh`: it re-pulls the event logs, then renders the card with its data-as-of date. `npm run card:usdc-share` renders from the existing snapshot and refuses one that is not from today (`--allow-stale` overrides).
@@ -89,16 +109,36 @@ docs/             Methodology, contract research, decisions and the study guide
 - **Chain supply and price.** public RPCs and chain APIs on 13 chains, priced with `OndoOracle` on Ethereum. The per-chain table is in [docs/METHODOLOGY.md](docs/METHODOLOGY.md).
 - **Holders.** Blockscout token holders API for Ethereum (OUSG, USDY) and the stellar.expert asset holders API for Stellar USDY.
 - **Benchmark.** [DefiLlama](https://defillama.com) TVL per product, plus redemption terms quoted from each issuer's docs.
-- **Stablecoin flows.** Ethereum event logs through Tenderly's public gateway, DefiLlama stablecoin supply and FRED `DFEDTARU`. The backfill scripts freeze these as dated JSON in `src/data/`, with `asOf` 2026-10-24.
-- **Last good values.** `src/data/last-good.json` holds real reads from the sources above, written on 2026-10-26 (Superstate USTB and OpenEden TBILL rows date from 2026-09-24).
+- **Stablecoin flows.** Ethereum event logs through Tenderly's public gateway, DefiLlama stablecoin supply and FRED `DFEDTARU`. The backfill scripts freeze these as dated JSON in `src/data/`: event logs with `asOf` <!--asof:flows-->2026-10-25<!--/asof-->, supply by chain <!--asof:supply-->2026-10-24<!--/asof-->, rates <!--asof:rates-->2026-10-24<!--/asof--> (UTC dates).
+- **Last good values.** `src/data/last-good.json` holds real reads from the sources above, written on <!--asof:lastGood-->2026-10-26<!--/asof--> (Superstate USTB and OpenEden TBILL rows date from 2026-09-24).
+
+## How this was built
+
+<!-- DRAFT for Rahil to confirm or edit before this merges. Written by an AI agent from the git history, docs/DECISIONS.md, docs/METHODOLOGY.md and DEVLOG.md. Check who made which decision and what you checked yourself. -->
+
+AI coding agents wrote much of the code and documentation. Claude Code is credited as co-author on most commits (20 of 30 when this section was written; `git log --grep "Co-authored-by: Claude" -i`), and `.claude/` holds the agents, commands and skill it ran with. Rahil Bhavan is the author of the project and its PRD, and owns what it claims.
+
+What Rahil designed, as the repo records it:
+
+- **The question and the scope.** The PRD sets the goal: an independent, on-chain view of Ondo Nexus adoption. The decision records in [docs/DECISIONS.md](docs/DECISIONS.md) fix the scope: OUSG and USDY InstantManager activity stands in for Nexus, which has no contracts of its own (ADR-003); volume and frequency replace a settlement delta that atomic redemptions make unmeasurable (ADR-004); Circle CCTP transfers and Paxos's mistaken 300 trillion PYUSD mint are not issuance (ADR-009).
+- **The integrity rules.** Every metric carries a live, estimated or last good badge, and no number is invented: a failed read shows an earlier real read with its date, or nothing (ADR-010, `CLAUDE.md`).
+- **The metric definitions** in [docs/METHODOLOGY.md](docs/METHODOLOGY.md) and the Dune SQL in `queries/`.
+
+What was checked against an independent source, as the docs record it:
+
+- The decoded weekly Ondo flows matched Dune query 8822192 on all 157 (week, token) rows through the week of 2026-09-14, to the cent on volume (ADR-008).
+- Chain supply matched an independent explorer to the token on Ethereum USDY and OUSG, Mantle USDY and Plume USDY, and Stellar USDY matched stellar.expert within 0.000005% (`scripts/verify-supply.ts`, METHODOLOGY Pillar 3).
+- Log-derived net issuance for USDC and PYUSD is reconciled against DefiLlama's Ethereum supply change in `src/lib/issuance.check.ts`.
+
+What was not independently checked: supply on the other chains rests on each chain's own RPC or API (METHODOLOGY Pillar 3); the Superstate USTB and OpenEden TBILL rows are an old snapshot; and the textbook's reading list was written from memory and its citations still need verifying (DEVLOG, 2026-10-17).
 
 ## Scope and limits
 
 - Independent project. It is not affiliated with or endorsed by Ondo Finance, Circle, Tether, PayPal, Paxos, BlackRock or Securitize.
 - Ondo Nexus has no contracts of its own. The dashboard uses OUSG and USDY InstantManager activity as its on-chain proxy ([ADR-003](docs/DECISIONS.md)).
 - It counts instant mints and redemptions only. Secondary transfers, DEX trades and bridged balances are out of scope.
 - Gross flows are Ethereum only. Tron holds about half of USDT supply, so Ethereum logs understate USDT and USDC issuance. Net issuance for the three stablecoins uses all-chain supply instead.
-- The stablecoin data are a snapshot from 2026-10-24. It is stale until someone reruns the backfill scripts.
+- The stablecoin data are a dated snapshot (event logs read <!--asof:flows-->2026-10-25<!--/asof-->), not a live read. A weekly workflow proposes a refresh as a pull request, so the snapshot is as fresh as the last merged refresh.
 - The rate-cut comparison covers two easing episodes, so it is descriptive only.
 - On 2026-10-21, DefiLlama's `/tvl/<slug>` endpoint returned `0` for all four benchmark products. The USTB and TBILL rows therefore show their 2026-09-24 last good value, labeled **last good**.
 

STATUS.md

@@ -1,5 +1,5 @@
 # Nexus Dashboard — Current Status
-Last updated: 2026-10-24
+Last updated: 2026-10-29
 
 ## Phase
 [x] Phase 0 — Contract Research
@@ -10,7 +10,8 @@ Last updated: 2026-10-24
 [x] Study textbook for the outreach (`docs/textbook/index.html`, rebuilt from the data snapshots)
 [x] Deployed to Vercel (ondo-nexus.vercel.app, personal scope)
 [x] Benchmark: BUIDL live from DefiLlama's stablecoin series (#54); USTB and TBILL stay labeled snapshots (no free source)
-[x] Stablecoin snapshots rerun 2026-10-24
+[x] Stablecoin snapshots rerun 2026-10-25 (event logs), 2026-10-24 (supply, rates)
+[x] Weekly `refresh-snapshots` workflow proposes snapshot refreshes as a pull request; `npm test` runs offline and checks the README against the snapshots
 
 ## Live Pages
 - `/` — home: KPIs, top holders, chain breakdown, heatmap, 12-week flows, benchmark
@@ -28,7 +29,8 @@ Last updated: 2026-10-24
 
 ## Open Items
 - After refreshing the snapshots, rebuild the textbook: `npx tsx docs/textbook/build.ts`.
-- Flows tracker data are dated snapshots (last run 2026-10-24): rerun `node scripts/backfill-issuance.mjs` (about 25 min for the first pass; the dense USDC, PYUSD and BUIDL series take about 40 min at the 10k-block window and Tenderly's public gateway may 429, so rerun those with `node scripts/backfill-issuance.mjs USDC PYUSD BUIDL`) and `node scripts/backfill-context.mjs`, then `npx tsx src/lib/issuance.check.ts`.
+- Flows tracker data are dated snapshots (event logs read 2026-10-25). `.github/workflows/refresh-snapshots.yml` reruns the backfill scripts every Monday and opens a pull request; merge it after reading the hand-written README bullet on the rate cuts. By hand: `node scripts/backfill-issuance.mjs` (about 25 min; the dense USDC, PYUSD and BUIDL series can take 40 min and Tenderly's public gateway may 429), `node scripts/backfill-context.mjs`, `npm run sync:readme`, `npm test`, then `npx tsx docs/textbook/build.ts`.
+- Rahil action: confirm or edit the README section "How this was built" (drafted by an agent from the history), and check Settings > Actions > General allows Actions to create pull requests.
 - Flows tracker is Ethereum logs only; Tron USDT and non-Ethereum USDC issuance are covered through supply, not logs.
 - BUIDL chain count and redemption terms: no issuer page to cite, shown as "Not disclosed".
 - Rahil action: rotate the Dune API key (the app no longer uses Dune; only the MCP server does).