Skip to content

P3 (part 2): archive the Gate 1 products, pin the run commit, recover the use case (#46) - #55

Merged
mdenolle merged 3 commits into
docs/sign-convention-manuscriptfrom
iter3/P3-archive
Nov 1, 2026
Merged

mdenolle merged 3 commits into
docs/sign-convention-manuscriptfrom
iter3/P3-archive

Conversation

@mdenolle

@mdenolle mdenolle commented Nov 1, 2026

Copy link
Copy Markdown
Member

Advances #41 and acts on the answers in #46. Part 2 of package P3: the field products are archived, the run is pinned, and the use case is recovered from the data rather than assumed.

The four answers, and what each became

Answer in #46 What it became here
Archive the other three bands too All four octave bands at the three stations are tracked (0.8 MB with the pre-correction originals). The comparison figure now regenerates from any checkout, so the sidecar check in CI covers it instead of skipping it
SCEDC-derived products may be archived and disseminated, with the licensing tracked paper/data/gate1/DATA_LICENCE.md quotes the grant verbatim from the open-data registry entry, names the two citations it requires, and states that only derived series are redistributed. Both DOIs are in the bibliography and cited in the data availability statement
Keep the use-case flag The use case is recorded as part of the provenance, and recovered as groundwater
Pin the latest cloud commit be1538b

The two claims are checkable, so they were checked

Use case. The dv/v stage hashes its effective configuration into the config_hash column of every product. Recomputing that digest over the six use cases reproduces the stored value for groundwater on all four bands, and for nothing else, including the no-use-case fallback:

band (Hz) stored reproduced by
1-2 dc93a54800a6 groundwater
2-4 2976066eb87b groundwater
4-8 ab49fee56135 groundwater
8-16 d06385415eed groundwater

At 2-4 Hz that gives the 2 to 8 s coda window the ensemble was built around, which matches the hint in the issue.

Commit. The products were written at 15:25 on 2026-08-09. be1538b (15:39 UTC that morning) is the last commit touching the dv/v stage before that, nothing at all landed in the pipeline repository between the two, and the next change to the stage is 567fb8b on 2026-08-17. It is also the latest of the three candidates the audit listed, so the instruction and the evidence agree.

Also here

  • scripts/gate1_manifest.py writes MANIFEST.json and CHECKSUMS.sha256 for every file under the data directory and verifies them with --check.
  • tests/test_gate1_archive.py pins both provenance claims: every archived file matches its published digest, and the groundwater reproduction holds on all four bands. A change to the use-case recommendations now fails a test rather than leaving the paper asserting something untrue.
  • The data availability statement, the data README and the figure sources note are updated; the "remain to be supplied" sentence is gone.

One decision left for the author

The licence for our own derived products. CC BY 4.0 is proposed in DATA_LICENCE.md and marked as needing confirmation before the deposit. The versioned deposit with a DOI stays in #41 and belongs with the release.

Verification

  • pytest tests/test_gate1_archive.py: 9 passed.
  • The comparison figure regenerates from the archived inputs and matches its committed sidecar.
  • PDF rebuilds to 88 pages; no unresolved references, comma-percent artefacts, script names or module names in its text; both SCEDC citations render in the reference list.

🤖 Generated with Claude Code

… answers #46)

The author's answers to #46: archive all four bands, SCEDC-derived
products may be redistributed for academic research with the licensing
tracked, keep the use-case flag, pin the latest cloud commit.

Data. All four octave bands at the three stations are now tracked
(0.8 MB), with their pre-correction originals, so the field comparison
figure regenerates from any checkout and the continuous-integration
sidecar check covers it instead of skipping it. scripts/gate1_manifest.py
writes MANIFEST.json and CHECKSUMS.sha256 over everything under
paper/data/gate1/ and verifies them with --check.

Rights. paper/data/gate1/DATA_LICENCE.md quotes the SCEDC grant verbatim
from its open-data registry entry, names the two citations it requires
(doi:10.7909/C3WD3xH1 and doi:10.7914/SN/CI, both added to the
bibliography and cited in the data availability statement), states that
only derived series are redistributed, and records the comparison
product's source. The licence for our own derived products is proposed as
CC BY 4.0 and marked as the author's to confirm before the deposit.

Use case: groundwater, recovered rather than assumed. The dv/v stage
hashes its effective configuration into the config_hash column; that
digest is reproduced by the groundwater recommendation on all four bands
and by no other use case, nor by the no-use-case fallback. At 2-4 Hz it
gives the 2-8 s window the ensemble was built around.

Pipeline commit: be1538b. The products were written at 15:25 on
2026-08-09; be1538b (15:39 UTC that morning) is the last commit to the
dv/v stage before that, nothing landed in the repository between the two,
and the next change to the stage is 567fb8b on 2026-08-17. It is also the
latest of the three candidates the audit listed.

Tests. tests/test_gate1_archive.py pins both provenance claims: every
archived file matches its published digest, and the groundwater
reproduction of the configuration digest holds on all four bands. If
codameter.use_cases changes, the test fails rather than the paper
asserting something untrue.

Still open in #41: the licence confirmation and the versioned deposit
with a DOI, both at the release.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mdenolle mdenolle added this to the Iteration 3 revision milestone Nov 1, 2026
@mdenolle mdenolle added P3 provenance, release, real-data figures review-iter2 from the iteration-2 pre-submission review labels Nov 1, 2026
Copilot AI balanced review requested due to automatic review settings November 1, 2026 15:01
@mdenolle mdenolle added the review-iter2 from the iteration-2 pre-submission review label Nov 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An unresolved moderate issue remains where --check mode calls describe()/pd.read_parquet on every file, redundantly re-reading products and risking a traceback-aborted check on the corrupted files it is meant to detect.

1 open issue
What changed in this PR

This is part 2 of package P3 (issue #41), acting on the author's answers in #46. It moves the Gate 1 real-data products from untracked-local to archived-in-repo: all four octave bands at the three CI stations are committed with their pre-correction originals, SHA-256 digests, and licence terms. It pins the pipeline run to noisepy-dvv-cloud commit be1538b and records the groundwater use case, which is reconstructed from the config_hash the products carry rather than asserted. A manifest script and a test lock both provenance claims so a silent change to the data or to codameter.use_cases fails a test. The data-availability statement, README, licence file, and figure-source notes are updated to match, and the comparison figure can now regenerate from any checkout.

Changes:

  • Archive the four-band Gate 1 products (plus .v040 originals) with MANIFEST.json/CHECKSUMS.sha256, a DATA_LICENCE.md, a gitignore negation, and SCEDC/SCSN bib entries cited in the manuscript.
  • Add scripts/gate1_manifest.py (write/--check) and tests/test_gate1_archive.py pinning digest integrity and the groundwater config-hash reproduction across all four bands.
  • Update prose/provenance (README, .tex/.qmd data availability, SOURCES.md, gate1.py docstring, collaboration/revision logs) to state the products are archived and the run is pinned.
File Description
tests/​test_gate1_archive.py New tests: digest match, CHECKSUMS/MANIFEST agreement, coverage, and groundwater hash reproduction.
src/​codameter/​gate1.py Docstring and MissingInputs message updated from "untracked" to "archived".
src/​codameter/​figures.py Registry label for the comparison figure updated to reference archived products.
scripts/​gate1_manifest.py New manifest/checksum writer with --check verification; describe() runs in check mode (flagged, optional).
review/​REVISION_PLAN_ITER3.md Appended dated log line for P3 part 2.
review/​HUMAN_AI_COLLABORATION.md Added collaboration items 15-16 and an author-decision bullet.
paper/​references.bib Added SCEDC2013 and SCSN1926 dataset citations.
paper/​manuscript_marine.tex /​ .qmd Rewrote data-availability statement; removed "remain to be supplied"; cite both dataset DOIs.
paper/​data/​gate1/​README.md Expanded provenance section with the use-case/commit evidence and rights.
paper/​data/​gate1/​MANIFEST.json New manifest: provenance block plus per-file digests/metadata.
paper/​data/​gate1/​DATA_LICENCE.md New licence note; verbatim SCEDC grant; CC BY 4.0 proposed, pending author confirmation.
paper/​data/​gate1/​CHECKSUMS.sha256 New two-column checksum list for all archived files.
literature/​figs/​SOURCES.md Note updated: comparison figure regenerates from any checkout.
.gitignore Un-ignore paper/data/gate1/dvv2y/**/*.parquet with an explanatory comment.

Notes for the author: the manuscript's new SCEDC2013/SCSN1926 citations land in references.bib, and Clements2023 resolves via survey.bib (both are included by \bibliography{references.bib,survey.bib}), so references are intact. The config-hash table in the README matches MANIFEST.json on all four bands, the ~0.8 MB figure and dates check out, and DATA_LICENCE.md's "herby" is part of the verbatim upstream grant, not a typo to fix.

Review effort: Balanced

Comment thread scripts/gate1_manifest.py Resolved
#41, Copilot review)

--check no longer calls the parquet reader: it compares digests, catches
an unreadable file as a reported condition rather than a traceback, and
exits 1 on it. A test corrupts a copy of a product and asserts the
report.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 1, 2026 15:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

An unresolved finding remains, where a stale sentence in SOURCES.md still says the comparison figure's inputs "are to be pinned" even though this PR already records that provenance, contradicting the surrounding text.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Update stale provenance sentence in Gate 1 README

literature/​figs/​SOURCES.md:43

This sentence is now stale: this PR pins the Gate 1 run commit (be1538b) and --use-case (groundwater) in paper/data/gate1/README.md, so the "are to be pinned" wording in the same paragraph you are rewriting contradicts the following sentence, which asserts the comparison figure's inputs are archived. Recommend updating it to reflect that the provenance is now recorded.

1 resolved since last review

Review effort: Balanced

#41, Copilot review)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mdenolle

mdenolle commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

Addressed in 02b13f3: the sources note states the pinned commit and use case instead of asking for them.

Copilot AI balanced review requested due to automatic review settings November 1, 2026 15:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The redistribution rights, verbatim licence text, and the pinned commit depend on author confirmation and external sources that cannot be verified in this checkout, leaving key facts unconfirmed.

No open issues 🎉

Review effort: Balanced

@mdenolle
mdenolle merged commit 5d2d223 into docs/sign-convention-manuscript Nov 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 provenance, release, real-data figures review-iter2 from the iteration-2 pre-submission review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants