Remove legacy pb.gz datasets in favor of their Parquet replacements

[skearnes]

Summary

Deletes all 550 legacy .pb.gz files under data/, leaving the 53 .parquet files as the only stored form of the corpus. This is the data half of the Parquet-only migration; the tooling and documentation half is stacked on top in #256, which targets this branch.

The split exists because a 550-file diff is past Greptile's review limit. There is deliberately no code in this PR — every reviewable change lives in #256.

No dataset is lost

Of the 550 deleted inputs, 51 have a same-id .parquet sibling and the other 499 were consolidated by the un-sharding in #241:

Was	Now	Dataset.name	Reactions
489 × uspto-grants-YYYY_MM	ord_dataset-1158e351757f315b93cbcbe7bc55f38e	uspto-grants	1,771,032
10 × Training data from https://doi.org/10.1039/C8SC04228D (N/10)	ord_dataset-e7830cd6b11158b43994ccfb5ee9acb3	Training data from https://doi.org/10.1039/C8SC04228D	409,035

As paths:

data/11/ord_dataset-1158e351757f315b93cbcbe7bc55f38e.parquet
data/e7/ord_dataset-e7830cd6b11158b43994ccfb5ee9acb3.parquet

Those two are the only dataset ID changes. Every other dataset kept its ID, so for those only the suffix changes. Per-reaction patent provenance for the USPTO data is still on Reaction.provenance.patent, so the monthly bucket a reaction came from remains recoverable without the per-month datasets.

How coverage was verified

• scripts/convert_to_parquet.py --dry-run classified all 550 inputs as 51 singletons, 489 uspto-grants shards and 10 C8SC04228D-training shards, and reported all 53 outputs as already existing. The two merged outputs are addressed by the id the script derives from its sorted source ids, so the paths it reports as existing are themselves the check that the merge groups are exactly these inputs.
• A row-count audit loaded every .pb.gz and compared reaction counts against the Parquet files: every singleton matches its same-id sibling, and both merged outputs match their shard totals exactly.
• Comparing full protos rather than counts, 49 of the 51 singletons are byte-identical. The two exceptions, ord_dataset-ac78456835404910b3a4c840248b6ac9 and ord_dataset-d319c2a22ecf4ce59db1a18ae71d529c, differ only in Dataset.description, where the Parquet carries the whitespace fix from Insert the missing whitespace in two dataset descriptions #259 and the .pb.gz was left stale. Every reaction in both is byte-identical, so the Parquet is the better copy in each case.
• No .pb.gz was ever committed after its .parquet sibling, so no later edit is stranded — which matters because process_dataset.py keeps an edited dataset in the format it already has and would not have refreshed the sibling.
• After the deletion, data/ is byte-for-byte the same set of 53 Parquet files as main.

Updating downstream code

Read datasets with ord_schema.datasets.load_dataset(path), which dispatches on suffix and returns a streaming DatasetView for Parquet; pass as_dataset=True when you need a materialized Dataset for serialization or mutation. message_helpers.load_message has no Parquet path. The README examples in #256 show both, and the API notes there go into more detail.

Notes for review

• On merge, the Hugging Face mirror propagates the 550 deletions to the HF dataset. scripts/upload_to_huggingface.py maps D entries to CommitOperationDelete, a directly tested path, and with no uploads in the plan the LFS pull short-circuits — so this lands as one HF commit of 550 deletes with no LFS transfer.
• The README on main still documents .pb.gz as the storage format, and the mirror republishes it as the HF dataset card in this same commit. Assert validation coverage and finish the Parquet-only migration #256 rewrites it, so it should merge immediately after this one.
• .lfsconfig routes LFS reads to the HF mirror, so a checkout of a pre-merge commit resolves the deleted .pb.gz oids against a mirror whose HEAD no longer references them. The verify-hf-mirror skill reports all 603 objects on main served by HF today, and HF still serves an object superseded by Insert the missing whitespace in two dataset descriptions #259 a month ago and unreferenced since — its LFS store keeps objects that a git deletion drops. Those objects also stay in GitHub's LFS store, which is where writers push.
• Labeled skip-update-submission so process_dataset.py does not re-process the deletions.
• .gitattributes keeps the *.pb/*.pb.gz LFS filters under data/. They match nothing after this merge: a submission is staged at the repository root, where no LFS filter applies, and process_dataset.py --update writes it into data/ as Parquet. They stand as a safety net for a deliberate --output_format write, not for the submission path.

🤖 Generated with Claude Code

[skearnes]

@greptileai review

[greptile-apps]

Too many files changed for review (550 files, 500 file limit).