Skip to content

feat(esm_catalog): PR-A3 — shard federation (merge_shards)#1488

Open
siligam wants to merge 1 commit into
esm-catalog/pr-a2-duckdb-shardfrom
esm-catalog/pr-a3-federation
Open

feat(esm_catalog): PR-A3 — shard federation (merge_shards)#1488
siligam wants to merge 1 commit into
esm-catalog/pr-a2-duckdb-shardfrom
esm-catalog/pr-a3-federation

Conversation

@siligam

@siligam siligam commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds storage/federation.py: merge_shards(shard_paths, output_path) — uses DuckDB ATTACH to copy collections, items, and collection_item_props rows from any number of per-experiment .duckdb shards into a single global catalog file. Rows are upserted (INSERT OR REPLACE / INSERT OR IGNORE) so the command is safe to re-run as shards are refreshed. Missing paths are skipped with a warning.
  • Adds esm-catalog merge <shard>... --output <global.duckdb> CLI subcommand.
  • 7 new tests: two-shard merge, idempotency (no duplicates on re-merge), incremental merge into a non-empty global, missing-shard tolerance, property index propagation, and CLI end-to-end.

This is Miguel's third unit — "sync shard to global DB". Together with A1b-1/A1b-2 (scan) and A2 (shard), the full ingest pipeline is now:

esm-catalog scan <exp-dir> --db exp-alpha/catalog.duckdb
esm-catalog scan <exp-dir> --db exp-beta/catalog.duckdb
esm-catalog merge exp-alpha/catalog.duckdb exp-beta/catalog.duckdb --output global.duckdb

Test plan

  • pytest src/esm_catalog/tests/test_federation.py -v — 7 tests pass locally
  • Full suite pytest src/esm_catalog/tests — 62 tests, 0 failures
  • CI green on Python 3.9 + 3.12

🤖 Generated with Claude Code

@siligam
siligam force-pushed the esm-catalog/pr-a2-duckdb-shard branch from 25af643 to 55decf4 Compare June 17, 2026 12:12
@siligam
siligam force-pushed the esm-catalog/pr-a3-federation branch from 97c7c22 to ca2ed54 Compare June 17, 2026 12:12
@siligam
siligam force-pushed the esm-catalog/pr-a2-duckdb-shard branch from 55decf4 to f1469e5 Compare June 17, 2026 15:57
@siligam
siligam force-pushed the esm-catalog/pr-a3-federation branch from ca2ed54 to e9dde21 Compare June 17, 2026 15:57
- Add storage/federation.py: merge_shards(shard_paths, output_path) uses
  DuckDB ATTACH to upsert collections, items, and collection_item_props
  from any number of per-experiment shards into a single global catalog.
  Safe to re-run (INSERT OR REPLACE / INSERT OR IGNORE); missing shard
  paths are skipped with a warning.
- Add 'esm-catalog merge' CLI: wraps merge_shards() for the command line.
- 7 new tests: two-shard merge, idempotency, incremental merge into
  non-empty global, missing-shard tolerance, props indexed, CLI end-to-end.

This is Miguel's "sync shard to global DB" unit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@siligam
siligam force-pushed the esm-catalog/pr-a2-duckdb-shard branch from f1469e5 to e748bf2 Compare June 17, 2026 16:34
@siligam
siligam force-pushed the esm-catalog/pr-a3-federation branch from e9dde21 to fd065b9 Compare June 17, 2026 16:34
@siligam

siligam commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the corrected PR-0 base (#1483) — test relocation to tests/test_esm_catalog/, restored dedicated CI workflow at the new path, and a couple of CI fixes (doctest-collection exclusion, eval_type_backport for pydantic on Python 3.9). No change to this PR's own diff/scope. CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant