Skip to content

Add license-safe real-world test fixtures with offline verification - #35

Merged
zhemingfan merged 2 commits into
mainfrom
feature/real-fixtures
Jul 4, 2026
Merged

Add license-safe real-world test fixtures with offline verification#35
zhemingfan merged 2 commits into
mainfrom
feature/real-fixtures

Conversation

@zhemingfan

Copy link
Copy Markdown
Owner

Summary

Vendors a small, license-vetted set of real-world genomics files so BioFmt's detection, validators, and indexed-query path are exercised against actual caller/portal output — not just hand-made samples. Adapted from a fixture-sourcing research plan, but re-cast into this repo's toolchain: no Python/pytest/shell-tabix at test time, JSON manifest (no new dep), and a hard offline/network wall.

What's here

  • test/fixtures/real/sources.json — manifest and source of truth: per fixture, the source URL, license, extraction command, pinned sha256, and expect assertions.
  • scripts/fetch-real-fixtures.ts (npm run fetch:fixtures) — dev-only, the only thing that touches the network. Downloads → slices (tabix/bgzip region extraction, or header+N-row slices) → reindexes → writes the fixture + provenance/<id>.json → pins sha256. Never run by the test suite.
  • test/unit/real-fixtures.test.tsfully offline Mocha suite. Per fixture: presence → checksum → format is supported → the real validator runs without throwing (robustness) → minimum row count → for indexed VCFs, a region query through @gmod/tabix (the lib BioFmt ships, not shell tabix). Entries without a committed file skip.
  • provenance/*.json + ATTRIBUTION.md — source, checksum, and per-source license.

Fixtures (11, ~544 KB total)

Fixture Format License
DeepVariant golden VCF + gVCF vcf BSD-3
FreeBayes tiny VCF vcf MIT
GRIDSS example SV VCF vcf GPL-3
GIAB HG002 VCF slice (indexed) vcf NIST public benchmark
GIAB HG002 BED bed NIST public benchmark
1000 Genomes chr22 VCF slice (indexed) vcf IGSR open data
GENCODE v50 GTF + GFF3 (chr22) gtf/gff3 GENCODE/EMBL-EBI open
UCSC hg38.chrom.sizes chrom.sizes UCSC open
UCSC bigPsl example psl UCSC open

Robustness result: every real file validated without throwing — the validators hold up against real DeepVariant/GRIDSS/GENCODE/1000G output. Both indexed VCFs answer @gmod/tabix region queries offline against the committed .gz + .tbi.

Deliberately excluded

API-resolved / non-deterministic or terms-encumbered sources: ENCODE (API pin), TCGA MAF/GCT (attribution + controlled API), 10x MTX (site redistribution terms), minimap2 PAF (tool absent), PRIDE mzTab/MGF (unpinned), Manta (Polyform Strict — never vendor), CNVkit .seg/.cns (not a BioFmt format). The manifest is shaped so these can be pinned in follow-up commits.

Testing

Offline suite green (46 assertions across the 11 fixtures); full unit suite 848 pass; lint, type-check, and build green.

Follow-ups (not in this PR)

  • Pin the deferred sources as their license/size tradeoffs are accepted.
  • A malformed/truncated variant per fixture for graceful-failure assertions.
  • Wire c8 coverage to include this suite; use a larger slice as a perf-gate input.

Vendors 11 small, sliced excerpts of publicly redistributable genomics data
(DeepVariant, FreeBayes, GRIDSS, GIAB HG002, 1000 Genomes, GENCODE, UCSC) so
BioFmt's detection, validators, and indexed-query path are exercised against real
caller/portal output rather than only hand-made samples.

- test/fixtures/real/sources.json: manifest (source URL, license, extraction
  command, pinned sha256, assertions) as the source of truth.
- scripts/fetch-real-fixtures.ts: dev-only fetch/slice/reindex/pin. Network is
  used only here, never by the test suite (npm run fetch:fixtures).
- test/unit/real-fixtures.test.ts: offline per-fixture checks — presence,
  checksum, format support, the real validator runs without throwing, minimum
  row count, and an @gmod/tabix region query for indexed VCFs.
- provenance/*.json and ATTRIBUTION.md record source, checksum, and license.

Only redistribution-safe sources are vendored; API-resolved, terms-encumbered,
and restrictively-licensed sources (Manta, 10x, controlled-access TCGA) are
intentionally excluded. Total footprint ~544 KB.
Adds a manifest-completeness assertion to the real-fixtures suite: any entry with
a pinned sha256 must have its committed file (and .tbi index, for indexed
entries) present. The per-entry checks skip absent files by design, so without
this guard a deleted fixture would silently stop being tested; dropping one now
requires a visible edit to sources.json.
@zhemingfan
zhemingfan merged commit 7d7f318 into main Jul 4, 2026
3 checks passed
@zhemingfan
zhemingfan deleted the feature/real-fixtures branch July 4, 2026 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant