Skip to content

Resolve companion/auxiliary files (checksums, indexes) — propagate from parent or tag by type, genomic dims not_applicable #312

Description

@NoopDog

~64K of the ~80K currently-unclassified files are non-primary-data, and a large chunk are companions of a real file.md5 checksums (14,179), plus indexes (.bai/.fai/.gzi). Today they're silent gaps. They should be explicitly resolved, like classify_index_files.py already does for .tbi/.crai.

Idea (from the field, 2026-07-31)

"Classify the checksums as checksums and then call everything else NA, or link it to the thing it is a checksum of, like we do with index."

Two acceptable resolutions per companion file:

  1. Tag by type.md5 → data_type checksum (or similar), genomic dimensions (reference/modality/assay/platform) not_applicable. A positive determination, not an unknown (cf. Distinguish 'unreadable' (fetch failed) from 'read but no expected content' — VCF/FASTQ mislabel a clean read as content_unreadable #299 unreadable-vs-not-this-type).
  2. Propagate from parentx.vcf.gz.md5 inherits x.vcf.gz's classification (as .tbi/.crai already do). Better when the parent is classified; keeps the companion consistent with what it accompanies.

Likely: propagate when the parent is present/classified, else tag by type.

Scope

Why it matters

  • Turns silent gaps into positive not_applicable/propagated answers → the residual "unclassified" count then means "real gap," not "auxiliary file."
  • Consistent with the existing index-propagation pattern and the accuracy-over-coverage principle (a checksum having no reference is a fact, not a failure).

DoD

  • .md5 resolved (propagate-from-parent, else checksum + genomic dims not_applicable).
  • .bai/.fai/.gzi propagate from parent like .tbi/.crai.
  • Companion resolution reuses/extends classify_index_files.py rather than a new one-off.
  • Coverage re-measured; unclassified drops toward genuine-gap-only.

Relates to #311 (data-format tail), #299 (resolved vs unknown), #301 (producing method).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions