Skip to content

Finer CI assertion gates: per-phase pruning, stats coverage, max files read #133

Description

@cdelmonte-zg

The current gates are --min-pruning (total pruning percentage) and --assert-stats (table-wide stats presence). They cannot express which phase must do the work, nor bound the absolute cost of a scan, nor catch a predicate column whose statistics are missing on the files that matter.

Proposed additions:

  • Per-phase minimum pruning thresholds, one for partition pruning and one for data skipping (exact flag names to be settled in the design; the phase vocabulary is the one already in the report).
  • A stats-coverage gate over the per-column coverage shipped in Per-column stats coverage for stats-safe predicate columns #132 (analysis.stats_coverage): fail unless every predicate column's coverage over the candidate files meets a threshold. 100% recovers the original "listed columns must carry stats" case; 0% is the structural no-op, which by design does not degrade confidence (see the resolution in Per-column stats coverage for stats-safe predicate columns #132) - the gate, not the label, is the CI channel for it. Requirement-aware for free, since the block already distinguishes min_max from null-count coverage. Exact flag shape to be settled in the design (a global --min-stats-coverage <pct>, a per-column form, or both).
  • --max-files-read <N>: absolute bound on surviving files, complementing the percentage-based gates on tables where percentages hide real cost.

All of these follow the existing gates model: pure assertion records in gates.rs, deterministic failure messages, additive JSON.

The stats-coverage gate asserts on data the report already computes: #132 shipped analysis.stats_coverage with schema_version 0.5.0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions