Skip to content

Nightly job silently skips half of its inputs #641

Description

@shunk031

Summary

The nightly job completes successfully even though it processes only about half of the inputs it receives. The skipped inputs are not reported, retried, or reflected in the job status, so the resulting output looks healthy while being incomplete.

Expected behavior

Every input should be processed exactly once, or the job should clearly report which inputs were skipped and fail or retry when the final processed count does not match the input count.

Impact

Nightly results can be incomplete without an alert. Any downstream consumer that assumes a successful run contains all inputs may make decisions from partial data.

Acceptance criteria

  • Record the total input count and processed, skipped, failed, and retried counts for each nightly run.
  • Ensure the job does not silently drop inputs; skipped or failed inputs are retried or listed in the run output.
  • Mark the run failed, or otherwise raise an actionable alert, when the processed result is incomplete.
  • Add a regression test covering a full nightly input set and verifying that every input is accounted for.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions