Skip to content

verify: neither of fail-on's failing modes is exercised #4

Description

@vyncint

Todayfail-on has three modes, and only the one that never fails is exercised anywhere:

mode meaning tested
never report and pass track.yml, and all four verify.yml cases
behind fail when today is a letter day and it is short nowhere
holed fail when the text can no longer be drawn cleanly nowhere

fail-on is the input that decides whether the job goes red. It is the whole reason the action is usable as a gate rather than a report, and both of its failing modes are untested — in mossaic's suite and here.

Why it is worth fixing — this repository is where the action meets a real contribution graph, so it is the only place the failing modes can be tested against real data rather than a fixture. And a gate that never fails in testing is a gate nobody knows works: the failure mode is silent in the worst way, since a broken fail-on: behind looks exactly like a day you were not behind.

The behind mode also carries a documented subtlety worth pinning, because it is the kind of thing that regresses invisibly:

behind counts letter days only: with background set, an unstarted background day is short too, and 290 of them a year failing the job would make the signal worthless.

So fail-on: behind on a background day that is short must pass. That is a real assertion with a real off-by-one waiting in it, and nothing checks it.

Fix — add cases to verify.yml, using continue-on-error: true on the gate step and asserting steps.<id>.outcome rather than letting the job's own result carry the claim:

  • fail-on: holed against the live plan — currently holed (the 2026 plan is holed: 5 lit days inside the letters, and start-week 6 is frozen #3), so it must fail. If the 2026 plan is holed: 5 lit days inside the letters, and start-week 6 is frozen #3 is resolved by re-planning, this case needs a deliberately-holed plan of its own instead of the live one, which is the more robust construction anyway.
  • fail-on: holed against a plan with no holes — must pass.
  • fail-on: behind on a background-only day with background set — must pass, per the note above.
  • fail-on: behind where today is a short letter day — must fail. Needs a plan whose letter days land on today, which --start-week can arrange; today-kind says which kind today is, so the case can assert its own precondition and skip cleanly rather than pass vacuously when the calendar does not cooperate.

That last point is the trap: a fail-on: behind case that runs on a day when nothing was owed passes without testing anything, and would go on passing forever. Each case should assert today-kind first and fail loudly if the precondition it needs is absent.

Done whenverify.yml covers both failing modes and the background-day exemption, and every case asserts its own precondition so none of them can pass vacuously.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions