Skip to content

[bug] CI failure bot posts "No Failures Found" instead of staying silent #694

Description

@nemesifier
Image

Describe the bug

The CI failure bot (reusable-bot-ci-failure.yml, used via bot-ci-failure.yml) posts a comment saying "No Failures Found" / "No Failures Detected" when its log analysis finds no failures. It should stay silent in that case: if there is nothing actionable for the author, it should not comment at all.

This was very visible on openwisp/openwisp-monitoring#813, where the bot posted "No Failures Found" several times:

Steps To Reproduce

  1. Open a PR whose CI run fails on a transient/infra failure (so the bot triggers and/or auto-restarts the failed jobs), or rerun a failed run while the bot is processing it.
  2. Let the failed jobs be re-run (by the bot's own transient auto-restart, e.g. [tests] Fixed flaky timeseries tests caused by UDP write/read race openwisp-monitoring#813 (comment), and/or by a manual gh run rerun) so the latest attempt passes.
  3. The bot, which was triggered by the earlier failed attempt, fetches the run logs by run_id and analyzes them.
  4. Because the run's log endpoint returns the latest attempt (now passing), the analysis finds no failures and the bot posts "No Failures Found".

Expected behavior

  • When the analysis detects no failures, the bot should exit without posting any comment.
  • Ideally the bot should also analyze the logs of the attempt that actually failed, not whatever the latest attempt happens to be by the time it runs.

Root cause analysis

There are two distinct issues:

  1. The no-failure code path posts a comment instead of being a no-op. This is the bug to fix first and it fully removes the unwanted "No Failures Found / Detected" comments.

  2. A race between the bot's log fetch and reruns of the same run. The bot is triggered by a failed attempt N, but it fetches logs by run_id, and the GitHub run-logs endpoint returns the latest attempt. If a rerun (the bot's own transient auto-restart, or a manual rerun) has already produced a passing attempt N+1, the bot analyzes the passing logs and concludes there are no failures. The calling workflow passes run_id (github.event.workflow_run.id) but not run_attempt, so the reusable workflow cannot pin the analysis to the failing attempt.

The race is amplified when the bot's transient auto-restart and a maintainer's manual reruns act on the same run at the same time (which is what happened in #813 during repeated CI reruns).

Potential solutions

  1. In reusable-bot-ci-failure.yml, when the analysis returns zero failures, return/exit without commenting (guard the comment-posting step so it only runs when there is a failure analysis to report).
  2. Pin the analysis to the failing attempt: pass github.event.workflow_run.run_attempt from bot-ci-failure.yml to the reusable workflow and fetch that attempt's logs (/actions/runs/<run_id>/attempts/<run_attempt>/logs) instead of the default latest-attempt logs.
  3. Optionally, before analyzing, re-check the current run conclusion/attempt and skip if it is no longer failure (the failure was already retried away).

Solution 1 stops the noise immediately; solution 2 fixes the underlying race; solution 3 is a cheap extra guard.

Screenshots

Image

System Information:

  • Component: GitHub Actions reusable workflow .github/workflows/reusable-bot-ci-failure.yml (openwisp-utils@master), invoked from openwisp-monitoring/.github/workflows/bot-ci-failure.yml
  • Runner OS: ubuntu-latest (GitHub-hosted)
  • Bot account: openwisp-companion[bot]

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Fields

    No fields configured for Bug.

    Projects

    Status
    Backlog
    Status
    To do (general)
    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions