Skip to content
ci

test(acceptance): run a broadcast long enough for the slow faults to show #1648

Sign in to view logs

test(acceptance): run a broadcast long enough for the slow faults to show

test(acceptance): run a broadcast long enough for the slow faults to show #1648

Workflow file for this run

.github/workflows/ci.yml at 8901031
# Continuous integration.
#
# Runs on pull requests, on main, and on demand. Deliberately does NOT run the
# container acceptance suite on every commit — that suite builds an image and
# publishes several real streams into it, which takes minutes. It runs on a
# schedule and before a release instead, where its cost buys something.
name: ci
on:
# main only, NOT ["**"]. A PR branch fires both `push` and `pull_request`, so
# matching every branch here produced two runs per commit and left the
# concurrency group below to kill one of them. Those cancellations were
# harmless but they attach cancelled contexts to the head commit, which the
# PR reports as UNSTABLE and `gh pr checks` prints as "fail" -- indis-
# tinguishable at a glance from a real regression, and it cost someone a
# pointless seventeen-job rerun to find that out.
push:
branches: [main]
pull_request:
# The escape hatch for the case the old ["**"] was protecting: a branch with
# no PR open now gets no automatic CI, so run it here by hand. A draft PR
# does the same thing and keeps running on every push.
workflow_dispatch:
# The container suite is the only thing that catches base-image drift: an
# Alpine branch republishing FFmpeg at a new -r revision makes the pinned
# build fail, and that failure is the prompt to bump on purpose. Nothing in
# this repository changes when that happens, so only a timer finds it.
schedule:
- cron: "17 6 * * 1"
# One run per branch. A push that supersedes another cancels it rather than
# leaving both to finish -- which matters here because these suites publish real
# streams and wait on them, so a superseded run can sit for its full 20-minute
# ceiling consuming a runner nobody is waiting on.
concurrency:
# Still needed, but for a different reason than it used to be. The `push`
# trigger above no longer matches PR branches, so there are no push/pull
# twins left to deduplicate. What remains is the ordinary case: pushing twice
# to a branch with an open PR fires two `pull_request` runs, and the older one
# should die rather than sit for its full 20-minute ceiling publishing real
# streams nobody is waiting on.
#
# head_ref || ref_name rather than github.ref: head_ref is the source branch
# on a pull_request and empty on a push, ref_name is the bare branch name on a
# push. Together they give one stable key per branch across both events, which
# github.ref does not -- it is refs/heads/BRANCH on a push and
# refs/pull/N/merge on a pull_request.
group: ci-${{ github.head_ref || github.ref_name }}
cancel-in-progress: true
# LEAST PRIVILEGE, PER JOB. This was `contents: read` at the workflow level,
# which grants it to every job whether or not the job reads the repository --
# Sonar's githubactions:S8264. `{}` here means a job that declares nothing gets
# NOTHING, so a new job cannot inherit a token by being added; it has to ask.
# Each job below states what it needs, and `contents: read` is all any of them
# needs: nothing in this file writes to the repository, opens an issue, or
# pushes a package.
permissions: {}
jobs:
# Does this pull request contain anything any suite in here could break?
#
# THE EXPENSIVE HALF OF THIS WORKFLOW RAN ON MARKDOWN. Three documentation-only
# PRs on 2026-08-14 each fired all 28 checks -- the full acceptance matrix,
# three-OS builds, Docker and browser suites -- to validate files no job reads.
# One of them was a single file. See #350.
#
# WHAT THIS GATES, and #378 is the second half of it. #351/#357 stopped the
# acceptance matrix; the three-OS test matrix, the go job and the cross-compile
# kept running, which is most of the twenty minutes #350 was actually
# complaining about. All five now read this output:
#
# go go build, vet, test
# crossplatform test: ubuntu-latest / macos-latest / windows-latest
# ui ui typecheck, lint, build
# cross cross-compile all release targets
# acceptance acceptance: <thirteen suites>
#
# Every name in the right-hand column is a required status context in the
# branch-protection ruleset. That is the whole reason for the shape below.
#
# WHY THIS GATES STEPS AND NOT JOBS. A leg's context has to be added to the
# ruleset by hand or it reports without gating anything. A workflow-level
# `paths:` filter stops the workflow running at all, so a required check never
# reports and the pull request stays pending forever with no way to merge it.
#
# A JOB-LEVEL `if:` ON A MATRIX IS THE SAME BUG WEARING A DISGUISE, and #351
# shipped it. A skipped ordinary job does report, and a skip does satisfy the
# requirement -- but a skipped MATRIX job never expands its matrix, so the
# per-leg contexts are never created at all. The checks list showed one entry
# named literally `acceptance: ${{ matrix.suite }}`, the required
# contexts were absent rather than skipped, and #349 -- a documentation-only
# pull request, the exact case this was built for -- was unmergeable. Fifteen
# green checks and no way in.
#
# So the matrix always expands and every leg reports. What is conditional is
# the WORK: each step carries the `if:`, the job costs a runner allocation and
# nothing else, and the required context reports success either way.
#
# ONE SHAPE FOR ALL FIVE, INCLUDING THE THREE THAT HAVE NO MATRIX. `go`, `ui`
# and `cross` are ordinary jobs today, and a job-level `if:` on them would be
# correct today: a skipped ordinary job reports skipped and satisfies its
# requirement. It is not written that way, because the difference between the
# safe spelling and the outage is one `strategy:` block that nobody would think
# to connect to branch protection while adding it -- a Go-version matrix on
# `go`, a Node-version matrix on `ui`. The rule is therefore flat, has no
# exceptions to remember, and is enforced rather than remembered:
# internal/testenv/docsgate_test.go fails if this output is ever read from a
# job-level `if:`. The price is four runner allocations that do nothing on a
# documentation-only PR, against twenty minutes of compute they replace.
#
# IT FAILS TOWARD RUNNING. `code` is false only when EVERY changed path is
# documentation; anything unrecognised makes it true. A new top-level directory
# gets the full matrix until somebody decides otherwise, which is the right
# direction for a mistake to point.
#
# THIS FILE COUNTS AS CODE, for the reason the container-suites job already
# gives about itself: a change to how the suites run must run the suites.
changes:
name: which changes
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
code: ${{ steps.detect.outputs.code }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- id: detect
env:
EVENT: ${{ github.event_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
# Anything that is not a pull request runs everything. A push to main
# and a scheduled run have no diff to reason about.
if [ "$EVENT" != "pull_request" ]; then
echo "code=true" >> "$GITHUB_OUTPUT"
echo "not a pull request: running everything"
exit 0
fi
CHANGED="$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")"
echo "changed files:"; echo "$CHANGED" | sed 's/^/ /'
# Every line documentation? Then no suite can be affected by it.
if [ -n "$CHANGED" ] && ! grep -qvE '^(docs/|[^/]*\.md$)' <<<"$CHANGED"; then
echo "code=false" >> "$GITHUB_OUTPUT"
echo "documentation only: skipping the acceptance matrix"
exit 0
fi
echo "code=true" >> "$GITHUB_OUTPUT"
echo "contains code: running everything"
go:
name: go build, vet, test
permissions:
contents: read
# The documentation gate, #378. On every STEP below, never on this job --
# see the `changes` job for why that distinction is load-bearing and for
# what it cost the one time it was got wrong.
needs: changes
runs-on: ubuntu-latest
# THE COMPARISON IS AGAINST START-OF-STEP, NOT START-OF-JOB, AND THAT IS
# WHAT WAS WRONG HERE.
#
# The intent has always been right: keep this ceiling above the `go test
# -timeout` below, so a hung test dies by Go's clock and leaves a goroutine
# dump rather than by GitHub's, which leaves a bare "cancelled" and nothing
# to read. What the old value compared was 25 against 20 and called it five
# minutes of headroom.
#
# It is not, because `go test` does not start when the job does. gofmt,
# build, vet, the Windows vet, the route-coverage preflight and the
# internal/api coverage run first and take between six and seven minutes.
# So the test step had 25 - 6.5 = 18.5 minutes of wall clock to spend
# against its own 20-minute deadline: THIS ceiling always fired first, the
# diagnostic could never be produced, and the invariant the comment
# asserted had been false for as long as the preflights existed.
#
# Observed rather than theorised: three runs on 2026-09-01 died at 25m16s,
# 25m17s and 25m20s. The suite is not flaky, it is over budget -- the job
# sat on its ceiling and a few seconds of runner variance decided each one.
#
# So the value is now derived rather than picked: the steps before the test
# (~7m) plus the test's own deadline (20m) plus headroom for the runner to
# act on it (~8m). Raise it by the same arithmetic when a preflight is added
# -- and note the instruction below still stands, that raising `-timeout`
# itself means splitting the package first. ci-timeout_test.go fails if this
# number ever drops back under the sum.
timeout-minutes: 35
env:
# #187. The comment below the checkout has said since the beginning that a
# missing FFmpeg makes those tests SKIP, "which is worse, because the run
# goes green having checked nothing" -- and then nothing in this file made
# that impossible. This does. With it set, internal/testenv.FFmpegBinary
# fails and names the binary instead of skipping, so deleting the install
# step above turns the upload gate's tests red rather than silent.
#
# It is set on THIS job only. The crossplatform job installs FFmpeg from
# three different package managers and its post-install checks now assert
# ffprobe as well, but arming a hard failure on runner images I have not
# measured is how a gate becomes a revert; the full suite runs here, and
# here is where the guarantee is worth having. Widening it is a small
# change once someone has watched the three images pass with it.
POLYEMESIS_REQUIRE_FFMPEG: "1"
steps:
# Says out loud why a green check did no work, so a reader of the
# checks list is never left guessing whether this ran or no-opped.
# THE DOCS-ONLY PATH IS NOT A NO-OP ANY MORE, AND THAT WAS A REAL HOLE.
#
# The guards that check documentation against code are Go tests --
# internal/oauth/platforms_doc_drift_test.go reads docs/PLATFORMS.md,
# internal/api/api_docs_route_table_test.go reads docs/API.md, and so on.
# Every Go step below is gated on `code == 'true'`, so the ONE class of
# pull request that can drift the docs was the one class on which the
# drift guards never ran. That is how docs/UPGRADING.md came to tell an
# operator mid-migration that a CI-tested feature does not work, and how
# docs/MODULES.md came to name a base image and a Go version that no
# Dockerfile uses. #651.
#
# These tests read markdown and compare it to Go declarations. They need
# no ffmpeg, no database and no network, so they cost seconds -- which is
# why the docs-only path can afford to run them and could never afford
# the full suite.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: needs.changes.outputs.code != 'true'
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
if: needs.changes.outputs.code != 'true'
with:
go-version-file: go.mod
cache: true
- name: Documentation-only change, so only the doc-drift guards ran
if: needs.changes.outputs.code != 'true'
run: |
set -euo pipefail
# Discovered, not listed. A hardcoded package list is a second thing
# to keep in step with the tests, and the failure mode of letting it
# fall behind is silence -- exactly the shape of the bug this step
# exists to close. internal/testenv/docdrift_packages_test.go asserts
# this grep and the one it runs stay the same.
pkgs="$(grep -rlE 'docs/[A-Z][A-Za-z-]*\.md' --include='*_test.go' internal \
| xargs -n1 dirname | sort -u | sed 's|^|./|' | tr '\n' ' ')"
echo "doc-drift packages: ${pkgs}"
if [ -z "${pkgs// /}" ]; then
echo "::error::found no test package reading docs/*.md; the discovery grep has gone stale and this check would pass having run nothing"
exit 1
fi
# -run matching NOTHING exits 0, so counting what actually ran is the
# difference between a check and a green tick.
go test ${pkgs} -run 'Doc|Drift|Documented|Matches|Agrees|Restate' -v 2>&1 | tee /tmp/drift.log
ran="$(grep -c '^=== RUN' /tmp/drift.log || true)"
echo "doc-drift tests executed: ${ran}"
if [ "${ran:-0}" -lt 5 ]; then
echo "::error::only ${ran} doc-drift tests ran; the -run pattern no longer matches the guards, so this check is passing by finding nothing"
exit 1
fi
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: needs.changes.outputs.code == 'true'
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
if: needs.changes.outputs.code == 'true'
with:
go-version-file: go.mod
cache: true
# polyemesis shells out to ffmpeg and ffprobe, and a large part of the
# test suite asserts on real filter-graph behaviour rather than on
# strings. Without FFmpeg present those tests do not fail — they skip,
# which is worse, because the run goes green having checked nothing.
# WHY A STEP TIMEOUT, when the job already has one.
#
# For the same reason the go test timeout below is kept under it: whichever
# ceiling fires first decides how much you learn. A job timeout kills the
# runner and reports "cancelled" with no indication of which step was
# stuck -- and on 2026-08-04 an acceptance job spent its entire 20 minutes
# inside THIS step, never ran its suite, and looked exactly like the
# stream-waiting hangs issue #38 tracks. It was apt.
#
# Six minutes is far past a normal install (~30s) and far short of the
# job budget, so it fires only on a genuinely stuck mirror.
# KEYED ON THE PINNED ASSET, plus an epoch that can be bumped by hand.
#
# BtbN's `latest` release tag is ROLLING: the same asset name can serve a
# newer build over time. A cache keyed on the name therefore FREEZES the
# build CI runs, which is a deliberate trade and the right one here --
# the Dockerfile pins FFMPEG_VERSION=8.1.2-r0, so a CI FFmpeg that also
# stops moving is closer to what users get, not further from it. The
# weekly scheduled container suite is what catches upstream drift, and it
# builds the image rather than reading this cache.
#
# Bump the -v1 suffix to take a newer build on purpose.
- name: Cache FFmpeg
if: needs.changes.outputs.code == 'true'
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/ffcache
key: ffmpeg-${{ runner.os }}-n8.1-latest-linux64-gpl-8.1-v1
- name: Install FFmpeg
if: needs.changes.outputs.code == 'true'
timeout-minutes: 6
env:
FFCACHE: /tmp/ffcache
run: |
# Ubuntu's own package is 6.1.1, and the Docker image ships 8.1.2.
# Testing against 6.1.1 meant CI never exercised the FFmpeg users
# actually get, and the two differ in ways that matter: 6.1.1 accepts
# an 80-channel amerge where 8.1 stops at 64, and 6.1.1 cannot demux
# multitrack FLV at all. Pin the same 8.1 line the image uses.
set -euo pipefail
cd /tmp
# -c rather than a heredoc: inside a YAML block the heredoc body and
# its terminator are indented, which Python reads as a bad indent and
# bash does not accept as a delimiter.
# RETRIED, because this is one un-mirrored third-party download and
# every Linux job in this file makes it. It has hard-failed two jobs
# in one day -- acceptance-postprod and acceptance-pull -- both by
# sitting on a stalled connection until the 6-minute ceiling fired.
# A stuck mirror is exactly what that ceiling is for, but a single
# attempt turns a transient stall into a red job and a manual rerun.
#
# curl rather than urllib: --retry understands the difference between
# a connection that failed and one that is merely slow, and
# --retry-all-errors covers the 5xx a release CDN returns under load.
# The per-attempt --max-time is well inside the step ceiling, so three
# attempts still cannot outlast it -- the ceiling keeps its meaning.
# A CACHE HIT SKIPS THE DOWNLOAD ENTIRELY, and that is the point.
#
# This artefact failed SEVEN times in one day across Linux and Windows
# -- 503s and stalls from the release CDN -- while already wrapped in
# `curl --retry 3 --retry-all-errors` AND the three-attempt loop below.
# It has retries and still lands red, because a retry re-asks a host
# that is refusing. The cache stops asking.
#
# Same treatment the Playwright browser got: a pinned artefact need not
# be fetched on every run after the first.
if [ -x "$FFCACHE/ffmpeg" ] && [ -x "$FFCACHE/ffprobe" ]; then
echo "ffmpeg restored from cache"
else
for attempt in 1 2 3; do
# --proto/--proto-redir pin the whole exchange to https. -L
# follows redirects, and without these curl will happily follow an
# https -> http hop and fetch the binary CI is about to run over
# plaintext. SonarCloud githubactions:S6506 flagged the copy of
# this block added for the acceptance job; the other two had the
# same hole and are fixed here too, because leaving a known-bad
# copy beside a fixed one is how the fix gets reverted later by
# someone copying the wrong neighbour.
if curl -fsSL --proto '=https' --proto-redir '=https' \
--retry 3 --retry-all-errors --retry-delay 5 \
--connect-timeout 20 --max-time 90 \
-o /tmp/ff.tar.xz \
"https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-linux64-gpl-8.1.tar.xz"; then
break
fi
echo "ffmpeg download attempt $attempt failed"
[ "$attempt" = 3 ] && exit 1
sleep 5
done
mkdir -p ffbuild && tar xf ff.tar.xz --strip-components=1 -C ffbuild
mkdir -p "$FFCACHE"
cp ffbuild/bin/ffmpeg ffbuild/bin/ffprobe "$FFCACHE/"
fi
sudo install -m 0755 "$FFCACHE/ffmpeg" /usr/local/bin/ffmpeg
sudo install -m 0755 "$FFCACHE/ffprobe" /usr/local/bin/ffprobe
# Fail loudly rather than silently testing a build without SRT.
ffmpeg -hide_banner -protocols | tr ' ' '\n' | grep -qx srt
ffmpeg -hide_banner -version | head -1
# ffprobe TOO, and not because the install could half-succeed. #187:
# every test that proves the upload probe gate works starts by looking
# ffprobe up and skipping if it is absent, so an ffprobe that stopped
# arriving would have taken the whole gate's verification with it while
# this step still printed an ffmpeg version and passed. The one binary
# that was verified was the one the skips did not turn on.
ffprobe -hide_banner -version | head -1
# The gate is "gofmt named no files", and it has to rest on gofmt having
# RUN. It used to be `gofmt -l ./cmd ./internal | tee /tmp/fmt && test !
# -s /tmp/fmt`, and the default shell for a `run:` block is `bash -e {0}`
# -- no pipefail, unlike an explicit `shell: bash`. So the pipeline's
# status was tee's, gofmt's was discarded, and a gofmt that died on a
# parse error wrote nothing to /tmp/fmt and left `test ! -s` to report
# the tree as correctly formatted. Every way of failing to check produced
# the same green as checking and finding nothing.
#
# A command substitution under set -e instead: gofmt's own exit status
# now fails the step, and the emptiness of its output is a separate
# question asked afterwards.
- name: gofmt
if: needs.changes.outputs.code == 'true'
run: |
set -euo pipefail
unformatted=$(gofmt -l ./cmd ./internal)
if [ -n "$unformatted" ]; then
echo "these files are not gofmt'd:"
echo "$unformatted"
exit 1
fi
echo "gofmt: clean"
- if: needs.changes.outputs.code == 'true'
run: go build ./...
- if: needs.changes.outputs.code == 'true'
run: go vet ./...
# WINDOWS TOO, because `go vet ./...` above never sees a _windows.go file.
# internal/supervisor and internal/fsperm both have Windows-only sources
# that use unsafe, and they were type-checked by nothing in this repo until
# now -- a build tag is enough to hide a package from every gate.
#
# It will not catch every unsafe misuse: vet's unsafeptr check looks for
# uintptr -> Pointer, and the violation found in #440 is the other
# direction. It does catch what vet catches, on files that had no gate at
# all, which is the gap being closed rather than the whole class.
- name: go vet, for the Windows build
# The documentation gate goes on the STEP, never on the job. A skipped
# matrix job does not expand its matrix, so its per-leg required contexts
# are never created -- absent rather than skipped -- and the PR can never
# satisfy branch protection. #351 shipped that and #349 sat unmergeable at
# fifteen green checks until #357 undid it. internal/testenv's docs gate
# test caught this step missing the condition.
if: needs.changes.outputs.code == 'true'
run: GOOS=windows go vet ./...
# EVERY SHELL SCRIPT PARSES, which was true of none of them as far as CI
# was concerned.
#
# This repository has 44 files under scripts/*.sh and 25 of them are
# acceptance suites that run only in the acceptance matrix, on dispatch,
# or not at all. scripts/acceptance-multistream.sh is the extreme case:
# 70 KB, run NOWHERE in CI by deliberate decision (docs/TESTING.md
# explains it -- with credentials it publishes to a real account), and yet
# three Go tests cite its behaviour as the thing they are keeping in step
# with. A syntax error in it would have been found by whoever next ran it
# by hand, which is a schedule nobody controls.
#
# `bash -n` is the cheapest possible half of that: it parses without
# executing, so it needs no FFmpeg, no ports, no credentials and no
# network, and the whole directory finishes in well under a second. It
# cannot tell you the script is CORRECT -- only that it is not garbage --
# and that is exactly the class of breakage an unrun script accumulates,
# because nothing else looks at it at all.
#
# release.yml already did this for two scripts by name. Naming scripts is
# how you end up covering the two somebody remembered; the glob covers the
# ones nobody did.
#
# `make sh-syntax` rather than an inline loop, so `make check` runs the
# same thing -- see internal/testenv/checkparity_invocation_test.go.
- name: Every shell script parses
if: needs.changes.outputs.code == 'true'
timeout-minutes: 2
run: make sh-syntax
# THE PREFLIGHT'S OWN LIVENESS, and it runs HERE because it was not
# running anywhere.
#
# internal/api forces its route coverage ledger through a SECOND m.Run with
# the caller's -run, -skip and -count set aside, so that no test filter can
# switch the ledger off. `make preflight-guard` proves that mechanism is
# still wired by running the package under each of those three switches and
# requiring the preflight's marker to print anyway.
#
# It was reachable only as a prerequisite of `make test`, and this workflow
# invokes `go test` directly -- here and in the crossplatform job -- so the
# guard existed on developer machines and nowhere else. A local-only gate
# described in a comment as a gate is the species this whole change is
# about. ~4s; it runs the package three times with a filter that selects
# almost nothing.
- name: The route-coverage preflight is still wired
if: needs.changes.outputs.code == 'true'
run: make preflight-guard
# THE OTHER HALF OF THE SAME MECHANISM. #217/#223.
#
# The preflight runs in a SECOND m.Run rather than a first, because the
# coverage profile is written by testing.M.after under m.afterOnce, on the
# way out of whichever pass returns first. While the forced pass went
# first, `go test -cover ./internal/api` reported 22.0% for zero tests, for
# one test and for the whole suite alike -- a constant wearing a
# percentage, which #219 already lost a round to.
#
# Reordering is a one-line change and reverting it is a one-line change,
# so it needs a running gate rather than a comment. This one runs the
# package under three different selections and requires the numbers to
# differ. ~60s, dominated by the unfiltered probe, which is the number
# anybody would actually quote.
# THE STEP TIMEOUT THE FOUR BELOW GOT AND THIS ONE DID NOT, and on
# 2026-08-14 this is the step that hung: 24 minutes against a measured
# 98-114s, until the job's own ceiling cancelled it and named nothing.
#
# It can hang for a reason the note at :360 already gives about the
# watchdog -- "a background process holding the suite's stdout ... does
# not merely fail to report, it becomes the hang". The guard captures each
# probe with `out="$(go test ...)"`, and command substitution blocks until
# the pipe has no writers left, NOT until go test exits. A test that
# leaks a child holding inherited stdout hangs the capture after the test
# binary is gone and after Go's own timeout has already fired, so Go's
# diagnostic cannot be the backstop here. Only this can.
#
# 14 rather than something tighter, so the ordering the file argues for
# everywhere else holds: three probes at -timeout 4m is 12m of worst case,
# and Go has to be able to win that race and print the goroutine dump
# before this fires. Below the job's 35 either way, which is the property
# being claimed.
- name: internal/api coverage measures the tests, not the preflight
if: needs.changes.outputs.code == 'true'
timeout-minutes: 14
run: make coverage-instrument-guard
# -race because the engine reconciles from several goroutines and a data
# race here shows up in production as a stream that stops for one viewer.
#
# -timeout because Go's default is 10 minutes PER PACKAGE and internal/db
# runs close to it on a slow runner -- see the cross-platform job below
# for the measurements. -race makes every one of them slower still.
#
# Kept BELOW timeout-minutes on purpose. Go's timeout panics with a
# goroutine dump naming the test that was running; the job timeout just
# kills the runner and tells you nothing. Whichever fires first decides
# how much you learn, so Go's has to.
# POLYEMESIS_LEDGER=strict makes internal/api's route coverage ledger run
# its counterpart proofs from inside the ledger test as well as from their
# own. The registry's whole value is that an excuse cannot discharge on a
# test NAME, only on bytes that actually left the process -- and a `-run`
# filter matching one of the two tests and not the other would leave every
# counterpart undischarged while still printing ok. The full suite runs
# here with no filter, so strict mode costs nothing and closes that door.
- if: needs.changes.outputs.code == 'true'
# 20m, RAISED FROM 15m ON EVIDENCE THAT THIS IS ACCUMULATION AND NOT
# A HANG -- which is the only reason raising a timeout is ever the right
# move rather than the lazy one.
#
# internal/api hit the 15m wall on 2026-08-16 and reported 900.034s. The
# panic named the test that was running, which is exactly why Go's
# timeout is kept below this job's timeout-minutes: 35, and it read
#
# running tests:
# TestAMutationReconcilesEveryProgrammeRatherThanTheDefault (0s)
#
# ZERO SECONDS. Nothing was stuck; the package had simply spent fifteen
# minutes getting there. A hang would have named a test with a large
# number beside it, and the fix for that is never a bigger budget.
#
# internal/api is the largest suite in the tree -- the route ledger
# alone drives every method-pattern pair against a live fixture -- and
# the race detector multiplies all of it. Tests that were spending real
# wall clock on a pacing constant were trimmed in the same change, which
# is the part that should have been unnecessary; this is the part that
# was overdue.
#
# Still fifteen minutes below the job's own timeout, so Go keeps its
# goroutine dump. If this needs raising again, split the package first.
run: POLYEMESIS_LEDGER=strict go test -race -timeout 20m ./...
# The acceptance suites' shared diagnostic helpers, tested here rather
# than in their own job: it is pure shell, needs nothing installed and
# finishes in about a second, so a separate runner would cost more to
# schedule than to run.
#
# It earns a place in CI because the claim it guards is one a reader will
# act on. lib-observe.sh prints "the ceiling is too low; this is not a
# product failure" -- and a diagnostic that says that about a REAL failure
# sends the next person to the wrong component with the authority of a
# printed conclusion. The negative case is the one under test.
#
# THE FOUR SCRIPT STEPS IN THIS JOB CARRY STEP TIMEOUTS, for the reason
# :327-334 and :393-396 give for the two smoke steps: a job timeout names
# nothing, a step timeout names the step. Each of these four runs a shell
# harness that starts and kills processes, so each can hang exactly the way
# #179 hung, and behind this job's timeout-minutes: 35 a hang here costs 35
# minutes and reports "the job was cancelled".
#
# HONEST ABOUT THE NUMBERS: only one of the four has a measurement --
# test-lib-cleanup.sh at 35s, up from 20s when its SIGTERM-deaf case was
# added. The other three were not re-measured here, because running the
# teardown harnesses on a shared developer machine kills processes
# belonging to whatever else is running on it. So these are blast-radius
# ceilings, not fitted bounds: sized to be unreachable by a healthy run and
# far below the job's 35, which is the only property being claimed.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/test-lib-observe.sh
timeout-minutes: 6
# The acceptance suites' own deadline, which fires below this workflow's
# timeout-minutes so a hung suite reports instead of being cancelled.
# Tested here for the reason above and one that is specific to it: the
# watchdog is a background process holding the suite's stdout, so a bug
# in it does not merely fail to report -- it becomes the hang.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/test-lib-watchdog.sh
timeout-minutes: 6
# The SBOM guard, which only ever runs for real inside a release. That is
# exactly how it came to be wrong: it asserted a floor of 100 against a
# document containing 437, so it could not fail on the scenario its own
# comment described, and no one could discover that without cutting a
# release. Running its tests on every PR is the part that makes the fix
# a fix rather than a better guess. Pure jq and shell, about a second.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/test-sbom-guard.sh
timeout-minutes: 5
# release.yml's gates, for exactly the reason above and one worse. They
# only ever run for real inside a release, and that is how changelog-gate
# came to exit 0 on a workflow_dispatch that publishes -- with a comment
# asserting "Nothing here would be published either way", pointing at the
# expression that makes it false. The fixtures drive the real step bodies,
# read out of release.yml rather than transcribed: a temporary git
# repository with an annotated and a lightweight tag, a CHANGELOG dated
# today and dated last week, and the jq selector against fixture run
# listings. Pure shell, jq and git; about a second.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/test-release-gates.sh
timeout-minutes: 5
# The termination guard, and its tests, alongside the SBOM pair above for
# the same reason and one of its own. #210: after #179/#180 the class had
# ONE gate and its jurisdiction was .github/workflows -- a fixture carrying
# #179's body verbatim was dropped into that test's green directory and
# passed. For scripts/ there was no gate at all, which is how
# acceptance-mqtt.sh kept the exact shape that had just been rewritten in
# three other suites. Found by a sweep, not by CI.
#
# Both halves run: the guard against the real tree, and its own red/green
# fixtures. The guard without its fixtures is a check nobody has watched
# fail; the fixtures without the guard oblige nothing. Pure shell, under a
# second each.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/termination-guard.sh
timeout-minutes: 5
- if: needs.changes.outputs.code == 'true'
run: ./scripts/test-termination-guard.sh
timeout-minutes: 5
# The OBS container entrypoint's stop path. #208 was FILED rather than
# fixed on the grounds that it could not be run -- "a blind edit to a
# container entrypoint has two ways to go wrong that a review cannot see".
# Both ways are about a process, not about OBS, so the stop logic moved
# into scripts/obs/lib-stop.sh and this drives it with `sleep` and a
# SIGTERM-deaf stand-in. No OBS, no Xvfb, no container, about six seconds.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/test-obs-stop.sh
timeout-minutes: 6
# Outbound webhooks over a real socket. IN THIS JOB RATHER THAN THE
# acceptance MATRIX, because it needs neither FFmpeg nor `make build` --
# the far end is an http.Server the driver starts on a loopback port, so
# Go alone is the whole dependency and the suite finishes in about 20
# seconds. Scheduling it beside the FFmpeg suites would cost a six-minute
# apt step to test a package that never touches FFmpeg.
#
# ON EVERY PUSH RATHER THAN WEEKLY, which is the opposite of chat-live.yml
# and for the reason that workflow gives for its own schedule: that suite
# measures THEIR behaviour, so its failures arrive with no commit of ours.
# This one measures OURS, contacts nothing outside the runner, and has
# nothing to be flaky about. Its credentialed step skips here; no
# POLY_HOOKS_URL is configured for this workflow and none is needed --
# 29 of its 31 checks run without one.
- if: needs.changes.outputs.code == 'true'
run: ./scripts/acceptance-hooks.sh
timeout-minutes: 6
# The acceptance suites' shared teardown. Same argument as the guard
# above, and a sharper one: this teardown was silently leaking the ingest
# port on every run, which made acceptance-failover fail 10 times out of
# 20 back to back. A harness that measures itself is worse than no
# harness, because a real regression gets dismissed as "that suite is
# flaky". Needs lsof, which this job does not otherwise install.
- name: ./scripts/test-lib-cleanup.sh
if: needs.changes.outputs.code == 'true'
timeout-minutes: 10
run: |
# RETRIED. A mirror that is slow rather than down turns one unlucky
# fetch into a red run and a rerun, which is #430's complaint in
# miniature. Three attempts with a widening pause; the job ceiling
# above still bounds the whole thing.
for attempt in 1 2 3; do
sudo apt-get install -y --no-install-recommends lsof && break
echo "apt attempt $attempt failed"
[ "$attempt" = 3 ] && exit 1
sleep $(( attempt * 5 ))
done
./scripts/test-lib-cleanup.sh
# The same build and test, on the operating systems polyemesis actually ships
# for.
#
# `make release` cross-compiles darwin and windows binaries and every job
# above then runs on ubuntu, so until this existed the Windows and macOS
# builds were proven to COMPILE and never proven to WORK. That gap matters
# here more than in most projects: this codebase shells out to a child
# process, manages process groups and signals, binds sockets, and builds file
# paths that end up on an FFmpeg command line. Every one of those is a place
# where Linux and Windows genuinely differ.
#
# No -race: it needs cgo, and these builds are CGO_ENABLED=0 on purpose. The
# ubuntu job above runs the race detector, which is where a data race would
# show up anyway -- the goroutine scheduling is not what differs across these
# platforms.
crossplatform:
name: "test: ${{ matrix.os }}"
permissions:
contents: read
# The documentation gate, #378. On every STEP below, never on this job --
# see the `changes` job for why that distinction is load-bearing and for
# what it cost the one time it was got wrong.
needs: changes
runs-on: ${{ matrix.os }}
# Above the 15m per-package go test timeout below. Windows needs the room:
# internal/db alone has been measured at 265-300s and once past 600s.
timeout-minutes: 30
strategy:
# Never cancel siblings: when Windows and macOS both break, both answers
# are wanted, and fail-fast would turn one fix into two round trips.
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
# Says out loud why a green check did no work, so a reader of the
# checks list is never left guessing whether this ran or no-opped.
- name: Documentation-only change, so this job did no work
if: needs.changes.outputs.code != 'true'
run: |
echo "::notice title=test: ${{ matrix.os }}::this check did no work -- every changed path was documentation. See the 'which changes' job."
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: needs.changes.outputs.code == 'true'
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
if: needs.changes.outputs.code == 'true'
with:
go-version-file: go.mod
cache: true
# FFmpeg on every runner, for the reason the ubuntu job gives: without it
# the measurement tests SKIP rather than fail, and the run goes green
# having checked nothing. A cross-platform job that skipped every test
# that touches FFmpeg would be the most misleading kind of green.
# Same reasoning as the go job's install step.
# KEYED ON THE PINNED ASSET, plus an epoch that can be bumped by hand.
#
# BtbN's `latest` release tag is ROLLING: the same asset name can serve a
# newer build over time. A cache keyed on the name therefore FREEZES the
# build CI runs, which is a deliberate trade and the right one here --
# the Dockerfile pins FFMPEG_VERSION=8.1.2-r0, so a CI FFmpeg that also
# stops moving is closer to what users get, not further from it. The
# weekly scheduled container suite is what catches upstream drift, and it
# builds the image rather than reading this cache.
#
# Bump the -v1 suffix to take a newer build on purpose.
- name: Cache FFmpeg
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/ffcache
key: ffmpeg-${{ runner.os }}-n8.1-latest-linux64-gpl-8.1-v1
# Linux only: macOS installs from brew and Windows caches its own
# zip below. Without this the step runs on all three and stores an
# empty directory under two keys that nothing ever reads.
if: runner.os == 'Linux' && needs.changes.outputs.code == 'true'
- name: Install FFmpeg (Linux)
timeout-minutes: 6
if: runner.os == 'Linux' && needs.changes.outputs.code == 'true'
env:
FFCACHE: /tmp/ffcache
run: |
# Ubuntu's own package is 6.1.1, and the Docker image ships 8.1.2.
# Testing against 6.1.1 meant CI never exercised the FFmpeg users
# actually get, and the two differ in ways that matter: 6.1.1 accepts
# an 80-channel amerge where 8.1 stops at 64, and 6.1.1 cannot demux
# multitrack FLV at all. Pin the same 8.1 line the image uses.
set -euo pipefail
cd /tmp
# -c rather than a heredoc: inside a YAML block the heredoc body and
# its terminator are indented, which Python reads as a bad indent and
# bash does not accept as a delimiter.
# RETRIED, because this is one un-mirrored third-party download and
# every Linux job in this file makes it. It has hard-failed two jobs
# in one day -- acceptance-postprod and acceptance-pull -- both by
# sitting on a stalled connection until the 6-minute ceiling fired.
# A stuck mirror is exactly what that ceiling is for, but a single
# attempt turns a transient stall into a red job and a manual rerun.
#
# curl rather than urllib: --retry understands the difference between
# a connection that failed and one that is merely slow, and
# --retry-all-errors covers the 5xx a release CDN returns under load.
# The per-attempt --max-time is well inside the step ceiling, so three
# attempts still cannot outlast it -- the ceiling keeps its meaning.
# A CACHE HIT SKIPS THE DOWNLOAD ENTIRELY, and that is the point.
#
# This artefact failed SEVEN times in one day across Linux and Windows
# -- 503s and stalls from the release CDN -- while already wrapped in
# `curl --retry 3 --retry-all-errors` AND the three-attempt loop below.
# It has retries and still lands red, because a retry re-asks a host
# that is refusing. The cache stops asking.
#
# Same treatment the Playwright browser got: a pinned artefact need not
# be fetched on every run after the first.
if [ -x "$FFCACHE/ffmpeg" ] && [ -x "$FFCACHE/ffprobe" ]; then
echo "ffmpeg restored from cache"
else
for attempt in 1 2 3; do
# --proto/--proto-redir pin the whole exchange to https. -L
# follows redirects, and without these curl will happily follow an
# https -> http hop and fetch the binary CI is about to run over
# plaintext. SonarCloud githubactions:S6506 flagged the copy of
# this block added for the acceptance job; the other two had the
# same hole and are fixed here too, because leaving a known-bad
# copy beside a fixed one is how the fix gets reverted later by
# someone copying the wrong neighbour.
if curl -fsSL --proto '=https' --proto-redir '=https' \
--retry 3 --retry-all-errors --retry-delay 5 \
--connect-timeout 20 --max-time 90 \
-o /tmp/ff.tar.xz \
"https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-linux64-gpl-8.1.tar.xz"; then
break
fi
echo "ffmpeg download attempt $attempt failed"
[ "$attempt" = 3 ] && exit 1
sleep 5
done
mkdir -p ffbuild && tar xf ff.tar.xz --strip-components=1 -C ffbuild
mkdir -p "$FFCACHE"
cp ffbuild/bin/ffmpeg ffbuild/bin/ffprobe "$FFCACHE/"
fi
sudo install -m 0755 "$FFCACHE/ffmpeg" /usr/local/bin/ffmpeg
sudo install -m 0755 "$FFCACHE/ffprobe" /usr/local/bin/ffprobe
# Fail loudly rather than silently testing a build without SRT.
ffmpeg -hide_banner -protocols | tr ' ' '\n' | grep -qx srt
ffmpeg -hide_banner -version | head -1
- name: Install FFmpeg (macOS)
if: runner.os == 'macOS' && needs.changes.outputs.code == 'true'
# BREW CANNOT SUPPLY 8.1, AND THAT IS RECORDED RATHER THAN HIDDEN.
# `ffmpeg` is 9.0 and there is no `ffmpeg@8` formula -- the versioned ones
# jump from 7.1 straight to 9.0 -- and BtbN publishes no macOS build. So
# this arm cannot have the parity Linux and Windows now share, and macOS
# tests a MAJOR VERSION AHEAD of what the image ships.
#
# Left as brew deliberately: 7.1 would be behind rather than ahead, and
# building 8.1 from source costs more than this job is worth. The version
# is printed by the step below, so the divergence shows up in every run
# instead of being discovered by a test failing oddly.
run: brew install ffmpeg
# Windows pays the same CDN tax as Linux -- this artefact failed twice
# here today -- so it gets the same treatment. A separate key because it
# is a different asset, and a separate step because the Linux cache above
# is gated to runner.os == 'Linux'.
- name: Cache FFmpeg (Windows)
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
if: runner.os == 'Windows' && needs.changes.outputs.code == 'true'
with:
path: ${{ runner.temp }}/ffcache
key: ffmpeg-${{ runner.os }}-n8.1-latest-win64-gpl-8.1-v1
- name: Install FFmpeg (Windows)
if: runner.os == 'Windows' && needs.changes.outputs.code == 'true'
# THE SAME BUILD THE LINUX ARM INSTALLS, not choco, for two reasons and
# the second is the important one.
#
# Reliability: `choco install ffmpeg` took a 504 from
# community.chocolatey.org and left the next step reporting "'ffmpeg' is
# not recognized" -- a red job on an install that silently did not happen,
# and the third distinct FFmpeg-install failure this file saw in one day.
#
# VERSION PARITY, which is what actually matters. The Linux arm pins n8.1
# and says why: the image ships 8.1.2, and 6.1.1 "accepts an 80-channel
# amerge where 8.1 stops at 64, and cannot demux multitrack FLV at all".
# That argument is about FFmpeg, not about Linux. choco resolved to
# whatever it held, so this job tested a version nobody ships while
# carrying a comment explaining why that is wrong.
#
# BtbN publishes ffmpeg-n8.1-latest-win64-gpl-8.1.zip from the SAME
# release the Linux arm downloads: one source, one version, one retry.
run: |
$ErrorActionPreference = 'Stop'
$url = 'https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-win64-gpl-8.1.zip'
$cache = "$env:RUNNER_TEMP\ffcache"
# A CACHE HIT SKIPS THE DOWNLOAD. See the Linux cache step for the
# argument: this artefact failed seven times in one day across both
# platforms while already carrying retries, and a retry re-asks a host
# that is refusing.
if (Test-Path "$cache\ffmpeg.exe") {
Write-Host 'ffmpeg restored from cache'
} else {
for ($i = 1; $i -le 3; $i++) {
try { Invoke-WebRequest -Uri $url -OutFile "$env:RUNNER_TEMP\ff.zip" -TimeoutSec 90; break }
catch { if ($i -eq 3) { throw }; Write-Host "ffmpeg download attempt $i failed"; Start-Sleep -Seconds 5 }
}
Expand-Archive -Path "$env:RUNNER_TEMP\ff.zip" -DestinationPath "$env:RUNNER_TEMP\ff" -Force
$src = (Get-ChildItem "$env:RUNNER_TEMP\ff" -Filter bin -Recurse -Directory | Select-Object -First 1).FullName
if (-not $src) { throw 'no bin/ directory in the FFmpeg archive' }
New-Item -ItemType Directory -Force -Path $cache | Out-Null
Copy-Item "$src\*" -Destination $cache -Force
}
$bin = $cache
if (-not (Test-Path "$bin\ffmpeg.exe")) { throw 'no ffmpeg.exe after install' }
# GITHUB_PATH, not $env:PATH: the latter dies with this step, which is
# exactly how the choco failure stayed invisible until the NEXT step
# could not find the binary.
Add-Content -Path $env:GITHUB_PATH -Value $bin
# BOTH binaries, on all three. #187: the skip helpers key off ffprobe and
# this step only ever ran ffmpeg, so the binary that decides whether the
# upload gate's tests run at all was the one nothing here looked at. The
# go job additionally sets POLYEMESIS_REQUIRE_FFMPEG so a missing one is a
# test failure rather than a skip; this job does not, deliberately -- see
# the note on that env block.
#
# TWO STEPS rather than a two-line one. The default shell on the Windows
# runner is pwsh, whose wrapper exits with the LAST command's status, so a
# two-line block would have reported ffprobe's result and discarded
# ffmpeg's -- the same discarded-exit-status shape as the gofmt pipeline in
# the go job. A step boundary checks each one.
- name: FFmpeg version
if: needs.changes.outputs.code == 'true'
run: ffmpeg -hide_banner -version
- name: ffprobe version
if: needs.changes.outputs.code == 'true'
run: ffprobe -hide_banner -version
- if: needs.changes.outputs.code == 'true'
run: go build ./...
- if: needs.changes.outputs.code == 'true'
run: go vet ./...
# Go's default timeout is 10 minutes PER PACKAGE, and internal/db lands
# close enough to it on Windows that ordinary runner variance decides the
# build. Measured on ONE unchanged tree: 264s, 297s, and once past 600s,
# where it failed -- a commit whose identical code had passed an hour
# earlier.
#
# The cause is not a slow test but a slow platform. internal/db runs
# against modernc.org/sqlite, a pure-Go SQLite that is roughly 40x slower
# on a Windows runner than on a developer machine: ~6s locally against
# ~265-300s there.
#
# What that cost actually WAS: schema DDL plus six migrations, executed
# once per test rather than once per package. The test helpers now build
# one migrated database and copy the file. Measured on this runner after
# that change, against the 265-300s above:
#
# internal/db 46s
# internal/api 69s
# internal/engine 23s
# internal/recording 7s
#
# The whole windows-latest job is now under three minutes, so the ~40x
# platform ratio that made this package a coin flip against Go's
# per-package ceiling is gone -- it is the same pure-Go SQLite on the same
# slow filesystem, asked to do the work once instead of 181 times.
#
# 15m stays. It is now ~20x the slowest package rather than ~3x, which is
# slack rather than a reason to retune: the failure it exists for is a
# test that HANGS, and that failure does not get less likely because the
# suite got faster.
#
# 15m is well past any observed Windows run for the slowest package, and it is
# deliberately BELOW this job's timeout-minutes. Go's timeout panics with
# a goroutine dump naming the running test; the job timeout just kills the
# runner and tells you nothing. Whichever fires first decides how much you
# learn, so Go's has to.
# THROUGH scripts/cmd/gotest, which changes no verdict and adds one
# distinction: a Go RUNTIME abort is annotated as one rather than read as
# a flaky test.
#
# #440 has fired three times here in many hundreds of runs, with three
# different runtime messages, and has never reproduced. Each time the only
# thing separating it from an ordinary failure was somebody noticing that
# the line said `fatal error:` and not `--- FAIL:`. Two were nearly waved
# through. The issue's own conclusion is that the SAMPLE SIZE is the
# instrument -- and an instrument that depends on a human reading three
# thousand lines of log is not one.
#
# All three runners, not just Windows. The crash has only ever been seen
# on Windows, and "only ever been seen" is exactly the claim that needs
# something watching the other two to stay true.
- if: needs.changes.outputs.code == 'true'
run: go run ./scripts/cmd/gotest -timeout 15m ./...
# THE RACE DETECTOR, ON WINDOWS ONLY, ON THE ONE PACKAGE THAT CRASHES.
#
# #440 is an intermittent `fatal error: found pointer to free object` in
# internal/engine on windows-latest. The runtime's own message names the two
# diagnostics that would explain it -- "try -d=checkptr or -race" -- and
# neither has ever been run on Windows, because the only race job in this
# file is the ubuntu one. That is precisely the blind spot: the crash is
# Windows-only, and so is the code most likely to be responsible.
#
# -race ALONE, because -race implies -d=checkptr (cmd/compile's flag setup
# does this on every platform), so one flag delivers both. It is also the
# stronger of the two: the detector reports a racing pair whenever the two
# accesses execute, where checkptr only fires on an executed bad conversion
# and is blind to a torn interface or slice write.
#
# ONE PACKAGE, not ./..., because this is a diagnostic for a named crash
# rather than a new gate. internal/engine is where the crash happens and it
# is the package whose tests stand up hubs, supervisors and a database at
# once. Widening it is a decision to take on evidence, not by reflex.
#
# continue-on-error, deliberately: a data race that has been latent for
# months must not turn every Windows run red before anyone has seen what it
# says. The point is to make it SPEAK. When it names something, that finding
# gets its own change and this becomes a gate.
- name: race detector on the package that crashes (#440)
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
continue-on-error: true
timeout-minutes: 20
run: go test -race -timeout 15m ./internal/engine/
# Build and RUN it. Compiling proves the code is valid for the platform;
# it does not prove the process comes up.
#
# The failures this catches are the ones a compiler cannot: a data
# directory created with a path separator the OS does not accept, a
# listener that binds differently, an embedded asset that resolves on one
# filesystem and not another. `make release` already cross-compiles these
# targets, so before this the Windows binary had never been executed by
# anything.
#
# bash on all three: windows-latest ships Git Bash, so one script serves
# every runner rather than a PowerShell fork nobody maintains.
#
# Built in a step of its own so each half carries a ceiling that means
# something (#198). These two things are unrelated: the build is the long
# pole on a cold Windows cache, and the serve check is 2-4s measured
# (n=21). Sharing one step forced its timeout to be sized for the build,
# which put a 10-minute ceiling on a 3-second assertion and hid "the
# Windows build cache was cold today" inside a step called "check it
# serves". Split, the timing column says which one it was.
#
# -cover, so the smoke run MEASURES itself. See the covdata step at the
# end of this job for what that is for and what it costs.
- name: Build the smoke binary
if: needs.changes.outputs.code == 'true'
timeout-minutes: 8
shell: bash
run: go build -cover -o polyemesis-smoke ./cmd/polyemesis
# timeout-minutes because this step BACKGROUNDS A PROCESS, and a step that
# holds a pid can wait on something that never comes. 2 against a measured
# 2-4s (n=21) is still 30x headroom now that the build is not inside it.
# Deliberately below the job's timeout-minutes: 30, per the discipline at
# :154-158 and :305-311 -- whichever timeout fires first decides how much
# you learn, so the narrowest one has to.
- name: Start the server and check it serves
if: needs.changes.outputs.code == 'true'
timeout-minutes: 2
shell: bash
run: |
set -uo pipefail
mkdir -p covdata/serve
GOCOVERDIR=covdata/serve ./polyemesis-smoke -addr 127.0.0.1:8123 -data ./smoke-data -log warn &
pid=$!
# THE CLEANUP IS A TRAP, AND THE VERDICT IS EMITTED BEFORE IT RUNS.
#
# This step used to kill the server and then block in an unbounded
# shell `wait` on the server pid before printing its answer. That wait
# never returns against a `kill` Git-Bash cannot deliver to a native
# .exe: it hung for 28 minutes and was cancelled by the job timeout,
# discarding an answer the loop below had already decided in 2 seconds.
# Establishing that the branch was innocent then took building both
# refs, hashing binaries and surveying 14 runs, to recover a fact this
# step knew and threw away.
#
# A trap covers more than the old inline kill did (any early exit, not
# just the one path), and best-effort is sufficient: nothing later in
# this job binds 8123, polyemesis-smoke is built once and never
# rebuilt so there is no Windows sharing violation to fear, and the
# runner VM is destroyed at the end of the job.
trap 'kill "$pid" 2>/dev/null || true' EXIT
ok=no
for _ in $(seq 1 40); do
sleep 0.5
if curl -fsS http://127.0.0.1:8123/api/v1/health >/dev/null 2>&1; then ok=yes; break; fi
done
if [ "$ok" != yes ]; then
echo "the server never answered /health on ${{ matrix.os }}"
exit 1
fi
echo "server started and served /health on ${{ matrix.os }}"
# Push an actual broadcast through it, on every platform.
#
# /health proves the process comes up. It does not prove the product
# works: that needs a stream arriving, a layout being probed, two
# destinations compiling different routing graphs, FFmpeg children being
# spawned and supervised, and audio landing in a container. Those are the
# parts built on process groups, signals and file paths -- precisely
# where Windows differs -- so "it serves /health on Windows" was never
# the claim worth making.
#
# The stream is injected into the relay hub rather than pushed over SRT,
# because libsrt is not guaranteed in a runner's FFmpeg (Homebrew's has
# none). That substitutes only the ingest hop; everything downstream is
# the real path. SRT ingest itself is covered by the acceptance suites on
# ubuntu.
#
# Verification is a measurement, not an exit status: each destination
# gets a different pair of the three input tones, and the check reads
# per-band energy back out of the file. A destination that silently
# carried the wrong mix would pass any check that only asked whether
# FFmpeg exited 0.
#
# timeout-minutes for the same reason as the step above: it backgrounds a
# process. 8 against a measured 91-93s (n=21) is roughly 5x headroom, and
# it is below the job's 30 on purpose -- a step timeout names the step
# that hung, a job timeout names nothing.
- name: Push a broadcast through it and measure the output
if: needs.changes.outputs.code == 'true'
timeout-minutes: 8
shell: bash
run: |
set -uo pipefail
rm -rf ./data
mkdir -p covdata/bcast
GOCOVERDIR=covdata/bcast ./polyemesis-smoke -addr 127.0.0.1:8099 -data ./data -log warn > bcast-server.log 2>&1 &
pid=$!
# Same shape as the step above: a trap, and the verdict before the
# cleanup. The shell `wait` that used to sit between `rc` and the
# report was unbounded, and on Windows it is unbounded against a kill
# that may never land.
#
# THE PREVIOUS VERSION OF THIS COMMENT SAID "this step is the last in
# the job", AND THAT WAS FALSE. `Upload broadcast artefacts on failure`
# follows it in this same job and uploads bcast-server.log and
# data/recordings/ -- the two paths the server still holds open. The
# trap fires when this step ends, so on the failure path the artefacts
# used to be read while the writer was live, and on Windows the kill
# may never land at all. Low impact (if-no-files-found: ignore, failure
# path only) but the justification rested on an untrue claim, so the
# failure path below now does the work the claim assumed: verdict
# first, then a BOUNDED stop, then the diagnostics.
#
# The trap stays as the backstop for every other exit path, and
# best-effort is still sufficient there: nothing later binds 8099, the
# binary is built once and never rebuilt so there is no Windows sharing
# violation to fear, and the runner VM is destroyed at the end.
trap 'kill "$pid" 2>/dev/null || true' EXIT
go run scripts/smoketest.go
rc=$?
if [ "$rc" -ne 0 ]; then
# THE VERDICT, BEFORE ANY CLEANUP. #179 is what happens when this
# ordering is the other way round.
echo "the broadcast smoke test failed on ${{ matrix.os }} (rc=$rc)"
# Now quiesce the writer, BOUNDED, so the log and the recordings the
# upload step is about to collect are not being appended to while it
# reads them. 10s against a server that shuts down in well under one;
# the ceiling exists because on Windows this kill may never be
# delivered, and an unbounded wait here is the whole of #179.
kill "$pid" 2>/dev/null || true
gone=no
for _ in $(seq 1 40); do
if ! kill -0 "$pid" 2>/dev/null; then gone=yes; break; fi
sleep 0.25
done
if [ "$gone" != yes ]; then
echo "NOTE: the server was still running 10s after the kill, so the"
echo "artefacts below and in the upload step were read from under a"
echo "live writer and may be truncated mid-write."
fi
echo "--- server log ---"
cat bcast-server.log || true
exit 1
fi
# What the two runs above ACTUALLY executed in cmd/polyemesis.
#
# WHY THIS EXISTS. `go test -cover ./cmd/polyemesis` reports ~38%, and
# from that number sixteen functions look untested -- main, run, watch,
# reconcile, disconnect, stop, startMQTT, runService, reportStartup,
# reportTLS, newTLSProvider, managerEngines among them. That number is an
# INSTRUMENT ARTEFACT, the same class as #217 in internal/api: `go test`
# can only see what the test binary runs, and package main's entry points
# are by construction driven by the PROCESS, which the two steps above
# start on all three platforms on every PR. Measured from a real
# instrumented run (darwin/arm64, local):
#
# main 66.7 run 75.7 watch 85.7 startMQTT 100 stop 100
# runService 100 reportStartup 67.5 reportTLS 31.2 disconnect 58.3
#
# Twelve of the sixteen are executed today. The gap was never in the code,
# it was in what we were able to see, and a coverage gap you cannot see is
# the one that gets filled with tests that assert nothing.
#
# THIS STEP NEVER FAILS THE JOB, and that is deliberate. It is an
# instrument, not a gate. A ratchet here would be a threshold nobody
# measured on three platforms yet, and the first thing it would do is fail
# a PR for a reason unrelated to the PR. It also cannot assume data
# exists: a cover-instrumented binary writes its profile at NORMAL EXIT,
# and the steps above stop the server with `kill`, which Git Bash may
# never deliver to a native .exe (the same fact the trap comments above
# are built around). If windows-latest reports nothing, that is a real
# answer about the platform and is printed as one rather than swallowed.
#
# The module-wide total is deliberately NOT printed. A bare server start
# touches little of internal/*, so that number would be low, meaningless,
# and immediately mistaken for the project's coverage.
- name: Report what the smoke run executed in cmd/polyemesis
if: always() && needs.changes.outputs.code == 'true'
# BOUNDED, because this step says three times over that it is
# informational -- always(), "does not fail the job", "informational
# only" -- and then took the whole job down twice in one afternoon.
#
# On windows-latest it hung inside `go tool covdata textfmt` / `go tool
# cover` and never returned: measured at 10:49:16 with no completion,
# against a normal 1-3 seconds on the same runner. The job hit its
# 35-minute ceiling and was CANCELLED, which `gh pr checks` renders as
# a failure naming nothing, and the required-check rule then refused
# the merge. #694 and #696 both lost a merge to it.
#
# An informational step that can spend the entire budget is not
# informational. Three minutes is sixty times the observed cost and
# still bounds the hang; the step's own `exit 0` paths mean a timeout
# here costs the report and nothing else.
timeout-minutes: 3
shell: bash
run: |
set -uo pipefail
dirs=""
for d in covdata/serve covdata/bcast; do
if [ -d "$d" ] && [ -n "$(ls -A "$d" 2>/dev/null)" ]; then
dirs="${dirs:+$dirs,}$d"
fi
done
if [ -z "$dirs" ]; then
echo "no coverage data was written on ${{ matrix.os }}."
echo "A -cover binary flushes its profile at normal exit; if the stop"
echo "signal was never delivered there is nothing to report. This is"
echo "informational and does not fail the job."
exit 0
fi
echo "merging coverage from: $dirs"
go tool covdata textfmt -i="$dirs" -o=smoke-cmd.cov || {
echo "covdata textfmt failed on ${{ matrix.os }}; informational only."
exit 0
}
echo "--- cmd/polyemesis, as executed by the smoke run on ${{ matrix.os }} ---"
go tool cover -func=smoke-cmd.cov | grep 'cmd/polyemesis' || \
echo "(no cmd/polyemesis rows in the profile)"
- name: Upload broadcast artefacts on failure
if: failure() && needs.changes.outputs.code == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: broadcast-${{ matrix.os }}
path: |
bcast-server.log
data/recordings/
retention-days: 7
if-no-files-found: ignore
# THE WINDOWS UNINSTALLER HAD ZERO COVERAGE OF ANY KIND, #509. Unlike
# the systemd and Docker uninstallers -- ten assertions between them in
# scripts/acceptance-install.sh -- deploy/windows/uninstall.ps1's
# live-broadcast check and typed confirmation, added in #489 to match
# them, had never been parsed, linted or executed on any platform. A
# syntax error in a script whose job is to end live broadcasts would
# have shipped undetected. These three steps run on the windows-latest
# leg already in this job's matrix, so the marginal cost is
# milliseconds of pure PowerShell, not a new runner allocation. Each
# extracts the real code under test from the file's own AST rather
# than retyping it, so a rewrite that keeps the same names and
# behaviour keeps passing, and one that silently changes them fails
# here instead of on an operator's host.
- name: Parse deploy/windows/uninstall.ps1 (#509)
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
run: |
$ErrorActionPreference = 'Stop'
$path = 'deploy/windows/uninstall.ps1'
$parseErrors = $null
[void][System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$null, [ref]$parseErrors)
if ($parseErrors) {
$parseErrors | ForEach-Object { Write-Host "PARSE ERROR: $_" }
exit 1
}
Write-Host "uninstall.ps1 parses cleanly"
# Get-PublishingFfmpeg is the function deciding whether the host is
# safe to touch. It is extracted from the file's AST and exercised
# against a mocked Get-CimInstance -- spinning up a real publishing
# ffmpeg on a CI runner just to prove a regex would be more fragile
# than the regex itself.
- name: "uninstall.ps1: Get-PublishingFfmpeg detection (#509)"
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
run: |
$ErrorActionPreference = 'Stop'
$path = 'deploy/windows/uninstall.ps1'
$ast = [System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$null, [ref]$null)
$funcAst = $ast.Find({
param($n)
$n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $n.Name -eq 'Get-PublishingFfmpeg'
}, $true)
if (-not $funcAst) {
Write-Host "FAIL: Get-PublishingFfmpeg not found in $path -- has it been renamed or removed?"
exit 1
}
Invoke-Expression $funcAst.Extent.Text
$failures = [System.Collections.Generic.List[string]]::new()
# Case 1: the process table cannot be read at all -- must fail
# closed (the deliberate "cannot prove it is idle" path).
function Get-CimInstance { param($ClassName, $Filter, $ErrorAction); throw 'WMI unavailable (simulated)' }
$result = Get-PublishingFfmpeg
if ($result.Checked) {
$failures.Add("expected Checked=false when Get-CimInstance throws (fail-closed), got Checked=$($result.Checked)")
} else {
Write-Host "PASS: an unreadable process table reports Checked=false (fail-closed)"
}
# Case 2: a publishing ffmpeg is present and must be found.
function Get-CimInstance {
param($ClassName, $Filter, $ErrorAction)
[pscustomobject]@{ ProcessId = 4242; CommandLine = 'ffmpeg.exe -i in.mp4 -f flv rtmp://example.invalid/live' }
}
$result = Get-PublishingFfmpeg
if (-not $result.Checked -or $result.Procs.Count -ne 1) {
$failures.Add("expected Checked=true and 1 matching process, got Checked=$($result.Checked) Count=$($result.Procs.Count)")
} else {
Write-Host "PASS: a publishing ffmpeg (rtmp:) is detected"
}
# Case 3: ffmpeg is running but NOT publishing (e.g. a local
# transcode) -- must not match.
function Get-CimInstance {
param($ClassName, $Filter, $ErrorAction)
[pscustomobject]@{ ProcessId = 4343; CommandLine = 'ffmpeg.exe -i in.mp4 out.mp4' }
}
$result = Get-PublishingFfmpeg
if (-not $result.Checked -or $result.Procs.Count -ne 0) {
$failures.Add("expected Checked=true and 0 matches for a non-publishing ffmpeg, got Checked=$($result.Checked) Count=$($result.Procs.Count)")
} else {
Write-Host "PASS: a non-publishing ffmpeg process is correctly ignored"
}
# THE GAP THIS USED TO WARN ABOUT IS CLOSED, so it is now an
# assertion instead of a Write-Warning. It used to say: PowerShell
# assigns $null, not an empty array, to a variable captured from a
# pipeline that produced zero objects, so "nothing is publishing"
# and "could not check" arrived identically and every uninstall on
# an idle host refused. uninstall.ps1 answers with a two-field
# object now (Checked / Procs) and cannot express that confusion.
#
# Kept as a case rather than deleted, because the shape it forbids
# is the one a rewrite reaches for first: a bare pipeline, or
# `return ,$found`, both of which look right and neither of which
# can carry two facts. Case 3 above -- ffmpeg running, publishing
# nothing -- is the input that used to produce $null.
$rawResult = Get-PublishingFfmpeg
if ($null -eq $rawResult) {
$failures.Add('Get-PublishingFfmpeg returned $null for "no match", which is indistinguishable from "could not check". Return an object with Checked and Procs; a bare pipeline or `return ,$found` cannot carry both facts.')
} else {
Write-Host "PASS: an idle host gets an object, never `$null"
}
if ($failures.Count -gt 0) {
Write-Host ''
Write-Host 'FAILURES:'
$failures | ForEach-Object { Write-Host " - $_" }
exit 1
}
Write-Host ''
Write-Host 'Get-PublishingFfmpeg: fail-closed and filtering both verified'
# The -Force / live-check / typed-confirmation gate -- the single
# if-statement whose body calls both Get-PublishingFfmpeg and
# Read-Host -- is extracted the same way and run four times with both
# of those mocked out. This is the logic #489 added to match the
# systemd and Docker uninstallers; #509 is that nothing had ever run
# it.
- name: "uninstall.ps1: Force/live-check/confirmation gate (#509)"
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
run: |
$ErrorActionPreference = 'Stop'
$path = 'deploy/windows/uninstall.ps1'
$ast = [System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$null, [ref]$null)
$ifAst = $ast.Find({
param($n)
$n -is [System.Management.Automation.Language.IfStatementAst] -and
$n.Extent.Text -match 'Get-PublishingFfmpeg' -and
$n.Extent.Text -match 'Read-Host'
}, $true)
if (-not $ifAst) {
Write-Host "FAIL: could not locate the Force/live-check/confirmation if-block in $path -- its shape has changed"
exit 1
}
$ifBlockText = $ifAst.Extent.Text
$failures = [System.Collections.Generic.List[string]]::new()
function Invoke-Scenario($Text) {
# Invoke-Expression runs in the scope it is called from, which
# here is this helper's own scope -- a child of the script scope
# below, so it inherits read access to the $Force / $ServiceName
# / etc. variables and the Get-PublishingFfmpeg / Read-Host
# overrides defined there before each call.
Invoke-Expression $Text
}
# --- Scenario 1: -Force bypasses BOTH the live check and the confirmation ---
$Force = $true; $IgnoreLiveBroadcast = $false; $RemoveData = $false; $InstallDir = 'C:\Fake\Install'; $DataDir = 'C:\Fake\Data'; $ServiceName = 'polyemesis'
$script:calls = 0
function Get-PublishingFfmpeg { $script:calls++; [pscustomobject]@{ Checked = $true; Procs = @() } }
function Read-Host { param($Prompt); $script:calls++; 'polyemesis' }
try {
Invoke-Scenario $ifBlockText
if ($script:calls -ne 0) {
$failures.Add("Force: expected neither check to run, but they were called $script:calls time(s)")
} else {
Write-Host "PASS: -Force skips both the live check and the confirmation prompt"
}
} catch {
$failures.Add("Force: expected no throw, got: $($_.Exception.Message)")
}
# --- Scenario 2: a stubbed process table shows a publishing ffmpeg -> throws ---
$Force = $false; $IgnoreLiveBroadcast = $false; $RemoveData = $false; $InstallDir = 'C:\Fake\Install'; $DataDir = 'C:\Fake\Data'; $ServiceName = 'polyemesis'
function Get-PublishingFfmpeg { [pscustomobject]@{ Checked = $true; Procs = @([pscustomobject]@{ ProcessId = 4242; CommandLine = 'ffmpeg.exe -i in -f flv rtmp://example.invalid/live' }) } }
function Read-Host { param($Prompt); throw 'Read-Host must not be reached while a broadcast is live' }
try {
Invoke-Scenario $ifBlockText
$failures.Add('Live ffmpeg: expected a throw, none happened')
} catch {
if ($_.Exception.Message -match 'REFUSING') {
Write-Host "PASS: a publishing ffmpeg process makes the script refuse"
} else {
$failures.Add("Live ffmpeg: wrong error: $($_.Exception.Message)")
}
}
# --- Scenario 3: the process table cannot be read -> throws (fail closed) ---
$Force = $false; $IgnoreLiveBroadcast = $false; $RemoveData = $false; $InstallDir = 'C:\Fake\Install'; $DataDir = 'C:\Fake\Data'; $ServiceName = 'polyemesis'
function Get-PublishingFfmpeg { [pscustomobject]@{ Checked = $false; Procs = @() } }
function Read-Host { param($Prompt); throw 'Read-Host must not be reached when the process table could not be read' }
try {
Invoke-Scenario $ifBlockText
$failures.Add('Unreadable process table: expected a throw, none happened')
} catch {
if ($_.Exception.Message -match 'Cannot determine') {
Write-Host "PASS: an unreadable process table fails closed (throws) rather than proceeding"
} else {
$failures.Add("Unreadable process table: wrong error: $($_.Exception.Message)")
}
}
# --- Scenario 4: a wrong confirmation answer is rejected ---
$Force = $false; $IgnoreLiveBroadcast = $false; $RemoveData = $false; $InstallDir = 'C:\Fake\Install'; $DataDir = 'C:\Fake\Data'; $ServiceName = 'polyemesis'
function Get-PublishingFfmpeg { [pscustomobject]@{ Checked = $true; Procs = @() } }
function Read-Host { param($Prompt); 'definitely-the-wrong-answer' }
try {
Invoke-Scenario $ifBlockText
$failures.Add('Wrong confirmation: expected a throw, none happened')
} catch {
if ($_.Exception.Message -match 'Not confirmed') {
Write-Host "PASS: a wrong confirmation answer is rejected"
} else {
$failures.Add("Wrong confirmation: wrong error: $($_.Exception.Message)")
}
}
# --- Scenario 5: -IgnoreLiveBroadcast skips the ON-AIR check ONLY ---
# Two bypasses that used to be one keystroke. -Force skips the on-air
# check AND the confirmation, so an operator who only meant "yes, I
# know it is live" also turned off the typed confirmation -- and the
# habit is the damage, because -RemoveData -Force then deletes an
# operator-supplied path with no prompt at all. This switch is the
# narrow one: the broadcast check is skipped, the confirmation is not.
$Force = $false; $IgnoreLiveBroadcast = $true; $RemoveData = $false; $InstallDir = 'C:\Fake\Install'; $DataDir = 'C:\Fake\Data'; $ServiceName = 'polyemesis'
$script:asked = 0
function Get-PublishingFfmpeg { throw 'the on-air check must not run under -IgnoreLiveBroadcast' }
function Read-Host { param($Prompt); $script:asked++; 'polyemesis' }
try {
Invoke-Scenario $ifBlockText
if ($script:asked -ne 1) {
$failures.Add("IgnoreLiveBroadcast: the confirmation must still be asked exactly once, was asked $script:asked time(s)")
} else {
Write-Host "PASS: -IgnoreLiveBroadcast skips the on-air check and still asks for confirmation"
}
} catch {
$failures.Add("IgnoreLiveBroadcast: expected no throw, got: $($_.Exception.Message)")
}
if ($failures.Count -gt 0) {
Write-Host ''
Write-Host 'FAILURES:'
$failures | ForEach-Object { Write-Host " - $_" }
exit 1
}
Write-Host ''
Write-Host 'uninstall.ps1 Force/live-check/confirmation logic: all scenarios passed'
# -RemoveData IS A RECURSIVE FORCE DELETE OF AN OPERATOR-SUPPLIED PATH,
# and until #552 it had no guard of any kind: `-RemoveData -DataDir
# C:\ProgramData` -- a typo, a tab-completed parent, or simply the
# non-default path the install was given -- printed a warning and then
# deleted it. The Linux uninstaller has had all of these for releases.
#
# Extracted from the AST like everything else in this block, so a rename
# fails here rather than on an operator's host. The last case is the one
# that matters: the first three prove the path is safe to TYPE, only the
# content check proves it is OURS, and DataDir is baked in at install time
# and never re-read.
- name: "uninstall.ps1: -RemoveData path guard (#552)"
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
run: |
$ErrorActionPreference = 'Stop'
$path = 'deploy/windows/uninstall.ps1'
$ast = [System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$null, [ref]$null)
$funcAst = $ast.Find({
param($n)
$n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $n.Name -eq 'Assert-RemovableDataDir'
}, $true)
if (-not $funcAst) {
Write-Host "FAIL: Assert-RemovableDataDir not found in $path -- -RemoveData would be an unguarded recursive delete again"
exit 1
}
Invoke-Expression $funcAst.Extent.Text
$failures = [System.Collections.Generic.List[string]]::new()
$work = Join-Path $env:RUNNER_TEMP ("rmdata-" + [guid]::NewGuid())
$ours = Join-Path $work 'polyemesis'
New-Item -ItemType Directory -Force -Path $ours | Out-Null
Set-Content -LiteralPath (Join-Path $ours 'polyemesis.db') -Value 'x'
# Every one of these must be refused. $work itself is the "not ours"
# case: a real directory, safely nested, holding neither marker.
$mustRefuse = @{
'an empty path' = ''
'a drive root' = 'C:\'
'a top-level directory' = 'C:\ProgramData'
'a directory that is not ours' = $work
}
foreach ($case in $mustRefuse.GetEnumerator()) {
try {
Assert-RemovableDataDir $case.Value
$failures.Add("$($case.Key) ('$($case.Value)') was ACCEPTED for a recursive force delete")
} catch {
Write-Host "PASS: refused $($case.Key)"
}
}
# And a guard that refuses everything is not a guard.
try {
Assert-RemovableDataDir $ours
Write-Host "PASS: a real data directory is still accepted"
} catch {
$failures.Add("a legitimate data directory was refused: $($_.Exception.Message)")
}
# Either marker is enough -- a key file with no database is still ours.
Remove-Item -LiteralPath (Join-Path $ours 'polyemesis.db')
Set-Content -LiteralPath (Join-Path $ours 'secret.key') -Value 'x'
try {
Assert-RemovableDataDir $ours
Write-Host "PASS: secret.key alone identifies it as ours"
} catch {
$failures.Add("a data directory holding only secret.key was refused: $($_.Exception.Message)")
}
Remove-Item -Recurse -Force $work
# And the guard has to be CALLED. A function nothing invokes passes
# every case above.
$src = Get-Content -Raw -LiteralPath $path
if ($src -notmatch 'Assert-RemovableDataDir \$DataDir') {
$failures.Add('Assert-RemovableDataDir is defined but never called before Remove-Item on $DataDir')
}
if ($failures.Count -gt 0) {
Write-Host ''
Write-Host 'FAILURES:'
$failures | ForEach-Object { Write-Host " - $_" }
exit 1
}
Write-Host ''
Write-Host 'uninstall.ps1 -RemoveData guard: all cases passed'
# install.ps1 HAD NEVER BEEN PARSED BY ANYTHING. uninstall.ps1 got the
# three steps above in #509; the script that registers a service running
# as LocalSystem got none, and a syntax error in it would have shipped.
- name: Parse deploy/windows/install.ps1
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
run: |
$ErrorActionPreference = 'Stop'
$path = 'deploy/windows/install.ps1'
$parseErrors = $null
[void][System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$null, [ref]$parseErrors)
if ($parseErrors) {
$parseErrors | ForEach-Object { Write-Host "PARSE ERROR: $_" }
exit 1
}
Write-Host "install.ps1 parses cleanly"
# THE BINARY IS ABOUT TO BE REGISTERED AS A SERVICE RUNNING AS LocalSystem
# (#583). The Windows path took an already-downloaded .exe and never
# hashed it, while SHA256SUMS was published as a release asset that
# nothing here consulted -- and it had neither half of install.sh's
# policy: refuse on mismatch, refuse on absence unless told otherwise.
- name: "install.ps1: checksum policy (#583)"
if: needs.changes.outputs.code == 'true' && matrix.os == 'windows-latest'
run: |
$ErrorActionPreference = 'Stop'
$path = 'deploy/windows/install.ps1'
$ast = [System.Management.Automation.Language.Parser]::ParseFile($path, [ref]$null, [ref]$null)
$funcAst = $ast.Find({
param($n)
$n -is [System.Management.Automation.Language.FunctionDefinitionAst] -and $n.Name -eq 'Resolve-ExpectedHash'
}, $true)
if (-not $funcAst) {
Write-Host "FAIL: Resolve-ExpectedHash not found in $path -- the install path would accept an unverified binary again"
exit 1
}
Invoke-Expression $funcAst.Extent.Text
$failures = [System.Collections.Generic.List[string]]::new()
$work = Join-Path $env:RUNNER_TEMP ("sha-" + [guid]::NewGuid())
New-Item -ItemType Directory -Force -Path $work | Out-Null
$exe = Join-Path $work 'polyemesis.exe'
Set-Content -LiteralPath $exe -Value 'not a real binary' -NoNewline
$real = (Get-FileHash -LiteralPath $exe -Algorithm SHA256).Hash.ToUpperInvariant()
$sums = Join-Path $work 'SHA256SUMS'
Set-Content -LiteralPath $sums -Value "$real polyemesis.exe"
# THE DEFAULT REFUSES. This is the case an operator hits by following
# the README without reading it, and it is the one that used to
# install an unhashed binary.
try {
Resolve-ExpectedHash -Exe $exe -Expected '' -SumsPath '' -Unverified $false
$failures.Add('no checksum and no -AllowUnverified was ACCEPTED')
} catch {
Write-Host "PASS: with no checksum, installing is refused by default"
}
# -AllowUnverified is the only way past, and it is a flag you type.
try {
$r = Resolve-ExpectedHash -Exe $exe -Expected '' -SumsPath '' -Unverified $true
if ($null -ne $r) { $failures.Add("-AllowUnverified should yield no expected hash, got '$r'") }
else { Write-Host "PASS: -AllowUnverified installs without a hash, and warns" }
} catch {
$failures.Add("-AllowUnverified was refused: $($_.Exception.Message)")
}
# A SUMS file is looked up by file name.
try {
$r = Resolve-ExpectedHash -Exe $exe -Expected '' -SumsPath $sums -Unverified $false
if ($r -ne $real) { $failures.Add("SHA256SUMS lookup returned '$r', expected '$real'") }
else { Write-Host "PASS: the SHA256SUMS entry for this file is found" }
} catch {
$failures.Add("SHA256SUMS lookup threw: $($_.Exception.Message)")
}
# A SUMS file WITHOUT an entry for this file refuses even under
# -AllowUnverified: the operator supplied a checksum source and it did
# not vouch for this binary, which is a different thing from having
# none.
try {
Resolve-ExpectedHash -Exe (Join-Path $work 'absent.exe') -Expected '' -SumsPath $sums -Unverified $true
$failures.Add('a SHA256SUMS with no entry for the binary was ACCEPTED')
} catch {
Write-Host "PASS: a SHA256SUMS with no entry for this binary is refused"
}
# And the check has to run before the copy.
$src = Get-Content -Raw -LiteralPath $path
if ($src -notmatch 'Resolve-ExpectedHash') {
$failures.Add('Resolve-ExpectedHash is defined but never called')
}
$hashAt = $src.IndexOf('Get-FileHash')
$copyAt = $src.IndexOf('Copy-Item')
if ($hashAt -lt 0 -or $copyAt -lt 0 -or $hashAt -gt $copyAt) {
$failures.Add('the hash is not computed before the binary is copied into Program Files')
}
Remove-Item -Recurse -Force $work
if ($failures.Count -gt 0) {
Write-Host ''
Write-Host 'FAILURES:'
$failures | ForEach-Object { Write-Host " - $_" }
exit 1
}
Write-Host ''
Write-Host 'install.ps1 checksum policy: all cases passed'
ui:
name: ui typecheck, lint, build
permissions:
contents: read
# The documentation gate, #378. On every STEP below, never on this job --
# see the `changes` job for why that distinction is load-bearing and for
# what it cost the one time it was got wrong.
needs: changes
runs-on: ubuntu-latest
timeout-minutes: 15
defaults:
run:
working-directory: ui
steps:
# Says out loud why a green check did no work, so a reader of the
# checks list is never left guessing whether this ran or no-opped.
- name: Documentation-only change, so this job did no work
if: needs.changes.outputs.code != 'true'
working-directory: ${{ github.workspace }}
run: |
echo "::notice title=ui typecheck, lint, build::this check did no work -- every changed path was documentation. See the 'which changes' job."
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: needs.changes.outputs.code == 'true'
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
if: needs.changes.outputs.code == 'true'
with:
# Keep in step with the Dockerfiles' node tag and @types/node major.
node-version: 24
cache: npm
cache-dependency-path: ui/package-lock.json
- if: needs.changes.outputs.code == 'true'
run: npm ci --ignore-scripts
- if: needs.changes.outputs.code == 'true'
run: npx --no-install tsc -b --noEmit
- if: needs.changes.outputs.code == 'true'
run: npm run lint
- if: needs.changes.outputs.code == 'true'
run: npm run build
# Unit tests for the pure logic the browser suite cannot enumerate --
# platform link construction has five platforms times several missing-field
# cases, and driving each through a real browser would cost minutes to
# assert what a millisecond of vitest does.
- if: needs.changes.outputs.code == 'true'
run: npm test
# A vulnerable direct dependency should fail the build, not sit in a
# report nobody opens. --audit-level=high so a low-severity transitive
# advisory does not block a hotfix.
# THE ADVISORY SOURCE THAT ANSWERS -- see the long note on the matching
# job in security.yml. Briefly: `npm audit` POSTs the tree to npm's bulk
# advisories endpoint, which returned 503 for hours on 2026-09-04 while
# npm's status page called it operational. osv-scanner reads OSV.dev and
# answers in a second from the lockfile alone, with no install and no
# registry round trip.
#
# The positive control lives in security.yml, which owns dependency
# auditing for BOTH projects. This step exists because ui/ is built here
# and a build that ships known-vulnerable dependencies should fail where
# the build is, not only in another workflow.
- if: needs.changes.outputs.code == 'true'
name: npm audit (ui)
working-directory: ${{ github.workspace }}
run: |
set -uo pipefail
curl -sSfL --proto '=https' --tlsv1.2 --retry 3 --retry-all-errors \
--retry-delay 5 --max-time 120 -o /tmp/osv-scanner \
"https://github.com/google/osv-scanner/releases/download/v2.5.1/osv-scanner_linux_amd64"
echo "f9f25499a2c8cc367b3af45df2ea7eeca7fbccceab9c35079968f4b3652194be /tmp/osv-scanner" | sha256sum -c -
chmod +x /tmp/osv-scanner
out=$(/tmp/osv-scanner scan source --lockfile ui/package-lock.json 2>&1); rc=$?
printf '%s\n' "$out"
# osv-scanner exits 0 on a path it cannot resolve, so a typo would
# otherwise pass as a clean audit. The count is read back.
if ! printf '%s' "$out" | grep -qE 'Scanned .*package-lock\.json file and found [1-9][0-9]* packages'; then
echo "::error title=Nothing was scanned::osv-scanner did not report reading ui/package-lock.json with packages in it."
exit 1
fi
if [ "$rc" -ne 0 ]; then
echo "::error title=Vulnerable dependencies::osv-scanner found known vulnerabilities in ui. The advisories and their fixed versions are listed above."
exit 1
fi
# THE WORKFLOWS THEMSELVES WERE NEVER CHECKED.
#
# An invalid workflow file does not fail a job -- it prevents the job from
# existing. Run 33791030925 produced ZERO jobs and the entire report was
# "This run likely failed because of a workflow file issue", with no line, no
# file and no message. The cause was one pair of expression delimiters written
# inside a `run:` block, in a shell COMMENT, by a comment that was explaining
# why not to use them: Actions expands that syntax anywhere in a run block, an
# empty pair is a parse error, and GitHub then refuses the file whole.
#
# Nothing in this repository ran actionlint, so the first and only signal was a
# push that silently produced nothing. That is the same defect this branch is
# fixing one workflow at a time -- a red verdict that will not say what it
# caught -- and it belongs at the Control rung: a workflow that cannot parse
# should not be mergeable.
#
# Deliberately NOT gated on the `changes` job. Workflow files are exactly what
# this checks, a documentation-only pull request can still edit one, and the
# job costs about fifteen seconds. Gating it would reintroduce the hole for the
# one change most likely to fall into it.
#
# A pinned release binary rather than a third-party action: the version and its
# checksum are visible in the diff, and it adds no action to review for
# supply-chain risk. This comment used to describe a `go run ...@version` step
# and outlived it by one commit -- Sonar's S8545 failed that step for resolving
# the tool's own dependencies fresh every run, the step became the curl below,
# and the comment stayed. Left recorded rather than quietly rewritten, because
# a stale comment is the failure this repository audits for.
actionlint:
name: workflow lint
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# PINNED BY CHECKSUM, and not `go install ...@version`. Sonar's S8545
# failed the first version of this job for exactly that: `go install`
# resolves the tool's own dependency graph fresh on every run, with no lock
# file, so the bytes that lint this repository's workflows are not the same
# bytes twice. It is the same argument the gitleaks step in security.yml
# already makes, so this follows that step's shape -- pinned URL, landed to
# a file before extracting, and a version assertion afterwards -- and adds
# the checksum, because this one is cheap to have and the finding was fair.
#
# 1.7.12 rather than 1.7.7 for a reason worth recording: actionlint checks
# runner labels against a list baked into its own release, and 1.7.7 does
# not know macos-26-intel, so it failed srt-probe.yml over a runner that
# demonstrably works -- run 32892220990 has a green job named
# "macos-26-intel (macOS 26 x64)". The first fix was a .github/actionlint.yaml
# declaring the label under `self-hosted-runner`, which is the only hook
# actionlint offers and is a misnomer for a GitHub-hosted runner. Pinning
# the version that already knows the label is one moving part instead of
# two, and it matches what `brew install actionlint` puts on a developer's
# PATH -- which is how the stale list went unnoticed locally in the first
# place: 1.7.12 here, 1.7.7 in CI, and only CI complained.
#
# --proto '=https' because -L follows redirects: without it a redirect to
# http:// silently downgrades the transport for a binary about to run.
- name: install actionlint (pinned)
env:
ACTIONLINT_VERSION: 1.7.12
ACTIONLINT_SHA256: 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
run: |
set -euo pipefail
curl -sSfL --proto '=https' --tlsv1.2 \
--retry 3 --retry-all-errors --retry-delay 5 --max-time 120 \
-o actionlint.tar.gz \
"https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz"
echo "${ACTIONLINT_SHA256} actionlint.tar.gz" | sha256sum -c -
sudo tar -xz -C /usr/local/bin -f actionlint.tar.gz actionlint
rm -f actionlint.tar.gz
installed="$(actionlint -version | head -1)"
echo "actionlint $installed"
if [ "$installed" != "$ACTIONLINT_VERSION" ]; then
echo "::error::expected actionlint $ACTIONLINT_VERSION, got $installed"
exit 1
fi
# -shellcheck= and -pyflakes= disable those two integrations, and the
# reason belongs here rather than hidden in a flag. With shellcheck on,
# this job reports 38 findings on its first run, nearly all of them
# shellcheck failing to parse a run block that contains an Actions
# expression -- it reads the expansion as broken shell. A guard that fires
# 38 times on the day it is installed is one somebody turns off, and it
# would bury the class of finding this job exists for.
#
# What stays on is the part that matters: workflow schema, job and step
# keys, `needs` references, matrix shapes, action input names, and the
# expression parser -- which is exactly what rejected sonar.yml and left a
# run with no jobs and no message.
- run: actionlint -color -shellcheck= -pyflakes=
cross:
name: cross-compile all release targets
permissions:
contents: read
# The documentation gate, #378. On every STEP below, never on this job --
# see the `changes` job for why that distinction is load-bearing and for
# what it cost the one time it was got wrong.
needs: changes
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
# Says out loud why a green check did no work, so a reader of the
# checks list is never left guessing whether this ran or no-opped.
- name: Documentation-only change, so this job did no work
if: needs.changes.outputs.code != 'true'
run: |
echo "::notice title=cross-compile all release targets::this check did no work -- every changed path was documentation. See the 'which changes' job."
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: needs.changes.outputs.code == 'true'
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
if: needs.changes.outputs.code == 'true'
with:
go-version-file: go.mod
cache: true
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
if: needs.changes.outputs.code == 'true'
with:
node-version: 24
cache: npm
cache-dependency-path: ui/package-lock.json
# `make release` depends on the ui target, which needs the built assets
# for go:embed. Building them here keeps the check honest end to end.
- if: needs.changes.outputs.code == 'true'
run: make release
- if: needs.changes.outputs.code == 'true'
run: ls -lh dist/
# Which container suites this run needs.
#
# A separate job rather than an `if:` on the matrix below, for two reasons.
# The matrix context is not available to a job-level `if`, so "run only the
# browser suite" cannot be expressed there at all. And a suite that is not
# wanted should not EXIST as a skipped job: main's runs already carry enough
# grey "skipping" rows that a genuinely skipped check stops being noticed.
#
# Cheap: a checkout and a git diff, against three jobs that each build an
# image and publish real streams.
container-suites:
name: "which container suites"
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
suites: ${{ steps.pick.outputs.suites }}
skipped: ${{ steps.pick.outputs.skipped }}
steps:
# Full history: the diff below needs both endpoints of the PR, and the
# default shallow fetch has neither.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- id: pick
# Through env, never interpolated into the script. Two of these are
# attacker-influenced on a fork PR, and `run:` is a shell.
env:
EVENT: ${{ github.event_name }}
REF: ${{ github.ref }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
set -euo pipefail
ALL='["acceptance-docker","acceptance-multisource","acceptance-browser"]'
# ONE WRITER FOR BOTH OUTPUTS, so `suites` and `skipped` cannot drift
# apart. A suite that appears in neither list produces no check of
# that name at all -- and `container: acceptance-docker` and
# `container: acceptance-multisource` are REQUIRED contexts on main.
# A required check that never reports leaves the pull request blocked
# forever with nothing red to point at, which is how #650, #656 and
# #659 came to sit unmergeable behind a wall of green checks. The
# skipped list is what the shim job reports, so the context is always
# answered: either the suite ran, or it says why it did not.
emit() {
SEL="$1"; WHY="$2"
SKIP="$(jq -cn --argjson all "$ALL" --argjson sel "$SEL" '$all - $sel')"
# Fixed-value check: the two lists must reconstitute the full set.
# Getting this wrong reintroduces the exact bug above, silently.
N_ALL="$(jq -n --argjson a "$ALL" '$a|length')"
N_SEL="$(jq -n --argjson a "$SEL" '$a|length')"
N_SKIP="$(jq -n --argjson a "$SKIP" '$a|length')"
if [ "$((N_SEL + N_SKIP))" -ne "$N_ALL" ]; then
echo "::error::suite partition lost a suite: selected=$SEL skipped=$SKIP of $ALL"
exit 1
fi
{
echo "suites=$SEL"
echo "skipped=$SKIP"
} >> "$GITHUB_OUTPUT"
echo "$WHY"
exit 0
}
# main and the weekly schedule keep the full set. This job changes
# what PRs get, and takes nothing away from anywhere else.
if [ "$EVENT" = "schedule" ] || [ "$REF" = "refs/heads/main" ]; then
emit "$ALL" "full set: main or schedule"
fi
# A PR that touches ui/ gets the browser suite and only the browser
# suite. acceptance-docker and acceptance-multisource assert routing
# and token handling, which no UI change reaches -- running them here
# would cost two image builds to prove something the PR cannot break.
#
# Captured to a variable and matched with a here-string rather than
# piped into grep. Under `set -o pipefail`, `git diff | grep -q` can
# report the PIPELINE as failed when grep matches early and exits,
# leaving git to die of SIGPIPE. It did not reproduce at 196 files, so
# this is a latent hazard rather than an observed bug -- but its
# failure mode is a ui/ PR silently NOT getting the browser suite,
# which is the exact thing this job exists to stop happening.
# THIS FILE COUNTS AS WELL, and it was added after a change to the
# browser suite's OWN setup merged without that suite ever running.
#
# The Playwright install lives in this workflow, not under ui/, so a
# PR that rewrites how the browser is installed matched nothing here
# and the container job reported "skipping". A broken install step
# would have gone green on its own PR and failed on the next ui/
# change instead, pointing at whoever touched the UI.
#
# The whole file rather than a line range: a job's behaviour is not
# confined to the block that names it -- concurrency, permissions,
# the suite matrix and the runner image all sit elsewhere in here and
# all reach it. Matching the file is the honest boundary, and ci.yml
# changes are rare enough that one image build is the right price for
# never shipping an unexercised one again.
if [ "$EVENT" = "pull_request" ]; then
CHANGED="$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")"
# AND A SUITE'S OWN SCRIPT COUNTS, which is the same lesson as the
# ci.yml clause above, learned a second time one directory over. A
# PR that rewrote the EXIT traps in all three of these scripts --
# and in the two libraries every one of them sources -- matched
# nothing here and got "skipping". A broken trap would have gone
# green on its own PR and surfaced on the next ui/ change instead,
# pointing at whoever touched the UI and not at the change that
# actually did it.
#
# The full set rather than the one script that changed: lib-*.sh is
# sourced by all three, and working out which suite a shared
# library can break is exactly the reasoning that produced the gap.
if grep -qE '^scripts/(acceptance-(docker|multisource|browser)\.sh|lib-[a-z-]+\.sh)$' <<<"$CHANGED"; then
emit "$ALL" "full set: this PR changes a container suite's own script or a library it sources"
fi
if grep -qE '^(ui/|\.github/workflows/ci\.yml$)' <<<"$CHANGED"; then
emit '["acceptance-browser"]' "browser: this PR touches ui/ or this workflow"
fi
fi
emit '[]' "none: this PR touches neither ui/ nor this workflow nor a container suite's script"
# The suites that need the shipped IMAGE rather than a host binary: SRT ingest
# (Ubuntu's FFmpeg may lack libsrt, the image never does), the one-port
# token-routing path, and the browser running against the real artefact.
#
# Expensive — each builds the image and publishes real streams into it — so
# the full set runs on main and on a schedule rather than on every push.
#
# The browser suite is the exception, and #53 is why. Five guards were added
# there for three defects that had passed tsc, oxlint, go vet, go test -race
# and six reviews -- a nav that lost its CSS classes, a collapsed rail with no
# accessible names, and a toggle that destroyed keyboard focus. Every one of
# those guards contributed NOTHING to that PR's status, because this job only
# ran after the merge. Tests that cannot fail before you merge are not a gate.
# THE SHIM THAT ANSWERS A REQUIRED CHECK THE SUITE DID NOT RUN.
#
# `container: acceptance-docker` and `container: acceptance-multisource` are
# required contexts on main, and the job above deliberately does not run them
# on most pull requests -- they assert routing and token handling that a UI
# change cannot reach, and each costs an image build.
#
# Those two facts contradicted each other for as long as they both existed. A
# skipped job reports NOTHING, not success, so branch protection waited on a
# check that was never going to arrive: #659 sat at 40 of 40 green and
# unmergeable, #656 and #650 the same, and nothing anywhere was red. The
# failure had no symptom to search for, which is what made it expensive.
#
# So the contexts are always answered. Either the suite ran and reported its
# real result, or this job reports that it was not applicable and says why.
# The two matrices are disjoint by construction -- `emit` above computes
# skipped as ALL minus selected and refuses to write a partition that lost a
# suite -- so exactly one of them reports each name.
#
# WHAT THIS DOES NOT DO, deliberately: it does not make the suite's absence
# invisible. The reason is printed and appears in the check's own log, so
# "this passed" and "this did not run" remain distinguishable to a reader who
# looks. What it removes is only the state where nobody can merge and nothing
# can be pointed at.
container-not-applicable:
name: "container: ${{ matrix.suite }}"
permissions:
contents: read
needs: container-suites
if: needs.container-suites.outputs.skipped != '[]'
runs-on: ubuntu-latest
timeout-minutes: 5
strategy:
fail-fast: false
matrix:
suite: ${{ fromJSON(needs.container-suites.outputs.skipped) }}
steps:
- name: Not applicable to this pull request
env:
SUITE: ${{ matrix.suite }}
run: |
echo "::notice title=${SUITE} did not run::This pull request changes nothing the suite covers, so it was not built. It runs in full on main and on the weekly schedule. See the container-suites job for the rule that decided this."
echo "${SUITE}: not applicable to this pull request; runs on main and on schedule."
container:
name: "container: ${{ matrix.suite }}"
permissions:
contents: read
needs: container-suites
if: needs.container-suites.outputs.suites != '[]'
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
# acceptance-docker 29 checks: routing, passthrough, persistence
# acceptance-multisource 18 checks: two programmes, one port, by token
# acceptance-browser 30 checks: Playwright against the image
suite: ${{ fromJSON(needs.container-suites.outputs.suites) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: go.mod
cache: true
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: npm
cache-dependency-path: ui/package-lock.json
# THE BROWSER IS CACHED; THE SYSTEM LIBRARIES ARE NOT, and the split is
# the whole point of this block.
#
# `playwright install --with-deps chromium` did two unrelated things in
# one line: downloaded a browser build from Google's Chrome-for-Testing
# CDN, and apt-installed the shared libraries it links against. Only the
# first is cacheable -- the second writes to /usr, which no cache
# restores -- so they are now separate steps.
#
# THE CDN IS WHAT ACTUALLY FAILED. On 2026-08-12 it answered 403:
#
# Downloading Chrome for Testing 151.0.7922.34
# Error: Download failed: server returned code 403
#
# A cache hit removes that host from the path entirely, which is strictly
# better than retrying it: the retries elsewhere in this file exist for
# dependencies that must be fetched every time, and this one need not be.
#
# KEYED ON THE RESOLVED PLAYWRIGHT VERSION, not on a lockfile hash. The
# browser build is a function of that version alone, so an unrelated
# dependency bump must not evict a browser that is still correct. Read
# from playwright-core, which is the package that actually pins the
# revision; @playwright/test can float above it.
- name: Playwright version
if: matrix.suite == 'acceptance-browser'
id: playwright-version
working-directory: ui
run: |
v=$(node -p "require('./package-lock.json').packages['node_modules/playwright-core'].version")
[ -n "$v" ] || { echo "could not resolve playwright-core version"; exit 1; }
echo "version=$v" >> "$GITHUB_OUTPUT"
echo "playwright-core $v"
# No restore-keys. A near-miss would restore a browser for a DIFFERENT
# Playwright, which the install below would then have to download over
# anyway -- a partial hit here buys nothing and makes the cache's
# contents harder to reason about.
- uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
if: matrix.suite == 'acceptance-browser'
with:
path: ~/.cache/ms-playwright
key: ms-playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
# THE MIRROR, BEFORE ANYTHING ASKS IT FOR A PACKAGE.
#
# install-deps is apt, and the runner image's mirrorlist puts
# azure.archive.ubuntu.com first. When that host stalls rather than
# refuses, apt sits through its full timeout per index, per suite, and the
# log fills with `Ign:` lines that look like progress. Measured on this
# PR: 22 minutes into a step with no ceiling, still retrying.
#
# A RETRY RE-ASKS THE HOST THAT IS REFUSING. A different mirror does not,
# which is why this drops azure rather than only raising the retry count --
# the same distinction issue #430 draws for the FFmpeg install. The lines
# are removed rather than the file replaced, so whatever else the image
# ships stays; if that empties it, archive.ubuntu.com is written back.
#
# The options go in a drop-in because playwright SHELLS OUT to apt-get:
# there is no argv to pass -o through.
- name: Prefer a reachable apt mirror
if: matrix.suite == 'acceptance-browser'
timeout-minutes: 3
run: |
sudo tee /etc/apt/apt.conf.d/99polyemesis >/dev/null <<'CONF'
Acquire::Retries "3";
Acquire::http::Timeout "20";
Acquire::https::Timeout "20";
CONF
m=/etc/apt/apt-mirrors.txt
if [ -f "$m" ]; then
sudo sed -i '/azure\.archive\.ubuntu\.com/d' "$m"
[ -s "$m" ] || echo 'https://archive.ubuntu.com/ubuntu/' | sudo tee "$m" >/dev/null
echo "apt mirrors now:"; cat "$m"
fi
# A CEILING, for the same reason the suite step below carries one: without
# it this step hangs to the job's 45 minutes and names nothing. It is the
# step that has actually hung, and it was the one without a limit.
- name: Install Playwright
if: matrix.suite == 'acceptance-browser'
working-directory: ui
timeout-minutes: 12
run: |
npm ci --ignore-scripts
# install-deps runs UNCONDITIONALLY: it is apt writing to /usr, which
# the cache above does not cover, so a cache hit still needs it.
npx --no-install playwright install-deps chromium
# install runs unconditionally TOO, and is not gated on cache-hit on
# purpose. It checks the local browser registry first and skips the
# download when the revision is already there, so a cache hit costs
# no CDN request -- while a cache that restored incompletely still
# self-heals instead of failing later as "chromium not found".
npx --no-install playwright install chromium
# The same step timeout the `acceptance` job's suite step carries at
# :724, and missing here until now. This job's ceiling is 45 minutes,
# so a suite that hangs costs 45 minutes and names no step -- the exact
# cost #179 priced at 28. 35 leaves the image build and the log upload
# their share of the job budget while still firing first.
- run: ./scripts/${{ matrix.suite }}.sh
timeout-minutes: 35
- name: Upload logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ${{ matrix.suite }}-logs
path: |
/tmp/poly-*/**/*.log
ui/test-results/**
if-no-files-found: ignore
retention-days: 7
# The measurement suites. Everything above proves the code COMPILES and that
# its pure functions agree with themselves; these prove the product actually
# routes audio, copies video untouched and rides a source switch — by
# measuring the output rather than asserting on a return value.
#
# They ran only on a maintainer's laptop until now, which is exactly the gap
# that let three separate bugs through in one day: a missing .gitkeep that
# only broke a clean checkout, an FFmpeg listener colliding with the Go one
# that hid behind a local FFmpeg without libsrt, and a README publish URL that
# stopped working. A laptop is not a clean machine.
#
# A matrix rather than one job: each suite is a few minutes, they are
# independent, and a failure names itself in the checks list instead of being
# buried in a combined log.
acceptance:
name: "acceptance: ${{ matrix.suite }}"
permissions:
contents: read
runs-on: ubuntu-latest
# NO JOB-LEVEL `if:` HERE, DELIBERATELY, and #351 is why. These are required
# checks. A skipped matrix job never expands its matrix,
# so the per-leg contexts are not skipped -- they do not exist, and the
# pull request can never satisfy branch protection. The `if:` is on every
# step below instead: the matrix expands, all twelve report, and a
# documentation-only run costs a runner allocation rather than a suite.
needs: changes
env:
# See the deadline note on the suite step below. The library's 900s
# default is for a laptop, where nothing outranks it; here it has to fire
# before the job ceiling, and its clock starts minutes after the job's.
POLY_WATCHDOG_SECS: 600
# A ceiling, because GitHub's default is SIX HOURS. These suites publish
# real streams and wait on them, so the failure mode when something does
# not arrive is a wait, not a crash -- exactly the shape that sits there
# burning a runner. acceptance-renditions ran 25 minutes against a ~2
# minute local baseline before this existed.
timeout-minutes: 20
strategy:
# Never cancel siblings. When two suites break, both answers are wanted —
# fail-fast would hide the second and turn one fix into two round trips.
fail-fast: false
matrix:
suite:
- acceptance # end to end: ingest, fan-out, persistence
- acceptance-audio # per-track RMS through a bandpass
- acceptance-renditions
# The LADDER: three tiers at once, which acceptance-renditions cannot
# cover because every test in it uses ONE rendition and one is the
# only number it ever counts. This counts three, and counts them going
# up and down, which is the claim docs/ENCODING.md attaches a bill to.
#
# ADDING A LEG HERE ADDS A REQUIRED STATUS CHECK. The matrix is named
# `acceptance: ${{ matrix.suite }}` and every leg of it is required by
# branch protection, so this is not a free addition: the new context
# has to be added to the ruleset or pull requests will show it as an
# extra check that nothing waits for.
#
# It belongs in the matrix rather than in a workflow of its own for
# the same reason acceptance-renditions does: it needs nothing but
# FFmpeg and the built binary, it contacts no external service, and
# the thing it guards -- how many encodes an operator is paying for --
# is broken by ordinary changes to the engine's reconcile. A suite
# that only runs on dispatch is a suite that reports a regression
# after it has shipped. Measured at 75s locally, which is inside the
# spread of the legs already here.
- acceptance-ladder
- acceptance-encoders
- acceptance-tls
- acceptance-pull # dial-out ingest
- acceptance-playlist-phase0 # scheduled file broadcast, no encoder
- acceptance-synth # silence tier and the failover slate
- acceptance-failover # switch without restarting a destination
- acceptance-postprod
# A recording is finalised on SIGTERM rather than truncated. The unit
# file has promised this since it was written and nothing checked it;
# on Linux it was false, and the failure is silent -- a truncated
# Matroska is exactly the size it was when the process died, so only
# a decode tells you.
- acceptance-recording-stop
# Retained MQTT telemetry. Brings up eclipse-mosquitto as a container
# rather than installing a broker on the runner, so it needs no
# apt step of its own -- Docker is already present on ubuntu-latest.
- acceptance-mqtt
steps:
# EVERY STEP CARRIES THE GATE. There is no way to end a job early from a
# step without failing it, so "documentation only" has to be spelled out
# on each one rather than checked once at the top.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: needs.changes.outputs.code == 'true'
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
if: needs.changes.outputs.code == 'true'
with:
go-version-file: go.mod
cache: true
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
if: needs.changes.outputs.code == 'true'
with:
node-version: 24
cache: npm
cache-dependency-path: ui/package-lock.json
# Says out loud why a green check did no work, so a reader of the checks
# list is never left guessing whether the suite ran or silently no-opped.
- name: Documentation-only change, so this suite did not run
if: needs.changes.outputs.code != 'true'
run: |
echo "::notice title=${{ matrix.suite }} skipped::every changed path was documentation, so this suite could not be affected by it. See the 'which changes' job."
# MOVED TO THE PINNED 8.1 BUILD the other Linux jobs already cache.
#
# This was attempted once and reverted, because acceptance-failover went red
# on its first run and nobody knew why. Now it is known: the mismatch
# destination restarts, joins the relay late, and is handed no SPS/PPS --
# #460. Measured at 5 runs in 24 on 8.1.2 against 0 in 24 on 6.1.1, so the
# switch does not CAUSE that failure, it stops hiding it. The step that trips
# over it now names it and does not fail the suite for it.
#
# Two things this buys, both of which were the original argument:
#
# the suites stop testing an FFmpeg nobody runs. Users get 8.1.2 from the
# Dockerfile pin; CI tested apt's 6.1.1, and a defect fatal on one and
# survivable on the other was invisible here for months. That is how #398
# and #460 reached a release candidate unnoticed.
#
# it removes the un-mirrored apt fetch #430 is about -- the largest single
# source of red CI, which retries could not fix because its budget never
# fit its ceiling.
- name: Cache FFmpeg
if: needs.changes.outputs.code == 'true'
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /tmp/ffcache
key: ffmpeg-${{ runner.os }}-n8.1-latest-linux64-gpl-8.1-v1
- name: Install FFmpeg
if: needs.changes.outputs.code == 'true'
timeout-minutes: 8
env:
FFCACHE: /tmp/ffcache
run: |
# Ubuntu's own package is 6.1.1, and the Docker image ships 8.1.2.
# Testing against 6.1.1 meant CI never exercised the FFmpeg users
# actually get, and the two differ in ways that matter: 6.1.1 accepts
# an 80-channel amerge where 8.1 stops at 64, and 6.1.1 cannot demux
# multitrack FLV at all. Pin the same 8.1 line the image uses.
set -euo pipefail
cd /tmp
# -c rather than a heredoc: inside a YAML block the heredoc body and
# its terminator are indented, which Python reads as a bad indent and
# bash does not accept as a delimiter.
# RETRIED, because this is one un-mirrored third-party download and
# every Linux job in this file makes it. It has hard-failed two jobs
# in one day -- acceptance-postprod and acceptance-pull -- both by
# sitting on a stalled connection until the 6-minute ceiling fired.
# A stuck mirror is exactly what that ceiling is for, but a single
# attempt turns a transient stall into a red job and a manual rerun.
#
# curl rather than urllib: --retry understands the difference between
# a connection that failed and one that is merely slow, and
# --retry-all-errors covers the 5xx a release CDN returns under load.
# The per-attempt --max-time is well inside the step ceiling, so three
# attempts still cannot outlast it -- the ceiling keeps its meaning.
# A CACHE HIT SKIPS THE DOWNLOAD ENTIRELY, and that is the point.
#
# This artefact failed SEVEN times in one day across Linux and Windows
# -- 503s and stalls from the release CDN -- while already wrapped in
# `curl --retry 3 --retry-all-errors` AND the three-attempt loop below.
# It has retries and still lands red, because a retry re-asks a host
# that is refusing. The cache stops asking.
#
# Same treatment the Playwright browser got: a pinned artefact need not
# be fetched on every run after the first.
if [ -x "$FFCACHE/ffmpeg" ] && [ -x "$FFCACHE/ffprobe" ]; then
echo "ffmpeg restored from cache"
else
for attempt in 1 2 3; do
# --proto/--proto-redir pin the whole exchange to https. -L
# follows redirects, and without these curl will happily follow an
# https -> http hop and fetch the binary CI is about to run over
# plaintext. SonarCloud githubactions:S6506 flagged the copy of
# this block added for the acceptance job; the other two had the
# same hole and are fixed here too, because leaving a known-bad
# copy beside a fixed one is how the fix gets reverted later by
# someone copying the wrong neighbour.
if curl -fsSL --proto '=https' --proto-redir '=https' \
--retry 3 --retry-all-errors --retry-delay 5 \
--connect-timeout 20 --max-time 90 \
-o /tmp/ff.tar.xz \
"https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-linux64-gpl-8.1.tar.xz"; then
break
fi
echo "ffmpeg download attempt $attempt failed"
[ "$attempt" = 3 ] && exit 1
sleep 5
done
mkdir -p ffbuild && tar xf ff.tar.xz --strip-components=1 -C ffbuild
mkdir -p "$FFCACHE"
cp ffbuild/bin/ffmpeg ffbuild/bin/ffprobe "$FFCACHE/"
fi
sudo install -m 0755 "$FFCACHE/ffmpeg" /usr/local/bin/ffmpeg
sudo install -m 0755 "$FFCACHE/ffprobe" /usr/local/bin/ffprobe
# Fail loudly rather than silently testing a build without SRT.
ffmpeg -hide_banner -protocols | tr ' ' '\n' | grep -qx srt
ffmpeg -hide_banner -version | head -1
# ffprobe TOO, and not because the install could half-succeed. #187:
# every test that proves the upload probe gate works starts by looking
# ffprobe up and skipping if it is absent, so an ffprobe that stopped
# arriving would have taken the whole gate's verification with it while
# this step still printed an ffmpeg version and passed. The one binary
# that was verified was the one the skips did not turn on.
ffprobe -hide_banner -version | head -1
# The gate is "gofmt named no files", and it has to rest on gofmt having
# RUN. It used to be `gofmt -l ./cmd ./internal | tee /tmp/fmt && test !
# -s /tmp/fmt`, and the default shell for a `run:` block is `bash -e {0}`
# -- no pipefail, unlike an explicit `shell: bash`. So the pipeline's
# status was tee's, gofmt's was discarded, and a gofmt that died on a
# parse error wrote nothing to /tmp/fmt and left `test ! -s` to report
# the tree as correctly formatted. Every way of failing to check produced
# the same green as checking and finding nothing.
#
# A command substitution under set -e instead: gofmt's own exit status
# now fails the step, and the emptiness of its output is a separate
# question asked afterwards.
# Built here rather than earlier so a build failure is not reported as a
# missing binary by every one of thirteen suites at once. `make build` and
# not `go build ./cmd/...`: the suites run the real build.
- run: make build
if: needs.changes.outputs.code == 'true'
- name: ./scripts/${{ matrix.suite }}.sh
if: needs.changes.outputs.code == 'true'
timeout-minutes: 12
run: ./scripts/${{ matrix.suite }}.sh
# What was the process actually doing when the deadline fired?
#
# SIGQUIT is the point of this step. Go's default handler dumps every
# goroutine's stack and exits, so a server blocked on a read nobody will
# satisfy names the function it is blocked in. Without it the artifact
# holds the suite's own log, which by definition stops at the line before
# the thing that never happened.
#
# if: failure() rather than always(): on a green run there is no process
# left to signal and nothing to explain.
#
# The gate is ANDed on for completeness rather than for effect -- #357
# missed this one step, and it was harmless only because a documentation
# -only run has no suite step to fail, so `failure()` was never true. That
# is a coincidence of this job's shape, not a property, and the rule in
# internal/testenv/docsgate_test.go found it.
- name: Diagnostics (what was it waiting for)
if: failure() && needs.changes.outputs.code == 'true'
# Never fail the job from here. This step exists to describe a failure
# that already happened; a diagnostic that turns a legible suite
# failure into a confusing one about ps flags is worse than none.
continue-on-error: true
timeout-minutes: 3
run: |
set +e
echo "=== processes ==="
ps -eo pid,ppid,etime,stat,args | grep -E "polyemesis|ffmpeg|ffprobe|mosquitto" | grep -v grep
echo
echo "=== goroutine dumps ==="
# Every polyemesis, not just one: several suites run an ingest and a
# destination side, and the stalled one is not reliably the first.
pids=$(pgrep -x polyemesis)
if [ -z "$pids" ]; then
echo "no polyemesis process is running -- the stall is in the SCRIPT,"
echo "not the server: a wait on a file, a port or an ffmpeg that already exited."
else
for pid in $pids; do
echo "--- SIGQUIT $pid ---"
kill -QUIT "$pid"
done
# The dump goes to the process's own stderr, which the suites
# redirect into their log directory, so it lands in the artifact
# below rather than here.
#
# OBSERVED, NOT ASSUMED. This used to be a bare `sleep 5` -- a fixed
# interval standing in for an observation, in the file this round
# edited, in the diagnostic that exists to explain hangs. SIGQUIT to
# a Go program writes the dump and then exits, so the process no
# longer existing IS the dump having been written; poll for that and
# stop as soon as it is true.
#
# The 5s ceiling is kept as the give-up, not as the wait: a process
# wedged in uninterruptible I/O may never take the signal, and this
# step must not become the hang it was written to explain.
for _ in $(seq 1 20); do
still=""
for pid in $pids; do
kill -0 "$pid" 2>/dev/null && still="$still $pid"
done
[ -z "$still" ] && break
sleep 0.25
done
if [ -n "$still" ]; then
echo "still alive 5s after SIGQUIT:$still -- the dump for these may be"
echo "absent or truncated in the uploaded logs."
fi
fi
echo
echo "=== listening sockets ==="
ss -lntup 2>/dev/null | head -40
echo
echo "=== tail of every suite log ==="
# Last lines only: these get large, and the useful part of a stall is
# always the end -- the last thing that happened before nothing did.
for f in $(find /tmp artifacts logs -maxdepth 3 -name '*.log' -newermt '-30 minutes' 2>/dev/null | head -40); do
echo "--- $f ---"
tail -30 "$f"
done
exit 0
# The logs are where a measurement failure is actually diagnosed: the RMS
# figures, the process output, what FFmpeg said. Losing them turns a red
# check into a guess.
#
# THE TIMELINE EVIDENCE GOES WITH THEM, and #126 is why. The only occurrence
# of that bug anybody has seen outside a deliberate measurement happened
# here, in CI, and it left nothing behind but a count in the step output --
# no recording to re-probe, no packet list, no ledger. So the derived
# artifacts travel too: dts.csv is the packet list both halves of the check
# read, seams.txt is the engine's handover ledger, and onair.mkv is the
# recording, which is the only one of the three that can answer a question
# nobody thought to ask before the run.
#
# The mkv was 13 MB on a full local run of the failover suite, measured
# rather than guessed. That is the price of not waiting another three weeks
# for the next occurrence. Only that suite writes one; the others match
# nothing here and if-no-files-found already covers it.
- name: Upload logs
# always(), but not when the suite never ran: a documentation-only job
# has no artifact to collect, only a warning about its absence.
if: always() && needs.changes.outputs.code == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ${{ matrix.suite }}-logs
path: |
/tmp/polyemesis-acceptance*/**/*.log
/tmp/polyemesis-acceptance*/**/server.log
/tmp/polyemesis-acceptance*/dts.csv
/tmp/polyemesis-acceptance*/seams.txt
/tmp/polyemesis-acceptance*/data/recordings/onair.mkv
# The status payload captured at the instant a destination reported
# no process (#462). Without this the suite dumps it and the runner
# discards it -- which is how the one artifact that answers the
# question has been thrown away on every failing run so far.
/tmp/polyemesis-acceptance*/status-at-failure-*.json
/tmp/polyemesis-acceptance*/status-at-baseline-timeout.json
/tmp/polyemesis-acceptance*/engine-log-at-baseline-timeout.txt
/tmp/polyemesis-acceptance*/baseline-wait.txt
if-no-files-found: ignore
retention-days: 7