FFmpeg 8.1 writes only a header for a file destination fed a mid-stream resolution change

[rainmanjam]

Found by switching the acceptance jobs from Ubuntu's FFmpeg 6.1.1 to the pinned 8.1 build the other CI jobs already use. The switch was reverted from #397; this issue is the finding.

The bisect

Commit	acceptance-failover
b02d083 — before the switch	success
198b867 — before the switch	success
138fe97 — the switch	FAILURE
f681628 — after	FAILURE

Two green before, two red after, and nothing else changed in between. Every other suite passed on all four.

The failure

FAIL  the mismatch destination produced 18154ms of media and its file holds 293 bytes
      it WAS being delivered to, so this is not the closed-hub case. A file
      destination truncates and reopens when it restarts -- read the restart
      count above; if that is still 0 the truncation came from outside this step.
the mismatch recording declares:  -- for content that is 1280x720@60 for half its length

Summary
  26 passed, 1 failed

293 bytes is an EBML header and nothing after it. ffprobe reports no stream at all — the trailing declares: line is empty where it should print width,height,r_frame_rate.

The destination in question is the failover suite's mismatch destination: the one deliberately fed content whose parameters change halfway through, 1280x720@60 for half its length. out_time says 18154ms of media was produced, so delivery was happening; the muxer wrote a header and then nothing reached the file.

Why this probably matters to real users

Users run 8.1.2 — the Dockerfile pins FFMPEG_VERSION=8.1.2-r0. CI's 6.1.1 is the version nobody runs, which is exactly why the pin exists elsewhere in ci.yml:

Ubuntu's own package is 6.1.1, and the Docker image ships 8.1.2. Testing against 6.1.1 meant CI never exercised the FFmpeg users actually get, and the two differ in ways that matter.

So the plausible reading is that a file destination fed a mid-stream resolution change produces an empty recording on the FFmpeg version users actually have, and CI has never been able to see it. The pin did what it was added to do on its first run.

The alternative reading is that the suite's expectations are tuned to 6.1.1's tolerance for a parameter change mid-Matroska. Both are worth knowing and neither is established yet.

What has NOT been established

• Whether 8.1 refuses the parameter change, or accepts it and fails to flush.
• Whether this reproduces outside CI, on a real failover between mismatched sources.
• Whether other file destinations are affected, or only the mismatch case.
• Whether the Docker acceptance suites — which DO run 8.1.2 — already cover this and pass, which would narrow it considerably.

That last one is the cheapest next step.

Also blocked on this

Moving the acceptance jobs to the cached, pinned 8.1 build is worth doing on its own merits and cannot land until this is understood:

• It removes an uncached third-party fetch that has hit the 6-minute step ceiling four times — acceptance-postprod, acceptance-pull, acceptance-mqtt, acceptance-audio — each stalled mid-acquisition against azure.archive.ubuntu.com. Acquire::Retries=3 was added to bound it and did not, for the reason the unit job already records: "a retry re-asks a host that is refusing. The cache stops asking."
• The acceptance suites are the ones pushing real multitrack audio end to end, so they are the jobs that most need the version users run.

The reasoning is left at the step in ci.yml so the next person to notice finds out it was tried.

[rainmanjam]

Correction: my diagnosis in the issue body is probably wrong

I could not reproduce this, and the mechanism I implied does not appear to exist. Recording that before anyone spends an afternoon on it.

Reproduced against 8.1.2 — the exact version users get

Three scenarios, all with a mid-stream geometry change (1280x720@60 → 1920x1080@30, separate H.264 encodes concatenated as MPEG-TS), -c copy into Matroska:

Scenario	Result
File input → .mkv	4,692,603 bytes, 8.03s, declares 1280,720,30/1
Live pipe, realtime, SIGTERM at 6s	3,592,095 bytes, 6.53s
Live pipe, realtime, SIGKILL at 6s (no finalisation)	3,145,728 bytes

The muxer takes the first packets' parameters, writes them once, and accepts everything after without complaint — exactly the behaviour acceptance-failover's own comment describes and depends on. It survives an unfinalised kill with clusters on disk. No 293-byte file in any case.

And the mechanism I suggested does not exist

A research pass over the FFmpeg tree reports, as verified: there is no commit in libavformat/matroskaenc.c between 6.1 and 8.x that added a check rejecting or deferring a video parameter change. Video track parameters have always been written once and never revisited; the muxer only inspects AV_PKT_DATA_NEW_EXTRADATA for audio.

So "8.1 started refusing what 6.1.1 tolerated" — the reading in the issue body — is not supported.

What my bisect was actually worth

Less than I said. I called it "unambiguous". It was two runs green, two runs red, on an assertion the suite itself documents as timing-sensitive:

the file reads zero for the whole run even when everything is working, so it cannot tell these apart and out_time can

The same healthy run reads 12845056 bytes locally and 0 in CI, because a flush is a function of buffer pressure and timing.

Two samples either side of a change, on a check with a known history of environment-dependent readings, is weak evidence. I presented it as strong.

What is still worth investigating

One mechanism remains plausible and is explicitly inferred, not verified: FFmpeg 7.0 rewrote the CLI into a multi-threaded scheduler (fftools/ffmpeg_sched.c, sync_queue.c), and -progress out_time is measured on the demuxer/scheduler side rather than reflecting bytes written by avio. If a muxer thread stalls, the demuxer keeps consuming input and reporting out_time while nothing reaches disk — which would produce exactly the "18154ms produced, 293 bytes on disk" signature. Related: trac #11469.

My reproduction used a pipe, not polyemesis's actual path (relay → SRT/RTMP → destination process) and not a live cut under load. That gap is where the remaining doubt lives.

Also relevant and independently true: FFmpeg upstream considers mid-stream parameter changes during streamcopy unsupported by design (trac #4383, #6717), with segmenting or transcoding as the supported answers. That is worth knowing regardless of what caused this failure, because the failover mismatch case is doing the unsupported thing on purpose.

Where that leaves the FFmpeg pin

The revert stands, but for a weaker reason than the issue body gives: not "8.1 breaks this" but "one CI failure I have not explained, and I will not switch the version the acceptance suites run under while it is unexplained."

The reliability half — bounding each apt attempt so a stall costs 100s rather than the whole job — landed separately and does not depend on any of this.

Next step, and it is cheap: re-run acceptance-failover a few times on an unchanged branch to establish its actual flake rate before attributing anything to a version change. flake-rate.yml exists for exactly this.

[rainmanjam]

Dispatched flake-rate.yml against this suite — 20 runs, no code change: https://github.com/rainmanjam/polyemesis/actions/runs/31918793917

This is the step I said was cheapest, and it should have come before the attribution rather than after it.

Why 20 and not 5

The workflow states its own detectable floor at 95% confidence: 5 runs = 45%, 10 = 26%, 20 = 14%, 30 = 9.5%, 40 = 7.2%. At the default of 5, a clean result would only bound the rate below 45% — which would not distinguish "stable" from "fails one run in three" and would have told me nothing useful about a two-green/two-red observation.

20 costs more runner time and buys a floor low enough to be worth having. If it comes back 0/20, that bounds the rate below 14% and the FFmpeg version becomes a live suspect again. If it fails even once with no code change, the two-green-then-two-red bisect I ran was coincidence, exactly as suspected in the correction above.

Either result is worth the runner minutes, which is the test of whether to spend them.

What this does not settle

The floor is a floor. A green 20-run result means "not measurable here", not "not flaky" — the workflow's own input description says so, and it is the sentence people skip. If it comes back clean I will not report that as "the suite is fine".

[rainmanjam]

The measurement came back, and it settles the flake question

flake-rate.yml run 31918793917, dispatched from the comment above:

RESULT acceptance-failover: 0/20 failed

Recording the reading here because the result has been sitting in a completed run for two days and the issue still reads as open on the question.

Zero failures in twenty runs bounds this suite's flake rate below 14% at 95% confidence. Per the interpretation stated in advance:

If it comes back 0/20, that bounds the rate below 14% and the FFmpeg version becomes a live suspect again.

So the two-green-then-two-red bisect was probably not coincidence. A suite that does not flake at ≥14% producing two reds immediately after the switch is unlikely to be chance.

Where that leaves this issue

Two things are now both true, and they are not in conflict:

• The mechanism in the issue body is wrong. No matroskaenc.c commit between 6.1 and 8.x rejects a mid-stream video parameter change, and the reproduction produced megabytes rather than a 293-byte file. That retraction stands.
• The switch still appears to cause the failure. The flake explanation is now excluded, so something real about the pinned 8.1 build breaks this suite. We do not know what.

The title is therefore misleading and should probably become something like "acceptance-failover fails against the pinned FFmpeg 8.1 build, cause unknown" — the observation, without the mechanism that did not survive checking.

The remaining hypothesis, still inferred

The one candidate the correction names and does not verify: FFmpeg 7.0 rewrote the CLI into a multi-threaded scheduler, and -progress out_time is measured on the demuxer/scheduler side rather than reflecting bytes reaching avio. A stalled muxer thread would report progress while nothing is written — which matches "18154ms produced, 293 bytes on disk" exactly.

Testing that needs a reproduction against polyemesis's real path (relay → SRT/RTMP → destination process) under a live cut, not the pipe used before. That gap is where the doubt lives, and it is the same gap that blocks item 4 of #430.

[rainmanjam]

The cheapest next step has an answer, and it is no

This issue names its own cheapest next step:

Whether the Docker acceptance suites — which DO run 8.1.2 — already cover this and pass, which would narrow it considerably.

They do not cover it. The mismatch destination exists in exactly one place:

scripts/acceptance-failover.sh
scripts/acceptance_failover_driver.go

and acceptance-failover runs on Ubuntu's 6.1.1. No Docker suite exercises a file destination fed a mid-stream resolution change, so no existing run can settle this either way. Narrowing it requires a deliberate failover run against 8.1, which is work rather than a lookup.

That matters for sequencing: the first comment on this issue already retracted the bisect as weak evidence, and the remaining cheap discriminator has now been ruled out too. What is left is the measurement.

A signature worth comparing

#453's CI failure is the same destination failing the same way under a completely unrelated perturbation:

	media produced	file
#398 (FFmpeg 8.1)	18154ms	293 bytes — EBML header, no stream
#453 (a sibling destination added/removed)	8320ms	0 bytes

Two different causes producing "media was delivered, the file was never finalised" on the same destination is weak evidence for either specific cause and reasonable evidence that the mismatch destination's file finalisation is fragile in its own right. Worth holding in mind before attributing this to a version bump.

Blocking

#430 is gated on this and cannot move until it is understood.

[rainmanjam]

A newer FFmpeg passes this cleanly, and the suite is intermittent on identical code

Two findings from today, both of which change what a measurement here has to look like.

1. FFmpeg 9.0.1 does not reproduce it

acceptance-failover run on a Mac against FFmpeg 9.0.1, step 10 — the mismatch destination:

mismatch produced media: settled=7920ms back=16277ms; file=4577701 bytes
PASS  the mismatch destination actually wrote its output (4577701 bytes)
the mismatch recording declares: 1920,1080,30/1 -- for content that is 1280x720@60 for half its length

4.5 MB and a valid stream declaration, against the 293 bytes reported here. So "modern FFmpeg refuses a mid-stream resolution change" does not hold as a general claim — a newer build than the one this issue blames handles it correctly. That leaves two readings:

• 8.1 specifically regressed and 9.x fixed it. Then the Dockerfile's FFMPEG_VERSION=8.1.2-r0 pin is shipping the affected version and this is a real product concern.
• The original failure was environmental, consistent with the first comment here already retracting the bisect as weak.

9.0.1 is a fourth version nobody ships (CI has 6.1.1, Docker pins 8.1.2), so this narrows the question without answering it.

2. A single run cannot settle this

While investigating #453 I ran acceptance-failover twice on identical code, with the reverted sibling step restored:

run	step 11	file
1	FAIL	0 bytes, restarts 1 -> 1
2	PASS	301859 bytes, restarts 0 -> 0

This suite has demonstrated intermittency today independent of any FFmpeg version. So a single run on a pinned 8.1 build — green or red — proves nothing either way, in the same way the original two-green-two-red bisect proved nothing.

What the measurement has to be

Repeated runs on the pinned 8.1 build, reported as a rate rather than a verdict, with the same suite run on 6.1.1 as a control. That is what this issue's own next-step asked for ("a flake-rate measurement that has not been run against this question") and has still not been done.

Planned on the OVH host against the pinned 8.1 build the other CI jobs already cache.

Why it matters for the release

If 8.1 turns out to be genuinely affected, the version users get is the affected one, and that is a v0.7.0 concern rather than a CI concern. The tag is being held on this answer.