A half-delivered packet stops the whole inbound batch, and nothing says when one is ready to read

[andy5995]

Written by Claude (Opus 5), an LLM made by Anthropic, and posted at andy5995's direction.

@RealDeuce raised the handoff contract in discussion #121:

Also, a well documented method of triggering packet import and possibly packet export. You want to make sure that each step is atomic so things aren't transferring incomplete packets or importing a packet while data is being mutated etc. When generated packets are finalized, there should be an atomic operation that makes them available to be moved. For import, there's should be a way to indicate a packet is ready for import. Simple existence checks can lead to impossible to debug race conditions.

The export half already behaves that way: a packet is written under a temporary name and renamed into place, so a mover only ever sees a finished .brp. Nothing states that anywhere, so a transport author has no rule to rely on and nothing stops the behaviour changing.

The import half has no rule at all.

What goes wrong on import

Inbound is a plain directory scan — every .brp is read and parsed. A transport that copies a packet into that directory non-atomically (cp, scp, an FTP upload, a filebox sync) exposes a partial file under its final name, and a run that lands mid-copy reads it.

Parsing a truncated packet fails, and the failure aborts the whole batch: every packet after it in that run is skipped, including intact ones from other boards. A transfer that goes on to complete will parse on the next run, so the usual cost is a delayed batch. A transfer that is interrupted leaves a permanently unparseable file, and every later run aborts at the same place until someone deletes it by hand.

The open question

Which readiness signal? The warning that "simple existence checks can lead to impossible to debug race conditions" cuts against the obvious answer, because rename-into-place is an existence check — atomic on a POSIX filesystem, and not on classic NFS without locking, which the same comment names. So the choice between rename, a sidecar marker file, and a manifest is not settled, and it decides what gets written down.

@RealDeuce — of those, which would you expect a schlepping service to implement?

Shape of the fix

1. Write the contract down in docs/dev/ibbs-packet-format.md: what a mover may assume about a packet in outbound, and what it must do to make one visible in inbound.
2. Make the inbound scan skip an unparseable packet and carry on, so one bad file cannot hold up the rest.
3. Whatever readiness signal comes out of the question above.

Point 2 is worth doing regardless — a packet can be corrupt for reasons that have nothing to do with atomicity.