Skip to content

fix(transfer): interleave itemize rows in flist order (run_pipelined, step 1)#6560

Merged
oferchen merged 1 commit into
masterfrom
fix/286-2-itemize-treeorder-step1
Jul 13, 2026
Merged

fix(transfer): interleave itemize rows in flist order (run_pipelined, step 1)#6560
oferchen merged 1 commit into
masterfrom
fix/286-2-itemize-treeorder-step1

Conversation

@oferchen

Copy link
Copy Markdown
Owner

Summary

Display-only fix for the itemize (-i / -ii) row ordering on a pull. Transferred data, wire bytes, and metadata application are byte-identical; only the order of itemize rows changes. Directories are still created before their files - only the itemize emission moves to a deferred flush.

The divergence

oc's two-phase receiver emits every directory (.d) itemize row in the directory-creation pass, then every file (.f) row from the candidate pass. Upstream interleaves them in flist index (tree) order, so a directory row immediately precedes its children.

For the flist a/ a/f1 b/ b/f2:

Order
upstream .d a/, >f a/f1, .d b/, >f b/f2
oc (before) .d a/, .d b/, >f a/f1, >f b/f2

Upstream does a single flist-index-order walk, itemizing each entry as it is reached:

  • generator.c:2329-2344 generate_files - one loop over cur_flist->sorted[i] in index order, recv_generator per entry.
  • generator.c:1480-1483 recv_generator - directory itemize() at its own flist position.
  • log.c:788 rwrite prints the row. The flist is sorted by f_name_cmp (flist.c:1732-1736) = tree order.

Design: deferred single-pass ordered flush

oc already reassembles parallel output into upstream serial order, so a post-transfer flush drained in flist-index order is byte-identical to upstream's serial walk.

  • New per-transfer buffer itemize_rows: RefCell<BTreeMap<usize, Vec<String>>> on ReceiverContext (key = flist index -> automatic tree order; Vec tolerates a phase-2 redo re-itemizing an entry; RefCell suffices because every emit site runs on the main driver thread, never a rayon worker).
  • record_itemize(flist_idx, iflags, entry) renders via the existing render_itemize_line and buffers the row. It preserves emit_itemize's client-mode visibility gate exactly - a server receiver's row still travels as wire iflags and is printed by the client's sender (log.c:822 gates the FCLIENT write on !am_server), so nothing is recorded there and no row is doubled.
  • flush_itemize_rows(writer) drains the BTreeMap in ascending key order through the existing client sink, called once immediately before finalize_transfer.
  • An emit_or_record_itemize dispatcher routes each emit site: it buffers when the per-transfer defer_itemize toggle is set, and emits immediately otherwise.

Scope: only run_pipelined is converted

The emit sites live in helpers (create_directories, build_files_to_transfer, and the shared pipeline loop) that are also used by the sync, incremental, and async receive paths. To convert only run_pipelined without a sprawling diff, the deferral is an opt-in per-transfer toggle:

  • defer_itemize defaults to false - every other caller emits immediately, exactly as before (behaviour-identical).
  • run_pipelined sets defer_itemize = true and calls flush_itemize_rows before finalization.

Converted emit sites (all now route through the dispatcher): the directory row in create_directories, the transfer row and the up-to-date/no-change metadata row in build_files_to_transfer, and the server-mode transfer row in the pipeline loop (a no-op for display in both the old and new code).

Follow-ups (left unchanged here): the sync, pipelined_incremental, and all async receive twins; the -v name lines; and the --delete-pass rows.

Test

deferred_itemize_rows_interleave_in_flist_index_order drives the real record sites (create_directories + build_files_to_transfer) with deferral on over an a/ a/f1 b/ b/f2 flist and asserts the buffered rows land interleaved in flist-index order (.d a/, >f a/f1, .d b/, >f b/f2), not batched (.d a/, .d b/, ...). It fails if the two-phase batch emission returns: an immediate emit leaves the buffer empty, and recording without the per-index key would order both directory rows ahead of both file rows.

Validation

  • cargo fmt --all - clean.
  • cargo clippy -p transfer --all-targets --all-features --no-deps -- -D warnings - no issues.
  • Full test suite runs in CI. An -ii end-to-end seal against a real upstream binary on the Linux validation host is required before merge.

@github-actions github-actions Bot added the bug Something isn't working label Jul 13, 2026
… step 1)

The two-phase receiver emitted every directory (.d) itemize row in the
directory-creation pass before any file (.f) row from the candidate pass,
so `a/ a/f1 b/ b/f2` printed .d a/, .d b/, >f a/f1, >f b/f2. Upstream
itemizes in a single flist-index-order walk (generator.c:2329-2344 ->
recv_generator -> itemize, log.c:788 rwrite), where a directory row
immediately precedes its children: .d a/, >f a/f1, .d b/, >f b/f2.

Display-only: transferred data, wire bytes, and metadata application are
unchanged; directories are still created before their files. Only the
itemize emission moves to a deferred, flist-index-keyed flush.

Adds a per-transfer buffer (itemize_rows: RefCell<BTreeMap<usize,
Vec<String>>>) plus record_itemize / flush_itemize_rows / a
emit_or_record_itemize dispatcher, gated by an opt-in defer_itemize
toggle. The dir/file/no-change emit sites in the shared create_directories,
build_files_to_transfer, and pipeline loop now route through the
dispatcher, which buffers when deferral is on and emits immediately
otherwise. record_itemize preserves emit_itemize's client-mode visibility
gate exactly (a server receiver's row still travels as wire iflags).

Only run_pipelined opts in (sets defer_itemize and flushes before
finalize); the sync, incremental, and async receive paths keep emitting
immediately and are converted in follow-ups. Requires an lxhost -ii seal
before merge.
@oferchen oferchen force-pushed the fix/286-2-itemize-treeorder-step1 branch from df149f8 to 57a3757 Compare July 13, 2026 17:37
@oferchen oferchen merged commit 2275b28 into master Jul 13, 2026
64 checks passed
@oferchen oferchen deleted the fix/286-2-itemize-treeorder-step1 branch July 13, 2026 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant