Summary
RetinaNet cannot run at the dataset sizes the CLOSED rules themselves mandate. Rule 3.1.2 scales the minimum dataset to 5× aggregate client host memory, which for realistic client fleets lands in the multi-million-file range — where the DLIO/PyTorch dataloader enters a barrier busy-poll livelock and never finishes epoch 1. We located the failure boundary: ≈4.2M files still completes, ≈8.3M livelocks. Reproduced on a late-June main (3.0.2x) and again on v3.0.34 (bf59a06) with the current DLIO pin (DLIO_local_changes @ 59f94801) — the recent pin bump does not resolve it.
Environment: 4 client nodes (64-core Xeon 6767P, 256 GB RAM, kernel 6.8), 3 storage servers(48 NVMe), RetinaNet JPEGs (record_length_bytes=322957, buffered I/O), 1-epoch open-mode probes bounded with timeout.
Measured boundary
| num_files_train |
CLOSED meaning (256 GB/node) |
Result |
| 1,170,301 (upstream default) |
— |
completes in 90–150 s (na24–na96) |
| 4,172,524–4,190,000 |
1-node CLOSED size |
completes: na8 in 1225 s (AU 86.1, warm) / 1412 s (AU 74.7, semi-cold) |
| 8,345,048 |
2-node CLOSED size |
livelock — killed at 3600 s; NIC rx flat at ≈3.6 MB/s for the whole hour (healthy ≈1.2 GB/s) |
| 16,690,097 |
4-node CLOSED size |
livelock — killed at 4200 s, same signature; rank-count independent (na16 = na64) |
Diagnostics
Request
First, we would welcome verification — our measurements could be wrong, or we may have missed an intended configuration, so please let us know if this does not reproduce on your side. As far as we can tell, however, there is currently no configuration that both satisfies rule 3.1.2 and completes: the mandated file count is derived from client host memory, none of the CLOSED-tunable parameters reduce it, and that count lies past the livelock boundary for any fleet with two or more 256 GB-class client nodes.
If confirmed, could this be triaged as a CLOSED blocker for RetinaNet — either (a) fixing the per-step O(N) behavior for large num_files_train, or (b) decoupling the CLOSED minimum-dataset rule from this pathology until it is fixed?
Summary
RetinaNet cannot run at the dataset sizes the CLOSED rules themselves mandate. Rule 3.1.2 scales the minimum dataset to 5× aggregate client host memory, which for realistic client fleets lands in the multi-million-file range — where the DLIO/PyTorch dataloader enters a barrier busy-poll livelock and never finishes epoch 1. We located the failure boundary: ≈4.2M files still completes, ≈8.3M livelocks. Reproduced on a late-June
main(3.0.2x) and again on v3.0.34 (bf59a06) with the current DLIO pin (DLIO_local_changes @ 59f94801) — the recent pin bump does not resolve it.Environment: 4 client nodes (64-core Xeon 6767P, 256 GB RAM, kernel 6.8), 3 storage servers(48 NVMe), RetinaNet JPEGs (record_length_bytes=322957, buffered I/O), 1-epoch open-mode probes bounded with timeout.
Measured boundary
Diagnostics
comm.barrier()busy-poll, all ranks at 94–100% CPU; dataloader workers idle (0 JPEG opens in 6 s); ceph in-flight ops = 3 per node — storage idle.open()+read()over the same files, 128 workers on the same 4 nodes) sustains ≈91k files/s (27.4 GiB/s). The pathology is in the benchmark's dataloader path, not in storage or rank count.Request
First, we would welcome verification — our measurements could be wrong, or we may have missed an intended configuration, so please let us know if this does not reproduce on your side. As far as we can tell, however, there is currently no configuration that both satisfies rule 3.1.2 and completes: the mandated file count is derived from client host memory, none of the CLOSED-tunable parameters reduce it, and that count lies past the livelock boundary for any fleet with two or more 256 GB-class client nodes.
If confirmed, could this be triaged as a CLOSED blocker for RetinaNet — either (a) fixing the per-step O(N) behavior for large
num_files_train, or (b) decoupling the CLOSED minimum-dataset rule from this pathology until it is fixed?