diff --git a/docs/_static/figs/shuffle_epoch_loop.svg b/docs/_static/figs/shuffle_epoch_loop.svg new file mode 100644 index 00000000..cb7a199e --- /dev/null +++ b/docs/_static/figs/shuffle_epoch_loop.svg @@ -0,0 +1,80 @@ + +Shuffle test epoch loop +Epoch loop: data preparation, std::shuffle on m_fidx with shared RNG, each worker reads its window, DYAD transparently fetches remote files, process file data omitted, epoch complete, repeat. + + + + + + + + + + + + Data preparation + partition, generation, staging + + + + + + + worker.shuffle() + std::shuffle(m_fidx, m_gen) + +same seed → +identical order +across workers + + + + + + read_files(worker) + m_fidx[m_begin .. m_end] + + + + + + + open(work_dir/filename) + plain POSIX — app unaware + + + +file is local +read locally + + +file is remote +DYAD fetches transparently + + + + + + + + + + Process file data + omitted — I/O benchmark only + + + + + + + Epoch complete + Wait for all workers + + + + +next +epoch + +m_gen advances each epoch — different permutation per epoch, all workers identical + \ No newline at end of file diff --git a/docs/_static/figs/shuffle_sample_files.svg b/docs/_static/figs/shuffle_sample_files.svg new file mode 100644 index 00000000..b9a1123d --- /dev/null +++ b/docs/_static/figs/shuffle_sample_files.svg @@ -0,0 +1,133 @@ + +Shuffle data structures with storage cylinders +Shared storage cylinder at top feeds into per-rank local storage cylinders, which map to m_flist partitions, then m_fidx shuffled index array below. + + + + + + + + + + + +Shared storage + + + +cat.dat +dog.dat, elk.dat + + + +fox.dat +gnu.dat, hen.dat + + + +ibis.dat +jay.dat + + + + + + +Local storage 0 + + + + + + +Local storage 1 + + + + + + +Local storage 2 + + +m_flist +filenames (never reordered) + + + + cat.dat + + dog.dat + + elk.dat + + fox.dat + + gnu.dat + + hen.dat + + ibis.dat + + jay.dat + + + + + + + +[0] +[1] +[2] +[3] +[4] +[5] +[6] +[7] + + +m_fidx +shuffled index array (identical on all workers after std::shuffle) + + + + 3 + + 7 + + 1 + + 5 + + 0 + + 6 + + 2 + + 4 + + + + + + + +rank 0: [0,3) +rank 1: [3,6) +rank 2: [6,8) + + +e.g. rank 1, index 5: + +m_fidx[5] = 6 + +m_flist[6] = "ibis.dat" +(owned by rank 2 → DYAD fetches) + + + + + \ No newline at end of file diff --git a/docs/design.rst b/docs/design.rst index a59009de..6a62ebf1 100644 --- a/docs/design.rst +++ b/docs/design.rst @@ -30,6 +30,9 @@ A producer and consumer scenario sharing a file Deep Learning Training by Distributed Stochastic Gradient Descent (SGD) ======================================================================= +.. figure:: _static/figs/shuffle_sample_files.svg + :alt: shuffling sample files + Deep learning training often requires randomizing the order of input samples at each epoch. In distributed or parallel training, where each worker processes a subset of the samples, the set of files assigned to each worker changes at every @@ -41,7 +44,7 @@ workers when needed. Initially, the dataset is partitioned across workers. When a file is reference for the first time, it is staged into the DYAD managed directory on the local storage of the worker who owns the partition. -Check out our use case with the `DLIO `_ +Check out our use case with the `DLIO `_ representing PyTorch dataloader :ref:`Devarajan et al. 2024 `. diff --git a/tests/shuffle/README.md b/tests/shuffle/README.md index a9dfdda8..422ea152 100644 --- a/tests/shuffle/README.md +++ b/tests/shuffle/README.md @@ -34,7 +34,7 @@ otherwise be fetched from another worker’s local storage or from shared storag It can reside on either local or shared storage, controlled by `--is-local`. When local, each worker has its own work-dir and owns an exclusive partition of the file list — files are either generated or staged into it before the -epoch loop begins. When shared, all workers reference the same work_dir — +epoch loop begins. When shared, all workers reference the same `work_dir` — files are either generated into it or assumed to already exist there. ## Options @@ -110,10 +110,50 @@ file_N-1.dat | Suffix | Multiplier | Example | |---|---|---| -| none | 1 | `--size 4096` → 4096 bytes | -| `K` | 1024 | `--size 4K` → 4096 bytes | -| `M` | 1024² | `--size 2M` → 2097152 bytes | -| `G` | 1024³ | `--size 1G` → 1073741824 bytes | +| none | 1 | `--size 4096` ->4096 bytes | +| `K` | 1024 | `--size 4K` -> 4096 bytes | +| `M` | 1024² | `--size 2M` -> 2097152 bytes | +| `G` | 1024³ | `--size 1G` -> 1073741824 bytes | + + +## Implementation + +![Shuffle data structures](../../docs/_static/figs/shuffle_epoch_loop.svg) + +**Epoch loop in the shuffle test** +Epoch loop in the shuffle test. Before the loop begins, each worker prepares +its partition of files through generation or staging into local storage. Each +epoch starts with `worker.shuffle()`, which runs `std::shuffle` on `m_fidx` using a +shared RNG seed — producing an identical file order across all workers with no +MPI communication. Each worker then iterates over its assigned window of `m_fidx` +and calls `open()` on each mapped filename via a plain POSIX call, unaware of +where the file physically resides. If the file is in the worker's own local +storage it is read directly; if it belongs to another worker, DYAD intercepts +the `open()` call and fetches the file transparently via the KVS and RDMA +transfer. In a real training scenario, each file would then be processed (e.g. +decoded and fed to the model), but the shuffle test omits this step as it +benchmarks I/O behavior only. The epoch ends with an `MPI_Barrier` that +synchronizes all workers before the next shuffle begins. + +![Shuffle data structures](../../docs/_static/figs/shuffle_sample_files.svg) + +**Data structures used by the shuffle test.** +`m_flist` stores the filenames in a fixed order shared by all workers. +Each worker owns a contiguous partition of files in local storage — this +ownership is static and does not change across epochs. +`m_fidx` is an index array shuffled each epoch using std::shuffle with the +same RNG seed on every worker, as described above. +Each worker reads a contiguous window of `m_fidx` determined by `split()` +— rank 0 gets indices [0,3), rank 1 [3,6), and rank 2 [6,8). +While the window into `m_fidx` is fixed, the shuffled indices point to +different files every epoch, so each worker processes a different subset +of files each time. +To read a file, a worker dereferences its assigned indices in `m_fidx` to +obtain positions in `m_flist`, then opens the corresponding file. If the +file resides in another worker's local storage (as shown: rank 1 reading +"ibis.dat" owned by rank 2), DYAD intercepts the `open()` call and fetches +it transparently via the KVS and RDMA transfer. + ## Examples