TransferBench v1.68.00

v1.68.00

Fixed

• Improper draining of writes that could artificially inflate transfer timing (bug introduced in v1.67.00)
• Potential timing bug when running GFX Executor in warp-subexecutor mode with small data sizes
• Keeping subiteration threads in sync per subiteration

TransferBench v1.67.00

v1.67.00

Highlights

This release of TransferBench enables scale-up support, and introduces a number of new presets.
To get more information about available presets, run TransferBench presets. The helper script LaunchTransferBench.sh simplifies using TransferBench across multiple nodes without mpi.

Added

• Added NIC_TRAFFIC_CLASS to set the DSCP/traffic class byte in the RoCE GRH for QPs (RoCE only)
• Added NIC_SERVICE_LEVEL to set the IB service level (sl) for QPs (IB and RoCE)
• Initial support for pod communication. Requires compatible hardware / ROCm version and subject to further testing
  • This potentially enables GFX/DMA executors to access SRC/DST memory locations on GPUs within the same pod
  • Pod membership requires amd-smi however can be skipped by setting TB_FORCE_SINGLE_POD=1
• Support for dumping executed Transfers to a config file specified by TB_DUMP_CFG_FILE
  • This will write Transfers that are executed (for example via a preset) to a config file that can then be executed
• Reporting number of iterations run when running in timed mode (NUM_ITERATIONS < 0)
• Adding NIC_CQ_POLL_BATCH to control CQ poll batch size for NIC transfers
• New "hbm" preset which sweeps and tests local HBM read performance
• Added a new TB_WALLCLOCK_RATE that will override GPU GFX wallclock rate if it returns 0 (debug)
• Adding new batched-DMA executor "B", which utilizes the hipMemcpyBatchAsync API introduced in HIP 7.1 / CUDA 12.8
• Added new "bmasweep" preset that compares DMA to batched DMA execution for parallel transfers to other GPUs
• Added new "wallclock" preset that compares wallclock counters across XCCs within a GPU
• Added new "smoketest" preset that runs a variety of DMA/GFX tests for simple correctness tests
• Added new "help" preset to show config file examples
• Added new "presets" preset to show available presets and their descriptions
• Added new "rings" preset that runs parallel rings of transfers (pod-capable)
• Added new "envvars" preset to show environment variables that can change TransferBench behavior
• Adding information on how to run multi-rank with TransferBench, when run with no args
• Added new "nica2a" preset (NIC all-to-all over GPUs via NIC executors, multi-node)
• Added new GFX_KERNEL to allow experimenting with copy-only GFX kernel. Currently this is opt-in only
• Added SHOW_PERCENTILES (e.g. 50,75,90,95,99) to show empirical percentiles of per-iteration duration
• Adding new LaunchTransferBench.sh script to simplify launching TransferBench across multiple nodes (via socket)
• New empty preset (EmptyKernel) to measure empty-kernel launch latency with BATCHSIZES/GRIDSIZES/BLOCKSIZES sweeps

Modified

• DMA-BUF support enablement in CMake changed to ENABLE_DMA_BUF to be more similar to other compile-time options
• Adding extra information to CMake and make build methods to indicate enabled / disabled features
• a2asweep preset changes from USE_FINE_GRAIN to MEM_TYPE to reflect various memory types
• a2asweep preset changes from NUM_CUS to NUM_SUB_EXECS to match with a2a preset naming convention
• scaling preset changes from using USE_FINE_GRAIN to CPU_MEM_TYPE and GPU_MEM_TYPE
• NIC_FILTER renamed to TB_NIC_FILTER for consistency
• DUMP_LINES renamed to TB_DUMP_LINES for consistency
• Dynamically size CQs for NIC transfers in high QPs case
• Switch to using hipMemcpyDeviceToDeviceNoCU instead of hipMemcpyDefault for DMA Executor if available (requires HIP >= 6.0)
• Allow for multiple destination memory locations for DMA/Batched-DMA Transfers
• Removed env vars printing and preset print when running TransferBench with no args
• Modification to simplify socket comm usage - first rank only needs to set TB_NUM_RANKS=X to see connection info

rocm-7.2.4

ROCm release v7.2.4

rocm-7.2.3

ROCm release v7.2.3

rocm-7.2.2

ROCm release v7.2.2

rocm-7.2.1

ROCm release v7.2.1

TransferBench v1.66.02

v1.66.02

Added

• Adding DMA-BUF support
  • TransferBench now supports GPU memory registration via DMA-BUF handle
    • Disabled by default in build time
    • To enable, DISABLE_DMABUF must be explicitly turned off for both CMake and Make process
  • DMA-BUF registration relies on both IB verbs registration and HSA dmabuf export functionality
    • Support for both are checked in compile time as well as runtime
• New preset: nicp2p
  • A point to point performance sweep for all paris of NIC on all nodes
  • The preset requires all participating nodes have identical topology
  • All NICs on every node will bind to closest device of user specified memtype, and communicate with all
    other NICs in the potentially multi-node world
  • Execution of node pairs, as well as NIC pairs under the same node pair, are parallelized by default
• Adding GPU affinity check for NIC devices

Fixed

• Fixing some typo in result table and NIC verbose output

rocm-7.2.0

ROCm release v7.2.0

TransferBench v1.66.01

v1.66.01

Fixed

• Adding support for TheRock
• Fixing parsing issue when using NULL memory type
• Fixing CUAD compilation flags when enabling NIC/MPI

Modified

• TransferBenchCuda must now be explicitly built with via 'make TransferBenchCuda'

TransferBench v1.66.00

v1.66.00

• Adding multi-node support, via sockets or MPI. (MPI support requires compiling against an MPI implementation like OpenMPI)
  • See CHANGELOG for more details on multi-node usage, including new rank notation
• Wildcard support
• CSV friendly tabular output
• Additional memory types
• "dryrun" preset
• Adding nicrings preset - This runs parallel transfers where NIC form rings connecting identical numbered NICS across ranks
• p2p and a2a presets have deprecated the use of USE_FINE_GRAIN in exchange for CPU_MEM_TYPE and GPU_MEM_TYPE to allow for more expansive testing