ibsend is a file transfer tool for Linux hosts on an InfiniBand or RoCE network. It sends files and directory trees directly between machines using RDMA, with peer discovery, resumable transfers and an optional desktop interface.
The receiver buffers incoming data in memory while writing it to disk. This allows transfers to make use of available network bandwidth when storage writes temporarily fall behind. Recorded QDR tests reached 3.47 GB/s; see Performance for the hardware and measurement scope.
ibsend is intended for trusted, private RDMA networks. It does not provide encryption or peer authentication. CLI and GUI messages are currently in Chinese; documentation is available in English, Simplified Chinese and Japanese.
Installation · Quick start · Commands · Performance · Limitations
- Direct RDMA transfers. File data and control messages use the same RDMA connection, without a separate TCP channel.
- Adaptive memory buffering. Pool sizing adjusts to available memory and memory-locking limits; disk writes run asynchronously.
- Resume and verify. Continue interrupted transfers from
.partfiles and check transferred bytes with CRC32C. - Files, folders and named peers. Send multiple paths in one command, discover receivers on the IPoIB subnet, or use the optional drag-and-drop GUI.
Both machines need Linux, an RDMA-capable adapter and working RDMA connectivity.
For InfiniBand, configure IPoIB addresses before starting ibsend. Building from
source requires a recent stable Rust toolchain, a C compiler and the development
headers for libibverbs and librdmacm from rdma-core.
On Debian or Ubuntu, install the build dependencies:
sudo apt-get install build-essential libibverbs-dev librdmacm-devBuild and install the CLI on both machines:
git clone https://github.com/HiroGitea/ibsend.git
cd ibsend
cargo install --path . --lockedMake sure $HOME/.cargo/bin is on PATH.
For the optional GUI, install the desktop dependencies and enable the gui
feature. On Debian or Ubuntu:
sudo apt-get install libwayland-dev libxkbcommon-dev
cargo install --path . --locked --features guiThis installs both ibsend and ibsend-gui. The default CLI build does not
require the GUI dependencies.
RDMA requires registered memory to remain resident in RAM. If ibsend reports that the memory-locking limit restricts its buffer pool, run:
ibsend authorizeThe command requests CAP_IPC_LOCK for the installed binary through polkit or
sudo, or prints a command for manual setup. It makes no change when the
available allowance is sufficient. Restart running ibsend processes after
authorization; the GUI restarts automatically when authorized from its interface.
The permission applies to anyone running that binary. Rebuilding or reinstalling can remove it, so authorization may be needed again after an update.
On the receiving machine, start a receiver named nas:
ibsend daemon --name nas --out ./receivedLeave it running. Incoming files are saved under ./received, relative to the
directory where the daemon was started.
On the sending machine, find the receiver and send a file or directory:
ibsend discover
ibsend send nas ./big.iso
ibsend send nas ./photosThese commands create received/big.iso and received/photos/ on the receiving
machine, preserving the directory's relative paths. Multiple paths can be sent
together with ibsend send nas ./big.iso ./photos.
To connect by address, replace nas with the receiver's RDMA IPv4 address:
ibsend send 10.0.0.1 ./big.isoAutomatic interface selection and discovery use IPoIB. For RoCE, bind the
receiver explicitly with --bind <RDMA-IPv4-address> and send to that address
directly. The nas name above is an ibsend discovery name.
For desktop use, launch ibsend-gui, select a peer and add files by dragging them
into the window. Press Ctrl+S to send.
| Command | Description |
|---|---|
ibsend daemon |
Keep a receiver running and available for discovery. |
ibsend discover |
List receivers on local IPoIB subnets. |
ibsend send <peer> <paths…> |
Send files or directories to a peer name or IPv4 address. |
ibsend recv |
Receive one transfer, then exit. |
ibsend authorize |
Set up memory-locking permission when needed. |
ibsend-gui [files…] |
Open the optional desktop interface. |
Options for daemon and recv:
| Option | Default | Description |
|---|---|---|
--bind <IP> |
First IPoIB address | Local RDMA address to listen on. |
--out <DIR> |
Current directory | Destination for received files. |
--name <NAME> |
Hostname | Name advertised during discovery. |
--pool <SIZE> |
Automatic | Size of the receiver's memory buffer pool. |
--slab <SIZE> |
Automatic | Transfer block size. |
Sizes accept K, M and G suffixes, using powers of 1024. For example,
--pool 2G requests a 2 GiB pool. The sender's --slabs <N> option controls
pipeline depth and defaults to 16. discover --timeout <MS> sets the per-address
resolution timeout and defaults to 300 ms. Run ibsend without arguments for
usage information.
To resume an interrupted transfer, run the same send command again with the same source files and destination. The receiver checks each destination path:
| Destination state | Behavior |
|---|---|
| A completed file with the expected size exists | Skip the file. |
A partial .part file exists |
Continue from its current byte offset. |
| Neither exists | Transfer the file from the beginning. |
Keep .part files to preserve resume progress. Completed files are skipped based
on size only; matching contents are not checked before skipping.
CRC32C checks compare the bytes read by the sender with those handled by the receiver's writer thread. For resumed files, only bytes sent in the current session are checked. The existing prefix is not revalidated, and destination files are not read back from disk for verification. Check the receiver's output for checksum results.
The following results were recorded on two hosts with dual-port 40 Gb QDR adapters and PCIe 2.0 x8 connections. IPoIB used connected mode with an MTU of 65520; the sender ran a rolling-release Linux distribution and the receiver ran Debian 12. These results describe that setup; throughput varies with hardware and workload.
For transfers smaller than the receiver's buffer pool:
| Sender pipeline | Transfer rate |
|---|---|
| 6 × 1 MB blocks | 3.47 GB/s |
| 3 × 2 MB blocks | 3.47 GB/s |
| 1 × 4 MB block | 1.94 GB/s |
In a separate transfer of approximately 2 GB to a ZFS target with a 1.5 GB buffer pool:
| Measurement | Elapsed time | Rate |
|---|---|---|
| Sender data transfer | 0.62 s | 3.48 GB/s |
| Receiver, including file writes | 2.58 s | 831 MB/s |
Sender time and receiver completion time measure different work. Memory buffering lets the sender finish while the receiver continues writing. Once the pool fills, the sender waits for space and sustained throughput is limited by the receiver's write rate. Actual performance also depends on source reads, adapter and PCIe bandwidth, and available memory.
ibsend uses a single reliable RDMA connection for file metadata, transfer control and payload data. The adapter writes incoming payloads directly into a registered memory pool. A separate writer thread consumes that data and releases buffer space for subsequent transfers. The daemon reuses the pool across sessions.
See the design documentation for the protocol, buffering, threading model and Rust library example.
- Trusted networks only. Transfers are neither encrypted nor authenticated. Restrict access at the RDMA fabric level.
- File contents and relative paths. Symlinks and special files are skipped; empty directories, file permissions, ownership and timestamps are not preserved.
- Size-based skipping. An existing file with the expected size is treated as complete, even if its contents differ. Resumed prefixes are not verified.
- IPoIB discovery. Automatic discovery scans IPv4 IPoIB subnets with at most 4096 addresses. Use a peer's IPv4 address when discovery is unavailable.
Bug reports, documentation improvements and pull requests are welcome. See CONTRIBUTING.md for the development setup, checks and the hardware details to include in performance reports.
Licensed under MIT or Apache-2.0, at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.