[HAL/Remote] Add remote HAL device execution

[benvanik]

Remote HAL turns an IREE HAL device into a distributed object: applications use the normal HAL resource and queue model in a client process, while iree-serve-device owns the native device and executes work in a server process. This separates applications, models, weights, and executable selection from the machine that physically hosts the accelerator without introducing a second execution API.

The primary deployment is a trusted private network or compute fabric. A server listens on the network by default so another machine can connect without hidden bind configuration, and startup prints the actual listener plus a directly usable client URI. TCP is plaintext and the protocol deliberately provides no peer authentication, authorization, confidentiality, or active integrity; deployments that require those properties establish them outside IREE with fabric isolation, IPsec, WireGuard, or an equivalent network boundary. SHM relies on operating-system endpoint permissions, RDMA relies on fabric admission, and exposing any transport directly to an untrusted network is outside the deployment contract.

The implementation is a transport-independent networking substrate plus a remote HAL client and server:

client HAL API
  |
  v
remote HAL client
  |
  |  iree/net session bootstrap and immutable device metadata
  |
  +-- control channel: typed resource and lifecycle RPCs
  +-- queue channel: frontier-ordered HAL queue operations
  +-- bulk channel: bounded large payload, file, map, and profile transfer
  |
  v
remote HAL server
  |
  v
native HAL device (AMDGPU, Vulkan, or another server-side driver)

Remote HAL maps HAL ownership and synchronization onto these channels instead of emulating every C function with a synchronous RPC. Buffers, executables, command buffers, semaphores, events, files, and virtual-memory reservations have explicit remote identities and release rules. Queue operations retain normal HAL semaphore dependencies, server failures advance or fail the affected signal frontiers, and disconnect converges outstanding work on one terminal session state.

The detailed code and debugging map lives in runtime/src/iree/hal/remote/README.md.

Native GPU proof

iree-remote-check is the smallest complete proof of the intended product shape. The client embeds AMDGPU HSACO and Vulkan SPIR-V artifacts, connects to the server, selects a target advertised by the native device, uploads only the selected executable, performs host-to-device and device-to-host transfers, dispatches real GPU work, waits on a HAL semaphore, and verifies the exact result. The client links the remote HAL and driver-neutral target-selection utilities but no AMDGPU, HIP, or Vulkan HAL driver; the server is the only process that loads a native driver.

Run the AMDGPU server and client on machines in the same trusted network:

# Server: expose AMDGPU device 0 on all IPv4 interfaces.
iree-bazel-run \
  --//runtime/config/hal:drivers=amdgpu \
  //tools:iree-serve-device -- \
  --device=amdgpu://0

# Client: select, upload, dispatch, and verify a matching embedded HSACO.
iree-bazel-run \
  --//runtime/config/hal:drivers=remote \
  --//runtime/config/hal:executable_artifacts=amdgpu \
  //tools:iree-remote-check -- \
  --device=remote-tcp://server-host:5000

Server startup prints the bound listener and a client device flag. For the wildcard default, replace <server-host> with a hostname or interface address reachable from the client; 0.0.0.0 is a bind address, not a client destination. A same-host client can use 127.0.0.1:5000 against the same default listener.

The same client can target a Vulkan server:

# Server: expose the default Vulkan device.
iree-bazel-run \
  --//runtime/config/hal:drivers=vulkan \
  //tools:iree-serve-device -- \
  --device=vulkan://

# Client: select and dispatch the embedded SPIR-V module remotely.
iree-bazel-run \
  --//runtime/config/hal:drivers=remote \
  --//runtime/config/hal:executable_artifacts=vulkan \
  //tools:iree-remote-check -- \
  --device=remote-tcp://server-host:5000

The smoke client is independent of IREE compiler and VM artifacts. Its AMDGPU artifact set follows the configured target selectors, and its Vulkan artifact is assembled independently as Vulkan 1.3 BDA SPIR-V. This keeps the proof focused on native HAL execution, executable compatibility, transfer, dispatch, synchronization, and process-boundary dependency isolation.

Architecture and review map

1. Exact asynchronous lifecycle contracts

The networking stack is built on production async primitives for socket operations, event sources, receive pools, proactor registration, cancellation, and notification-driven shutdown. io_uring, IOCP, POSIX, and fallback paths obey the same ownership contract: every accepted operation has exactly one terminal completion, callback storage remains alive until the backend proves release, and shutdown joins real completion state rather than sleeping for a guessed drain interval.

The highest-pressure invariant is teardown. Listener cancellation, event-source unregistration, multishot receive emulation, nested callback submission, io_uring registration, and proactor destruction all have explicit terminal states. Polling distinguishes internal progress and explicit wakeups from user callback count so completion-driven loops do not mistake a productive poll turn for failure.

2. Message carriers and portable TCP

iree_net_carrier_t defines transport-independent framed message movement, send reservations, direct operations, capabilities, and terminal failure behavior. The begin/commit/abort reservation model allows storage to be prepared without copying and permits out-of-order producer commit while preserving completion ownership. Backpressure is represented explicitly instead of becoming an unbounded allocation path.

Loopback provides an in-process semantic reference and TCP provides the default portable cross-machine path. Peer departure is terminal and observable, valid frames may wait for local stream activation during endpoint setup, send storage survives until completion, and listener shutdown cancels pending accepts. Carrier CTS applies the same lifecycle, reliability, pressure, reservation, direct-operation, and zero-copy contracts across implementations.

3. Sessions and semantic channels

iree_net_session_t turns one carrier connection into a negotiated set of endpoints. Bootstrap exchanges protocol versions, topology, frontier axes, transport capabilities, and immutable application payloads before endpoints become visible. The parser bounds every variable-length field and is fuzzed independently; incompatible protocol revisions fail instead of entering a compatibility mode.

Control channels carry typed request, response, and notification traffic. Queue channels carry commands and advances ordered by explicit frontier values rather than carrier arrival or FIFO assumptions. Bulk channels carry transfer-ID-keyed START, DATA, COMPLETE, ABORT, and CREDIT frames. Endpoint detach, GOAWAY, and carrier failure converge on one session failure so a semantic channel cannot remain half-alive after its transport has become terminal.

4. Remote HAL resources and queue execution

The remote client exposes an ordinary iree_hal_device_t; the server retains one or more ordinary local devices. Session-scoped resource IDs encode type, generation, server proactor placement, and table slot. Client-created resources may begin with provisional IDs so queue work can reference them before a control response returns; the server parks dependent work until the provisional ID resolves and reports the canonical mapping on the corresponding response or queue advance.

The protocol covers allocation, mapping, semaphores, events, executable upload, reusable and one-shot command buffers, queue allocation and deallocation, fill, update, copy, dispatch, file operations, profiling, extension operations, and batched release. Release messages carry the greatest submission epoch that could still reference a resource, preventing control traffic from freeing an object ahead of previously submitted queue work. Queue failures are serialized into ADVANCE messages and surface through the client HAL semaphore path used by local devices.

5. Device specifications and executable selection

The server serializes immutable device specifications, memory heaps, and executable target tables into the session bootstrap payload. The remote device is not published until that catalog has been validated and the queue and bulk endpoints are ready. Clients can therefore cache device facts, choose a compatible executable locally, and upload only the selected artifact without paying one synchronous network round trip per query.

Executable loads carry the stable target-table ordinal, queue affinity, load flags, specialization constants, and backend-native bytes. AMD target parsing and family compatibility live in driver-neutral HAL executable support so remote clients, HRX, HIP, and AMDGPU share vocabulary without linking each other's driver implementations. Wire sizes use explicit fixed-width representations, protocol peers must be little-endian, and mismatched revisions are rejected.

6. Bounded bulk transfer, files, and profiling

Large payloads move independently of latency-sensitive control and queue traffic. Transfer tables, chunk descriptors, peer credits, receive windows, staging pools, and owned upload/download components keep memory bounded under pressure. DATA chunks may arrive out of order, but each transfer reservation covers a precise byte range and completion, cancellation, or failure retires every descriptor exactly once.

Remote file operations support both client-local files bridged over bulk transfer and server-local files exposed through an explicit logical allow-list. Clients never submit arbitrary server paths; final path and access validation happen against the server-owned namespace. Profiling relays server sink callbacks into a client-owned sink while retaining session and callback state through terminal completion, so profile backlog cannot outlive its connection.

7. Virtual-memory operations

Remote allocator support carries virtual address reservation, physical allocation, map, unmap, and access changes across the protocol. Virtual and physical resources have distinct message namespaces, queue affinity and access-agent selection are preserved, and the server returns opaque HAL addresses rather than exposing a native driver object to the client.

AMDGPU implements the corresponding allocator operations and advertises only capabilities it can execute. Partial failures unwind mappings, reservations, and physical resources through the same ownership graph. This is the generic pointer-stability mechanism; higher-level sparse pool policy and HIP pointer semantics are intentionally not encoded into the remote HAL allocator contract.

8. Cross-process shared memory

The SHM carrier provides same-host IPC with shared rings, adaptive progress, a cross-process wake primitive, direct registered-buffer reads and writes, and endpoint bootstrap over Unix domain sockets or Windows named pipes. Unix descriptor and Windows HANDLE transfer support remote file sidebands without copying file contents through control messages.

Operating-system endpoint permissions define admission. Shared mappings and transferred handles have one owner, wakeups cannot be lost across the arm/sleep boundary, peer death wakes stranded waiters, and in-process carrier tests are supplemented by real cross-process multi-endpoint coverage on both POSIX and Windows implementations.

9. RDMA transport and remote memory windows

The Linux RDMA carrier uses librdmacm and libibverbs through dynamic loaders, so an RDMA-enabled binary can still start on a system without those libraries. The implementation owns CM event handling, device and protection-domain context, completion queues, queue pairs, receive credits, send windows, work-request retirement, registered staging, direct read/write, endpoint negotiation, and exported memory windows. It supports RoCE and InfiniBand deployment shapes and includes DMA-BUF registration probing for GPU-direct paths.

CM and CQ failures remain terminal, every work request retires once, queue depth and explicit credits bound outstanding traffic, and registration lifetime spans every DMA that references it. Requesting RDMA is a hard capability requirement and never silently downgrades to ordinary TCP bulk semantics. RDMA is Linux-only and opt-in in both build systems; disabled //... builds do not fetch its headers or pull its libraries.

10. Deployment and dependency boundaries

iree-serve-device defaults to tcp://0.0.0.0:5000 because the intended use is another trusted machine, not an accidental loopback-only demo. Readiness is printed only after the listener starts, wildcard addresses become a usable client template, explicit loopback binds are classified as same-host, and SHM reports its operating-system permission boundary. Network-visible startup reports the surrounding trust boundary and the absence of protocol authentication or encryption; any peer admitted through that boundary can submit arbitrary HAL work to the exposed device.

Build configuration lives under //runtime/config/hal and //runtime/config/net, with TCP enabled by default and SHM/RDMA explicit transport choices. Bazel is the source of build graph truth and generated CMake metadata carries equivalent conditions. Remote-only tools link the remote HAL client and neutral executable-target support but no native HAL driver; HIP never links AMDGPU, AMDGPU never links HIP, and native drivers remain on the server side of the process boundary.

Test architecture

Carrier CTS is reusable across loopback, TCP, SHM, and RDMA and exercises reservation ordering, backpressure, reliability, lifecycle, direct operations, and terminal errors. Remote HAL CTS wraps ordinary HAL CTS backends behind an in-process server/client pair so the same queue and resource contracts are exercised through serialization. Dedicated integration coverage exercises real TCP sessions, cross-process SHM, server slot reuse, file transfer, bounded bulk pressure, profiling relay, virtual memory, disconnect, bootstrap parsing, malformed frames, and native AMDGPU/Vulkan executable selection. Bootstrap, framing, control-channel, and carrier parsers have fuzz targets at their untrusted length boundaries.

HRX scope boundary

This branch includes the narrow HRX device-selection and lifecycle bridge used to prove that an HRX process can create a remote HAL device, derive its AMDGPU architecture from the bootstrapped device spec, and run ordinary stream, buffer, and transfer paths without loading a native GPU driver in the client. That probe exposed generic session startup ordering, allocator capability, and receive-pool lifetime bugs that are fixed at their owning layers here.

It does not claim production HIP-on-HRX remoting. That consumer requires a stronger native device-pointer identity contract for nested pointers, sparse virtual-memory-backed pools that do not commit most of device memory on the first small allocation, post-connect failure propagation through HIP synchronization, and a remote-only HRX dependency graph without local-task or VM implementations. Those concerns form a separate consumer review surface and are not hidden behind compatibility fallbacks in this PR.

Reviewer notes

The highest-value review questions are:

• Does every accepted asynchronous operation and relay reach exactly one terminal completion before caller-owned storage can be released?
• Does every cross-machine ordering edge pass through an explicit frontier, with exactly one advance or failure path?
• Does every remote resource have a session-local owner, a stable provisional-to-canonical resolution path, and a release condition tied to observed queue progress?
• Is every wire length, count, offset, and padded command size validated before allocation, pointer arithmetic, or copy?
• Can every large payload move without blocking control/queue traffic or allowing credits, descriptors, staging memory, or callbacks to grow without bound?
• Do SHM descriptors/HANDLEs and RDMA registrations/work requests remain owned through the final kernel or peer-visible completion?
• Does a remote-only client stay free of native HAL driver code across both Bazel and CMake dependency closures?
• Is the trusted-fabric boundary visible at build, startup, documentation, and failure diagnostics without implying that the protocol itself authenticates or protects peers?

The branch history is organized by these ownership and invariant boundaries so the individual mechanisms remain useful archaeology, but the review unit is the integrated system: lifecycle, transport, protocol, HAL semantics, native execution, and teardown must agree at the branch head.