+
+
+### 🚧 v1.1 — NIXLBench and KVBench Documentation
+
+**Milestone Goal:** Add NIXLBench and KVBench documentation pages under the Developer Guide section of the NIXL Fern docs site.
+
+- [x] **Phase 32: Navigation and Directory Setup** - Add Benchmarks subsections to `docs/index.yml` and create directory scaffolding before any content is authored (completed 2026-04-07)
+- [ ] **Phase 33: NIXLBench Overview and Build** - Author NIXLBench overview page and build page (Docker + native) as standalone Fern-compatible MDX
+- [x] **Phase 34: NIXLBench Usage, Troubleshooting, and Cross-References** - Author NIXLBench usage guide and troubleshooting page; apply ETCD callouts and backend cross-links throughout (completed 2026-04-07)
+- [x] **Phase 35: KVBench Overview and Build** - Author KVBench overview page (NIXLBench subprocess dependency stated in paragraph one) and build page (Docker + Python venv) (completed 2026-04-07)
+- [x] **Phase 36: KVBench Commands, Model Config, and LLM Examples** - Author KVBench command reference (all 5 subcommands), model configuration guide, and LLM architecture examples (completed 2026-04-07)
+- [x] **Phase 37: Terminology Normalization and Quality Audit** - Batch grep pass for terminology drift, validate all CLI flag tables against `--help` output, confirm `fern check` passes, audit cross-links (completed 2026-04-07)
+
+---
+
+## Phase Details
+
+### Phase 32: Navigation and Directory Setup
+**Goal**: The Fern navigation tree declares both NIXLBench and KVBench sections, and the content directories exist, so incremental builds work from the first content commit
+**Depends on**: Nothing (first phase of v1.1)
+**Requirements**: NAV-01, NAV-02, NAV-03
+**Success Criteria** (what must be TRUE):
+ 1. `docs/index.yml` contains two `section:` blocks (NIXLBench and KVBench) under Developer Guide, after "Building a Backend Plugin", each with the correct `path:` and `contents:` entries
+ 2. Directories `docs/development/benchmarks/nixlbench/` and `docs/development/benchmarks/kvbench/` exist with stub `.md` files for every navigation entry (so `fern check` resolves all declared paths)
+ 3. `fern check` passes with only stub content in place
+**Plans**: 1 plan
+Plans:
+- [x] 34-01-PLAN.md — Author usage.md with etcd coordination, communication patterns, storage examples, CLI tables, and troubleshooting
+
+### Phase 33: NIXLBench Overview and Build
+**Goal**: Developers can read what NIXLBench is and how to build it from a clean environment, with no duplication of steps already in the existing NIXL docs
+**Depends on**: Phase 32
+**Requirements**: NB-01, NB-02
+**Success Criteria** (what must be TRUE):
+ 1. The NIXLBench overview page (`index.md`) describes what NIXLBench is, its key features (backends, communication patterns, memory types, ETCD coordination), and system requirements
+ 2. The build page (`build.md`) presents Docker build and native build side-by-side in a `✅ v1.0 — Initial Documentation (Phases 21–31) — SHIPPED 2026-04-07
+ +Phases 21–31 delivered the full NIXL documentation site on Fern: Getting Started, User Guide (13 backends, ETCD, telemetry), Developer Guide (all build paths, backend plugin), Examples (6 examples), API Reference (C++, Python, Rust, Device, Plugin APIs), Resources (environment variables, troubleshooting), and full NVIDIA branding with dark/light theming, terminology normalization, and cross-page coherency. + +` collapsibles, no bare anchor links
+- **Code blocks:** Triple backtick with language identifier
+
+### Q6: What cross-links are needed?
+
+**Backend links (first mention per page) — paths from `docs/index.yml`:**
+- UCX: `/docs/user-guide/backends/ucx`
+- Libfabric: `/docs/user-guide/backends/libfabric`
+- Mooncake: `/docs/user-guide/backends/mooncake`
+- DOCA GPUNetIO: `/docs/user-guide/backends/gpunetio`
+- GPUDirect Storage (GDS): `/docs/user-guide/backends/gds`
+- GPUDirect Storage MT: `/docs/user-guide/backends/gds-mt`
+- POSIX: `/docs/user-guide/backends/posix`
+- HF3FS: `/docs/user-guide/backends/hf3fs`
+- OBJ: `/docs/user-guide/backends/obj`
+- Azure Blob: `/docs/user-guide/backends/azure-blob`
+- GUSLI: `/docs/user-guide/backends/gusli`
+
+**Other cross-links:**
+- Building NIXL from Source: `/docs/user-guide/building-nixl`
+- Metadata Exchange with ETCD: `/docs/user-guide/etcd-metadata-exchange`
+
+### Q7: What nav changes are needed?
+
+Current `docs/index.yml` under NIXLBench:
+```yaml
+- section: NIXLBench
+ collapsed: open-by-default
+ path: development/benchmarking/nixlbench/index.md
+ contents:
+ - page: Usage and Troubleshooting
+ path: development/benchmarking/nixlbench/usage.md
+```
+
+Need to add `build.md` entry per D-01:
+```yaml
+- section: NIXLBench
+ collapsed: open-by-default
+ path: development/benchmarking/nixlbench/index.md
+ contents:
+ - page: Building NIXLBench
+ path: development/benchmarking/nixlbench/build.md
+ - page: Usage and Troubleshooting
+ path: development/benchmarking/nixlbench/usage.md
+```
+
+### Q8: What existing page patterns should be replicated?
+
+The `docs/user-guide/building-nixl/docker.md` page is very concise (30 lines): frontmatter, one paragraph, code block, a `` and ``. The section index page (`building-nixl/index.md`) is also concise: frontmatter, bullet list of sub-pages, then a build options table.
+
+The overview page (`index.md`) should follow the section-index pattern but with more narrative content (problem-first per D-02) since it's describing a tool, not just listing sub-pages.
+
+The build page (`build.md`) should follow Docker page conciseness but use `` for Docker vs Native side-by-side per D-03.
+
+## Validation Architecture
+
+### Dimension 1: Structural Correctness
+- Both files exist at correct paths
+- YAML frontmatter with `title:` and `description:`
+
+### Dimension 2: Content Accuracy
+- NIXLBench features match README source
+- Build options match `build.sh` actual flags
+- System requirements match README
+
+### Dimension 3: Cross-Reference Integrity
+- Backend names link to correct User Guide pages
+- NIXL build link works
+- etcd link works
+
+### Dimension 4: Fern Compatibility
+- No GitHub-only constructs
+- `` component used correctly
+- `fern check` passes
+
+### Dimension 5: Non-Duplication
+- Build page links to existing NIXL build docs, not repeating them
+- Docker section shows essentials, links to README for full table
+
+## Key Findings
+
+1. **Nav update needed:** `docs/index.yml` must add `build.md` under NIXLBench section
+2. **New file needed:** `docs/development/benchmarking/nixlbench/build.md` does not exist yet
+3. **Stub replacement:** `docs/development/benchmarking/nixlbench/index.md` has stub content to be replaced
+4. **D-06 conflict with roadmap SC-1:** ROADMAP success criterion 1 says overview page should describe system requirements, but CONTEXT.md D-06 says system requirements go on the build page. CONTEXT.md decisions (user preferences) take precedence — system requirements go on build page.
+5. **Terminology:** Use `etcd` (lowercase) in prose, `plug-in` (hyphenated), consistent backend capitalization per QS-01
+
+## RESEARCH COMPLETE
diff --git a/.planning/phases/33-nixlbench-overview-and-build/33-VALIDATION.md b/.planning/phases/33-nixlbench-overview-and-build/33-VALIDATION.md
new file mode 100644
index 0000000000..328fadf39f
--- /dev/null
+++ b/.planning/phases/33-nixlbench-overview-and-build/33-VALIDATION.md
@@ -0,0 +1,74 @@
+---
+phase: 33
+slug: nixlbench-overview-and-build
+status: draft
+nyquist_compliant: true
+wave_0_complete: false
+created: 2026-04-07
+---
+
+# Phase 33 — Validation Strategy
+
+> Per-phase validation contract for feedback sampling during execution.
+
+---
+
+## Test Infrastructure
+
+| Property | Value |
+|----------|-------|
+| **Framework** | Fern CLI (`fern check`) |
+| **Config file** | `fern/fern.config.json` |
+| **Quick run command** | `cd fern && fern check` |
+| **Full suite command** | `cd fern && fern check` |
+| **Estimated runtime** | ~5 seconds |
+
+---
+
+## Sampling Rate
+
+- **After every task commit:** Run `cd fern && fern check`
+- **After every plan wave:** Run `cd fern && fern check`
+- **Before `/gsd-verify-work`:** Full suite must be green
+- **Max feedback latency:** 5 seconds
+
+---
+
+## Per-Task Verification Map
+
+| Task ID | Plan | Wave | Requirement | Test Type | Automated Command | Status |
+|---------|------|------|-------------|-----------|-------------------|--------|
+| 33-01-01 | 01 | 1 | NB-01, NB-02 | structural | `test -f docs/development/benchmarking/nixlbench/index.md && test -f docs/development/benchmarking/nixlbench/build.md` | pending |
+| 33-01-02 | 01 | 1 | NB-01 | content | `grep -q 'title:' docs/development/benchmarking/nixlbench/index.md` | pending |
+| 33-01-03 | 01 | 1 | NB-02 | content | `grep -q '' docs/development/benchmarking/nixlbench/build.md` | pending |
+| 33-01-04 | 01 | 1 | NB-02 | cross-link | `grep -q 'building-nixl' docs/development/benchmarking/nixlbench/build.md` | pending |
+| 33-01-05 | 01 | 1 | NB-01, NB-02 | fern | `cd fern && fern check` | pending |
+
+*Status: pending / green / red / flaky*
+
+---
+
+## Wave 0 Requirements
+
+*Existing infrastructure covers all phase requirements. `fern check` is already available.*
+
+---
+
+## Manual-Only Verifications
+
+| Behavior | Requirement | Why Manual | Test Instructions |
+|----------|-------------|------------|-------------------|
+| Visual rendering of Tabs component | NB-02 | Requires visual inspection in browser | Run `cd fern && fern docs dev`, navigate to NIXLBench build page, verify Docker/Native tabs render correctly |
+
+---
+
+## Validation Sign-Off
+
+- [x] All tasks have automated verify commands
+- [x] Sampling continuity: fern check runs after every task
+- [x] Wave 0 covers all MISSING references
+- [x] No watch-mode flags
+- [x] Feedback latency < 5s
+- [x] `nyquist_compliant: true` set in frontmatter
+
+**Approval:** pending
diff --git a/.planning/phases/33-nixlbench-overview-and-build/33-VERIFICATION.md b/.planning/phases/33-nixlbench-overview-and-build/33-VERIFICATION.md
new file mode 100644
index 0000000000..f3be3e3799
--- /dev/null
+++ b/.planning/phases/33-nixlbench-overview-and-build/33-VERIFICATION.md
@@ -0,0 +1,75 @@
+---
+status: passed
+phase: 33
+verified: 2026-04-07
+---
+
+# Phase 33 Verification: NIXLBench Overview and Build
+
+## Phase Goal
+Developers can read what NIXLBench is and how to build it from a clean environment, with no duplication of steps already in the existing NIXL docs.
+
+## Success Criteria Verification
+
+### SC-1: Overview page describes NIXLBench features and system requirements
+**Status:** PASS
+
+- `docs/development/benchmarking/nixlbench/index.md` has frontmatter with `title: NIXLBench` and `description:`
+- Opening paragraph describes what NIXLBench is (benchmarking tool for NIXL data transfer performance)
+- Features section covers: network backends (UCX, Libfabric, Mooncake, DOCA GPUNetIO), storage backends (GDS, GDS_MT, POSIX, HF3FS, OBJ, Azure Blob, GUSLI), communication patterns (pairwise, many-to-one, one-to-many, TP), memory types (DRAM/VRAM), etcd coordination
+- System requirements are on the build page per user decision D-06 (CONTEXT.md takes precedence over ROADMAP SC wording)
+
+### SC-2: Build page presents Docker and native side-by-side with Tabs
+**Status:** PASS
+
+- `docs/development/benchmarking/nixlbench/build.md` exists with ``, ``, ``
+- Docker tab: `./build.sh` invocation, `--build-type` and `--arch` options shown, link to README for full options
+- Native tab: meson build commands, 5-row options table, post-install environment setup
+- All `build.sh` options documented via link to README (essentials shown inline per D-04)
+
+### SC-3: Build page links to existing NIXL build docs
+**Status:** PASS
+
+- Line 6: "NIXLBench requires a NIXL installation -- see [Building NIXL from Source](/docs/user-guide/building-nixl)"
+- Line 70: Native tab also links to Building NIXL from Source
+- No NIXL build steps duplicated
+
+### SC-4: Both pages use valid Fern MDX
+**Status:** PASS
+
+- Both files have YAML frontmatter with `title:` and `description:`
+- No `` HTML comments
+- No GitHub-Markdown-only constructs
+- `fern check` passes with 0 errors (1 pre-existing warning about color contrast)
+
+## Requirements Traceability
+
+| Requirement | Status | Evidence |
+|-------------|--------|---------|
+| NB-01 | Satisfied | Overview page covers what NIXLBench is, key features (all backend types, communication patterns, memory types, etcd coordination) |
+| NB-02 | Satisfied | Build page uses `` for Docker/Native, documents build.sh essentials, links to existing NIXL build docs |
+
+## Context Decisions Honored
+
+| Decision | Honored | Evidence |
+|----------|---------|---------|
+| D-01 (separate pages) | Yes | index.md and build.md are separate files |
+| D-02 (problem-first) | Yes | Opening paragraph leads with what NIXLBench solves |
+| D-03 (Tabs component) | Yes | `` with Docker and Native tabs |
+| D-04 (essentials only) | Yes | 2 options shown inline, link to README for full table |
+| D-05 (sentence + link) | Yes | Single sentence with link, no callout box |
+| D-06 (sys reqs on build) | Yes | System Requirements section on build.md |
+| D-07 (rewrite prose) | Yes | Content rewritten for doc site tone |
+| D-08 (concise features) | Yes | Grouped bullet list, no deep breakdown |
+| D-09 (backend links) | Yes | All 11 backends linked on first mention |
+| D-10 (etcd link) | Yes | etcd linked to Metadata Exchange page |
+
+## Human Verification Items
+
+| Item | Description | Status |
+|------|-------------|--------|
+| Visual Tabs rendering | Verify Docker/Native tabs render correctly in browser via `fern docs dev` | Not tested (visual) |
+
+## Verification Result
+
+**PASSED** -- All automated success criteria met. One visual verification item (Tabs rendering) deferred to manual testing.
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-PLAN.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-PLAN.md
new file mode 100644
index 0000000000..6375c2f946
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-PLAN.md
@@ -0,0 +1,331 @@
+---
+phase: 34-nixlbench-usage-troubleshooting-and-cross-references
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+ - docs/development/benchmarking/nixlbench/usage.md
+autonomous: true
+requirements: [NB-03, NB-04, NB-05, NB-06]
+must_haves:
+ truths:
+ - "Developer can learn how to launch initiator and target NIXLBench workers with etcd coordination"
+ - "Developer can see examples of all four communication patterns (pairwise, many-to-one, one-to-many, TP)"
+ - "Developer can find storage backend examples (GDS and OBJ) with links to backend pages"
+ - "Developer can look up essential CLI flags in organized tables"
+ - "Developer can troubleshoot etcd connection failures, CUDA/GPU not found, backend library missing, and build failures"
+ - "Every backend name mentioned in prose links to its User Guide page on first mention"
+ - "A Warning callout alerts developers to the 60-second etcd join window barrier"
+ artifacts:
+ - path: "docs/development/benchmarking/nixlbench/usage.md"
+ provides: "NIXLBench usage guide and troubleshooting page"
+ contains: "title: NIXLBench Usage and Troubleshooting"
+ key_links:
+ - from: "docs/development/benchmarking/nixlbench/usage.md"
+ to: "/docs/user-guide/etcd-metadata-exchange"
+ via: "Fern inline link in etcd Coordination section"
+ pattern: "\\(/docs/user-guide/etcd-metadata-exchange\\)"
+ - from: "docs/development/benchmarking/nixlbench/usage.md"
+ to: "/docs/user-guide/backends/ucx"
+ via: "First-mention backend link"
+ pattern: "\\[UCX\\]\\(/docs/user-guide/backends/ucx\\)"
+ - from: "docs/development/benchmarking/nixlbench/usage.md"
+ to: "/docs/user-guide/backends/gds"
+ via: "First-mention backend link"
+ pattern: "\\[GPUDirect Storage\\]\\(/docs/user-guide/backends/gds\\)"
+ - from: "docs/development/benchmarking/nixlbench/usage.md"
+ to: "/docs/user-guide/backends/obj"
+ via: "First-mention backend link"
+ pattern: "\\[OBJ\\]\\(/docs/user-guide/backends/obj\\)"
+---
+
+
+Author the NIXLBench usage and troubleshooting page (usage.md), replacing the Phase 32 stub with complete content covering etcd coordination, all four communication patterns, storage backend examples, essential CLI flags, and troubleshooting for four common failure modes. Cross-link all backend names and etcd to their existing User Guide pages.
+
+Purpose: Developers need a single page to learn how to run NIXLBench end-to-end and troubleshoot common failures, with navigation to backend-specific and etcd documentation.
+Output: Complete `docs/development/benchmarking/nixlbench/usage.md` passing `fern check`.
+
+
+
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/workflows/execute-plan.md
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/templates/summary.md
+
+
+
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-CONTEXT.md
+@.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-RESEARCH.md
+@.planning/phases/33-nixlbench-overview-and-build/33-01-SUMMARY.md
+
+
+
+
+From docs/development/benchmarking/nixlbench/index.md:
+- Frontmatter: `title:` + `description:`
+- Backend link pattern: `[UCX](/docs/user-guide/backends/ucx)`, `[GPUDirect Storage](/docs/user-guide/backends/gds)`
+- etcd link: `[etcd](/docs/user-guide/etcd-metadata-exchange)`
+- Callout syntax: `... `
+
+Backend link map (all verified):
+| Backend Name in Prose | Link Target |
+|----------------------|-------------|
+| UCX | `/docs/user-guide/backends/ucx` |
+| GPUDirect Storage (GDS) | `/docs/user-guide/backends/gds` |
+| GPUDirect Storage MT (GDS_MT) | `/docs/user-guide/backends/gds-mt` |
+| POSIX | `/docs/user-guide/backends/posix` |
+| DOCA GPUNetIO | `/docs/user-guide/backends/gpunetio` |
+| Mooncake | `/docs/user-guide/backends/mooncake` |
+| Libfabric | `/docs/user-guide/backends/libfabric` |
+| HF3FS | `/docs/user-guide/backends/hf3fs` |
+| OBJ | `/docs/user-guide/backends/obj` |
+| Azure Blob | `/docs/user-guide/backends/azure-blob` |
+| GUSLI | `/docs/user-guide/backends/gusli` |
+
+
+
+
+
+
+ Task 1: Author usage.md with all sections
+ docs/development/benchmarking/nixlbench/usage.md
+
+ - docs/development/benchmarking/nixlbench/usage.md (current stub to replace)
+ - benchmark/nixlbench/README.md (primary source material -- lines 390-915)
+ - docs/development/benchmarking/nixlbench/index.md (established Fern MDX patterns, backend link format)
+ - docs/development/benchmarking/nixlbench/build.md (established Fern MDX patterns, Tabs/Warning usage)
+ - docs/user-guide/etcd-metadata-exchange.md (link target -- confirm path)
+ - .planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-CONTEXT.md (user decisions D-01 through D-12)
+ - .planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-RESEARCH.md (source material mapping, code examples, pitfalls)
+
+
+Replace the stub `usage.md` with a complete Fern MDX page. The page structure per D-01 and D-02 is:
+
+**Frontmatter:**
+```yaml
+---
+title: NIXLBench Usage and Troubleshooting
+description: How to run NIXLBench benchmarks and troubleshoot common issues.
+---
+```
+
+**Intro paragraph:** Brief 2-3 sentence overview of what this page covers (running benchmarks, communication patterns, troubleshooting). Link to the build page for installation prerequisites.
+
+**Section 1: etcd Coordination** (per D-03, D-04, D-12)
+- 2-3 sentences on when etcd is required vs optional (per D-04): network backends (UCX, GPUNETIO, Mooncake, Libfabric) and multi-node setups REQUIRE etcd; storage backends (GDS, GDS_MT, POSIX, HF3FS, OBJ, GUSLI) can run without it for single instances.
+- Docker one-liner to start etcd server (condensed from README lines 398-408):
+ ```bash
+ docker run -d --name etcd-server \
+ -p 2379:2379 -p 2380:2380 \
+ quay.io/coreos/etcd:v3.5.18 \
+ /usr/local/bin/etcd \
+ --data-dir=/etcd-data \
+ --listen-client-urls=http://0.0.0.0:2379 \
+ --advertise-client-urls=http://0.0.0.0:2379 \
+ --listen-peer-urls=http://0.0.0.0:2380 \
+ --initial-advertise-peer-urls=http://0.0.0.0:2380 \
+ --initial-cluster=default=http://0.0.0.0:2380
+ ```
+- `` callout for the 60-second join window barrier (per D-03, NB-05). Use this exact text:
+ ```
+
+ All NIXLBench workers in a benchmark group must connect to etcd within 60 seconds of the first worker joining. Workers that miss this window cause the barrier to fail and the benchmark to abort. For etcd setup and configuration details, see [Metadata Exchange with etcd](/docs/user-guide/etcd-metadata-exchange).
+
+ ```
+- Use `etcd` (lowercase) in prose, never `ETCD` except in CLI flag values like `--runtime_type ETCD`. Per QS-01.
+
+**Section 2: Communication Patterns** (per D-05, D-07)
+Four subsections, one per pattern. Each has 2-3 sentences explaining the pattern + a code example using UCX backend with `--scheme` flag.
+
+- **Pairwise** (default): Data transfers between matched pairs of initiators and targets. Show two-host example (per D-07) with `--scheme pairwise` on both hosts. Use `--etcd_endpoints http://etcd-server:2379 --backend UCX --initiator_seg_type VRAM --target_seg_type VRAM`.
+- **Many-to-one**: Multiple initiators send to a single target. Show `--scheme manytoone`.
+- **One-to-many**: Single initiator sends to multiple targets. Show `--scheme onetomany`.
+- **TP (Tensor Parallel)**: All-to-all exchange for tensor-parallel workloads. Show `--scheme tp`.
+
+For Pairwise, show multi-node (D-07): "On host 1 (initiator):" and "On host 2 (target):" with separate code blocks. For the other 3, a single code block is sufficient with a note about launching the appropriate number of workers.
+
+**Section 3: Storage Backend Examples** (per D-06)
+- GDS example: single-instance (no etcd), direct I/O flag. Link to [GPUDirect Storage](/docs/user-guide/backends/gds) page. Source: README line 629-637.
+ ```bash
+ nixlbench --backend GDS --filepath /mnt/storage/testfile --storage_enable_direct
+ ```
+- OBJ (S3) example: using env vars and CLI flags. Link to [OBJ](/docs/user-guide/backends/obj) page. Source: README lines 730-753.
+ ```bash
+ nixlbench --backend OBJ \
+ --obj_access_key $AWS_ACCESS_KEY_ID \
+ --obj_secret_key $AWS_SECRET_ACCESS_KEY \
+ --obj_region us-east-1 \
+ --obj_bucket_name my-bucket
+ ```
+- Brief note linking to backend User Guide pages for backend-specific flags rather than documenting them here.
+
+**Section 4: CLI Options** (per D-08, discretion)
+Two tables: Core Configuration and Memory and Transfer Configuration. Use markdown tables with Flag, Description, Default columns.
+
+Core Configuration table (6 flags from README lines 434-441):
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--config_file` | Configuration file in TOML format | None |
+| `--runtime_type` | Runtime coordination type | `ETCD` |
+| `--worker_type` | Worker transfer engine (`nixl`, `nvshmem`) | `nixl` |
+| `--backend` | Communication backend (UCX, GDS, GDS_MT, POSIX, GPUNETIO, Mooncake, HF3FS, OBJ, GUSLI) | `UCX` |
+| `--benchmark_group` | Group name for parallel runs | `default` |
+| `--etcd_endpoints` | etcd server URL for coordination | `http://localhost:2379` |
+
+Memory and Transfer Configuration table (12 flags from README lines 444-457):
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--initiator_seg_type` | Initiator memory segment type (DRAM, VRAM) | `DRAM` |
+| `--target_seg_type` | Target memory segment type (DRAM, VRAM) | `DRAM` |
+| `--scheme` | Communication pattern (pairwise, manytoone, onetomany, tp) | `pairwise` |
+| `--mode` | Process mode: SG (single GPU) or MG (multi GPU) | `SG` |
+| `--op_type` | Operation type (READ, WRITE) | `WRITE` |
+| `--check_consistency` | Enable data consistency checking | disabled |
+| `--total_buffer_size` | Total buffer size per process | `8GiB` |
+| `--start_block_size` | Starting block size | `4KiB` |
+| `--max_block_size` | Maximum block size | `64MiB` |
+| `--start_batch_size` | Starting batch size | `1` |
+| `--max_batch_size` | Maximum batch size | `1` |
+| `--recreate_xfer` | Recreate transfer handle per iteration | disabled |
+
+After the tables, add a brief 3-4 sentence note about `--config_file` accepting TOML files where CLI parameter names are used as keys, and CLI flags override config file values. Per RESEARCH discretion recommendation.
+
+Also add a 2-sentence note about `--worker_type nvshmem` for GPU-only VRAM transfers. Per RESEARCH discretion recommendation.
+
+**Section 5: Troubleshooting** (per D-09, D-10)
+Four required subsections plus two optional ones. Each has **Symptoms:** and **Resolution:** format. Order per RESEARCH recommendation:
+
+1. **etcd Connection Failures** -- Symptoms: workers fail to join, barrier timeout. Resolution: verify etcd health with `ETCDCTL_API=3 etcdctl endpoint health --endpoints=http://etcd-server:2379`, clean up stale keys with `ETCDCTL_API=3 etcdctl del "xferbench" --prefix=true`. Source: README 911-915.
+
+2. **Build Failures** -- Symptoms: compilation errors during native build. Resolution: UCX missing RDMA libs (`sudo apt-get reinstall -y libibverbs-dev librdmacm-dev rdma-core`), etcd-cpp-api errors (`sudo apt-get install -y libcpprest-dev`, protobuf), Docker cache issues (`docker system prune -a`). Source: README 836-869. Link to [build page](./nixlbench/build) for full build instructions.
+
+3. **CUDA / GPU Not Found** -- Symptoms: CUDA errors at launch. Resolution: export PATH/LD_LIBRARY_PATH, verify with `nvcc --version` and `nvidia-smi`. Source: README 822-893.
+
+4. **Backend Library Missing** -- Symptoms: library-not-found errors at runtime. Resolution: `sudo ldconfig`, `ldd /usr/local/nixlbench/bin/nixlbench`. Source: README 874-880.
+
+**Cross-linking rules (D-11, NB-06):**
+- Link every backend name on FIRST mention in prose (not in code blocks) to its User Guide page using the backend link map from the interfaces section above.
+- Use the Fern absolute path format: `[UCX](/docs/user-guide/backends/ucx)` -- never relative paths, never `.md` extensions.
+- After first mention per page, use the plain backend name without a link.
+
+**Terminology (QS-01):**
+- `etcd` lowercase in prose, `ETCD` only in CLI flag values
+- `plug-in` not `plugin` if the word appears
+- Backend names follow canonical capitalization: UCX, GDS, GDS_MT, POSIX, GPUNETIO, OBJ, GUSLI, HF3FS
+
+
+ cd /home/omrik/Projects/nixl.gitlab/fern && fern check 2>&1 | tail -5
+
+
+ - File `docs/development/benchmarking/nixlbench/usage.md` exists and is NOT the stub (does NOT contain "under construction")
+ - Frontmatter contains `title: NIXLBench Usage and Troubleshooting`
+ - Contains `` callout with "60 seconds" and link to `/docs/user-guide/etcd-metadata-exchange`
+ - Contains `## etcd Coordination` or `## etcd coordination` section header (lowercase etcd in prose)
+ - Contains four pattern subsection headers: grep finds `pairwise` AND `many-to-one` AND `one-to-many` AND `TP` or `Tensor Parallel`
+ - Contains `--scheme pairwise` in a code block
+ - Contains `--scheme manytoone` in a code block
+ - Contains `--scheme onetomany` in a code block
+ - Contains `--scheme tp` in a code block
+ - Contains GDS storage example with `--backend GDS`
+ - Contains OBJ storage example with `--backend OBJ`
+ - Contains two CLI tables with `--config_file` and `--initiator_seg_type`
+ - Contains four troubleshooting subsections: grep finds `etcd Connection` AND `Build Failures` AND `CUDA` AND `Backend Library`
+ - Contains `ETCDCTL_API=3 etcdctl del "xferbench" --prefix=true`
+ - `fern check` passes (zero errors; pre-existing accent color warning is acceptable)
+
+
+
+
+ Task 2: Verify backend cross-links and terminology compliance
+ docs/development/benchmarking/nixlbench/usage.md
+
+ - docs/development/benchmarking/nixlbench/usage.md (just written by Task 1)
+ - docs/development/benchmarking/nixlbench/index.md (backend link patterns to match)
+ - docs/user-guide/backends/ (directory listing to confirm all targets exist)
+
+
+Audit the completed `usage.md` for NB-06 compliance and terminology correctness. Perform these checks:
+
+1. **Backend first-mention audit (NB-06):** For each backend name that appears in prose (not code blocks), verify the FIRST occurrence has an inline link to the correct User Guide page. The required backend link targets are:
+ - UCX -> `/docs/user-guide/backends/ucx`
+ - GPUDirect Storage or GDS -> `/docs/user-guide/backends/gds`
+ - GPUDirect Storage MT or GDS_MT -> `/docs/user-guide/backends/gds-mt`
+ - POSIX -> `/docs/user-guide/backends/posix`
+ - DOCA GPUNetIO or GPUNETIO -> `/docs/user-guide/backends/gpunetio`
+ - Mooncake -> `/docs/user-guide/backends/mooncake`
+ - Libfabric -> `/docs/user-guide/backends/libfabric`
+ - HF3FS -> `/docs/user-guide/backends/hf3fs`
+ - OBJ -> `/docs/user-guide/backends/obj`
+ - Azure Blob -> `/docs/user-guide/backends/azure-blob`
+ - GUSLI -> `/docs/user-guide/backends/gusli`
+
+ If any backend mentioned in prose is missing its first-mention link, add it.
+
+2. **etcd link audit (NB-05, D-12):** Confirm `[etcd](/docs/user-guide/etcd-metadata-exchange)` or `[Metadata Exchange with etcd](/docs/user-guide/etcd-metadata-exchange)` appears in the etcd Coordination section. Confirm the `` callout contains the link.
+
+3. **Terminology audit (QS-01 prep):**
+ - grep for `ETCD` in prose (outside code blocks and CLI flag references) -- should be zero matches in descriptive text. Fix any to `etcd`.
+ - grep for `plugin` (should be `plug-in`) -- fix if found.
+ - Verify no relative links (`../`) or `.md` extensions in cross-page links.
+
+4. **Final fern check:** Run `cd fern && fern check` to confirm zero errors.
+
+If all checks pass with no fixes needed, document the verification results. If fixes are needed, apply them to usage.md.
+
+
+ cd /home/omrik/Projects/nixl.gitlab && grep -c '/docs/user-guide/backends/' docs/development/benchmarking/nixlbench/usage.md && grep -c '/docs/user-guide/etcd-metadata-exchange' docs/development/benchmarking/nixlbench/usage.md && grep -c '' docs/development/benchmarking/nixlbench/usage.md && cd fern && fern check 2>&1 | tail -3
+
+
+ - `grep -c '/docs/user-guide/backends/' usage.md` returns a number >= 3 (at minimum UCX, GDS, OBJ are linked)
+ - `grep -c '/docs/user-guide/etcd-metadata-exchange' usage.md` returns >= 1
+ - `grep -c '' usage.md` returns >= 1
+ - `grep -P '(?
+
+
+
+
+
+## Trust Boundaries
+
+| Boundary | Description |
+|----------|-------------|
+| Documentation content | Static MDX content with no user input or dynamic execution |
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-34-01 | Tampering | CLI examples in usage.md | accept | Documentation examples are read-only prose; no executable code paths. Users copy commands voluntarily. |
+| T-34-02 | Information Disclosure | etcd Docker example exposes default ports | accept | Standard etcd ports (2379/2380) on 0.0.0.0; this is the documented default and matches upstream etcd docs. Users configure firewalls per their environment. |
+
+
+
+1. `fern check` passes with zero errors
+2. `grep '' usage.md` finds the 60-second barrier callout
+3. `grep '/docs/user-guide/etcd-metadata-exchange' usage.md` confirms etcd cross-link
+4. `grep '/docs/user-guide/backends/' usage.md` confirms backend cross-links
+5. All four communication pattern `--scheme` values appear in code blocks
+6. All four troubleshooting failure modes have section headers
+
+
+
+- usage.md is a complete, non-stub Fern MDX page
+- Page covers: etcd coordination, 4 communication patterns, storage examples, CLI tables, 4 troubleshooting sections
+- Warning callout for 60-second etcd barrier with link to etcd-metadata-exchange page
+- Backend names linked to User Guide pages on first mention in prose
+- fern check passes with zero errors
+- No terminology violations (etcd lowercase in prose, plug-in not plugin)
+
+
+
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-SUMMARY.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-SUMMARY.md
new file mode 100644
index 0000000000..44f1b46690
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-SUMMARY.md
@@ -0,0 +1,92 @@
+---
+phase: 34-nixlbench-usage-troubleshooting-and-cross-references
+plan: 01
+subsystem: docs
+tags: [nixlbench, fern-mdx, etcd, benchmarking, troubleshooting, cross-links]
+
+# Dependency graph
+requires:
+ - phase: 33-nixlbench-overview-and-build
+ provides: NIXLBench index.md and build.md pages with established Fern MDX patterns and backend link format
+provides:
+ - Complete NIXLBench usage and troubleshooting page with etcd coordination, communication patterns, storage examples, CLI tables, and troubleshooting
+affects: [37-terminology-normalization-and-quality-audit]
+
+# Tech tracking
+tech-stack:
+ added: []
+ patterns: [first-mention backend cross-linking, Warning callout for critical timing constraints, Symptoms/Resolution troubleshooting format]
+
+key-files:
+ created: []
+ modified: [docs/development/benchmarking/nixlbench/usage.md]
+
+key-decisions:
+ - "All 11 backend names linked on first mention in the etcd Coordination paragraph for compact cross-referencing"
+ - "CLI tables split into Core Configuration (6 flags) and Memory/Transfer Configuration (12 flags) per plan spec"
+ - "Troubleshooting uses Symptoms/Resolution format for scannable problem-solving"
+ - "TOML config file and NVSHMEM worker type noted briefly after CLI tables per discretion recommendations"
+
+patterns-established:
+ - "Symptoms/Resolution troubleshooting format for developer-facing docs"
+ - "First-mention backend cross-linking consolidated in introductory paragraph"
+
+requirements-completed: [NB-03, NB-04, NB-05, NB-06]
+
+# Metrics
+duration: 2min
+completed: 2026-04-07
+---
+
+# Phase 34 Plan 01: NIXLBench Usage and Troubleshooting Summary
+
+**Complete usage guide with etcd coordination, four communication patterns (pairwise, many-to-one, one-to-many, TP), GDS and OBJ storage examples, 18-flag CLI reference, and four troubleshooting sections with cross-links to all backend User Guide pages**
+
+## Performance
+
+- **Duration:** 2 min
+- **Started:** 2026-04-07T20:16:16Z
+- **Completed:** 2026-04-07T20:18:20Z
+- **Tasks:** 2
+- **Files modified:** 1
+
+## Accomplishments
+- Replaced stub usage.md with 249-line complete Fern MDX page covering all NIXLBench usage scenarios
+- Cross-linked all 10 backend names (UCX, GDS, GDS_MT, POSIX, GPUNETIO, Mooncake, Libfabric, HF3FS, OBJ, GUSLI) to their User Guide pages on first mention
+- Warning callout for the 60-second etcd join window barrier with link to etcd metadata exchange page
+- Four communication patterns with multi-node examples for pairwise and single-command examples for the other three
+- Four troubleshooting sections covering etcd connection failures, build failures, CUDA/GPU not found, and backend library missing
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Author usage.md with all sections** - `9ed05e4b` (feat)
+2. **Task 2: Verify backend cross-links and terminology compliance** - no commit (verification-only, no fixes needed)
+
+## Files Created/Modified
+- `docs/development/benchmarking/nixlbench/usage.md` - Complete NIXLBench usage and troubleshooting page replacing Phase 32 stub
+
+## Decisions Made
+- Consolidated all 10 backend first-mention links into the etcd Coordination paragraph rather than spreading them across sections -- provides a dense but scannable cross-reference hub
+- Included TOML config file note and NVSHMEM worker type mention as brief paragraphs after CLI tables per RESEARCH discretion recommendations
+- Used "On host 1 (initiator):" / "On host 2 (target):" format for pairwise multi-node example per D-07
+
+## Deviations from Plan
+
+None - plan executed exactly as written.
+
+## Issues Encountered
+None
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- usage.md is complete and fern check passes with zero errors
+- All backend cross-links verified; terminology (etcd lowercase, no plugin) compliant
+- Ready for Phase 37 terminology normalization and quality audit
+
+---
+*Phase: 34-nixlbench-usage-troubleshooting-and-cross-references*
+*Completed: 2026-04-07*
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-02-PLAN.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-02-PLAN.md
new file mode 100644
index 0000000000..ad08c3dea1
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-02-PLAN.md
@@ -0,0 +1,161 @@
+---
+phase: 34-nixlbench-usage-troubleshooting-and-cross-references
+plan: 02
+type: execute
+wave: 1
+depends_on: ["34-01"]
+files_modified: ["docs/development/benchmarking/nixlbench/usage.md"]
+autonomous: true
+gap_closure: true
+requirements: ["NB-03"]
+
+must_haves:
+ truths:
+ - "Developer can understand what NIXLBench prints after a benchmark run completes"
+ - "Developer can identify the meaning of each output column (block size, batch size, bandwidth, latency, prep/post/transfer phases)"
+ - "Developer can distinguish between standard output and multi-worker pairwise output (which adds Aggregate B/W and Network Util columns)"
+ artifacts:
+ - path: "docs/development/benchmarking/nixlbench/usage.md"
+ provides: "Reading Benchmark Output section with column descriptions and units"
+ contains: "Reading Benchmark Output"
+ key_links: []
+---
+
+
+Close verification gap: add a "Reading Benchmark Output" section to usage.md explaining the NIXLBench output table format, column meanings, and units.
+
+Purpose: NB-03 requires coverage of "reading benchmark output" and the phase 34 verification found this missing. The source code (benchmark/nixlbench/src/utils/utils.cpp printStatsHeader/printStats) reveals the exact output format.
+
+Output: Updated usage.md with a new section between "CLI Options" and "Troubleshooting" explaining the benchmark results table.
+
+
+
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/workflows/execute-plan.md
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/templates/summary.md
+
+
+
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-CONTEXT.md
+@.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-RESEARCH.md
+@.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-01-SUMMARY.md
+@.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-VERIFICATION.md
+
+
+
+
+Standard output header (all patterns except multi-worker pairwise):
+ Block Size (B) | Batch Size | B/W (GB/Sec) | Avg Lat. (us) | Avg Prep (us) | P99 Prep (us) | Avg Post (us) | P99 Post (us) | Avg Tx (us) | P99 Tx (us)
+
+Multi-worker pairwise output header (pairwise with >2 workers):
+ Block Size (B) | Batch Size | B/W (GB/Sec) | Aggregate B/W (GB/Sec) | Network Util (%) | Avg Lat. (us) | Avg Prep (us) | P99 Prep (us) | Avg Post (us) | P99 Post (us) | Avg Tx (us) | P99 Tx (us)
+
+From the stats computation (utils.cpp lines 1099-1171):
+- Block Size (B): Transfer block size in bytes, sweeps from --start_block_size to --max_block_size (doubling each iteration)
+- Batch Size: Number of transfers per batch, sweeps from --start_batch_size to --max_batch_size (doubling)
+- B/W (GB/Sec): Throughput = (block_size * batch_size * num_iter) / total_duration, in GB/sec (base-10, 1e9 bytes)
+- Aggregate B/W (GB/Sec): Sum of all workers' B/W (multi-worker pairwise only, via MPI reduce)
+- Network Util (%): Aggregate B/W / (num_pairs * MAXBW) * 100 (multi-worker pairwise only)
+- Avg Lat. (us): Average latency per transfer in microseconds = total_duration / (num_iter * batch_size)
+- Avg Prep (us): Average prepare phase duration (buffer registration, handle setup)
+- P99 Prep (us): 99th percentile prepare duration
+- Avg Post (us): Average post-transfer phase duration (completion checking, cleanup)
+- P99 Post (us): 99th percentile post duration
+- Avg Tx (us): Average transfer phase duration (actual data movement)
+- P99 Tx (us): 99th percentile transfer duration
+
+The output prints one row per (block_size, batch_size) combination. Block sizes double from start to max, and for each block size, batch sizes double from start to max.
+
+
+
+
+
+
+ Task 1: Add Reading Benchmark Output section to usage.md
+ docs/development/benchmarking/nixlbench/usage.md
+
+ - docs/development/benchmarking/nixlbench/usage.md
+ - benchmark/nixlbench/src/utils/utils.cpp (lines 1057-1171 for printStatsHeader and printStats)
+
+
+Insert a new "## Reading Benchmark Output" section in usage.md between the "## CLI Options" section (which ends around line 159 with the NVSHMEM note) and the "## Troubleshooting" section (which starts around line 161).
+
+The new section must contain:
+
+1. An introductory paragraph (2-3 sentences) explaining that NIXLBench prints a results table after each run, with one row per block-size and batch-size combination. Mention that block sizes sweep from `--start_block_size` to `--max_block_size` (doubling each step) and batch sizes sweep similarly.
+
+2. A markdown table documenting each output column with three columns: Column, Unit, Description. Include these rows:
+
+| Column | Unit | Description |
+|--------|------|-------------|
+| Block Size (B) | Bytes | Transfer block size for this row |
+| Batch Size | Count | Number of transfers per batch |
+| B/W (GB/Sec) | GB/s | Per-worker throughput (total data transferred divided by elapsed time) |
+| Aggregate B/W (GB/Sec) | GB/s | Sum of all workers' throughput (multi-worker pairwise only) |
+| Network Util (%) | Percent | Aggregate bandwidth as a percentage of theoretical peak (multi-worker pairwise only) |
+| Avg Lat. (us) | Microseconds | Average latency per individual transfer operation |
+| Avg Prep (us) | Microseconds | Average time for the prepare phase (buffer registration and handle setup) |
+| P99 Prep (us) | Microseconds | 99th percentile prepare phase duration |
+| Avg Post (us) | Microseconds | Average time for the post phase (completion checking and cleanup) |
+| P99 Post (us) | Microseconds | 99th percentile post phase duration |
+| Avg Tx (us) | Microseconds | Average time for the transfer phase (actual data movement) |
+| P99 Tx (us) | Microseconds | 99th percentile transfer phase duration |
+
+3. A `` callout (Fern MDX component) after the table stating: "When running pairwise benchmarks with more than two workers, NIXLBench adds the Aggregate B/W and Network Util columns. These columns do not appear for other communication patterns or two-worker pairwise runs."
+
+4. A brief paragraph (2-3 sentences) explaining the three latency phases: Prep measures buffer registration and transfer handle setup overhead. Tx measures the actual data movement time. Post measures completion checking and cleanup. Together, Avg Lat. reflects end-to-end latency across all phases.
+
+Do NOT modify any other section of usage.md. Insert only between CLI Options and Troubleshooting.
+
+Use `etcd` (lowercase) in any prose if etcd is mentioned. Use Fern MDX `` syntax (not HTML, not GitHub admonition).
+
+
+ - grep "Reading Benchmark Output" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "B/W (GB/Sec)" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "Avg Lat. (us)" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "P99 Tx (us)" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "Aggregate B/W" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "Network Util" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "" docs/development/benchmarking/nixlbench/usage.md returns a match
+ - grep "multi-worker pairwise" docs/development/benchmarking/nixlbench/usage.md returns a match (lowercase)
+ - The Troubleshooting section still exists after the new section
+
+
+ cd /home/omrik/Projects/nixl.gitlab && grep -c "Reading Benchmark Output" docs/development/benchmarking/nixlbench/usage.md && grep -c "B/W (GB/Sec)" docs/development/benchmarking/nixlbench/usage.md && grep -c "P99 Tx (us)" docs/development/benchmarking/nixlbench/usage.md && grep -c "Aggregate B/W" docs/development/benchmarking/nixlbench/usage.md && grep -c "Troubleshooting" docs/development/benchmarking/nixlbench/usage.md && cd fern && fern check
+
+
+
+
+
+
+## Trust Boundaries
+
+No trust boundaries applicable -- this is a static documentation page with no user input, no API calls, and no dynamic rendering.
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-34-01 | N/A | Static MDX docs page | accept | Documentation-only change, no security surface |
+
+
+
+1. `grep "Reading Benchmark Output" docs/development/benchmarking/nixlbench/usage.md` confirms section exists
+2. `grep "P99"` confirms percentile metrics are documented
+3. `grep "Aggregate B/W"` confirms multi-worker pairwise output is covered
+4. `cd fern && fern check` passes with zero errors
+5. All existing sections (etcd Coordination, Communication Patterns, Storage Backend Examples, CLI Options, Troubleshooting) remain intact
+
+
+
+- NB-03 gap closed: usage.md now covers "reading benchmark output" with column descriptions, units, and explanation of output variants
+- fern check passes
+- No regressions to existing content (all prior verification truths still hold)
+
+
+
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-02-SUMMARY.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-02-SUMMARY.md
new file mode 100644
index 0000000000..c37eb63ff3
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-02-SUMMARY.md
@@ -0,0 +1,32 @@
+---
+phase: 34
+plan: "02"
+status: complete
+started: 2026-04-07
+completed: 2026-04-07
+gap_closure: true
+---
+
+# Plan 34-02: Reading Benchmark Output Gap Closure
+
+## What Was Built
+Added a "Reading Benchmark Output" section to `docs/development/benchmarking/nixlbench/usage.md` between CLI Options and Troubleshooting. The section documents all 12 output columns with units and descriptions, includes a `` callout explaining multi-worker pairwise extra columns, and explains the three latency phases (Prep, Tx, Post).
+
+## Key Files
+
+### Created
+(none)
+
+### Modified
+- `docs/development/benchmarking/nixlbench/usage.md` — Added ~25 lines: output column table, Note callout, latency phase explanation
+
+## Deviations
+None — plan followed exactly.
+
+## Self-Check: PASSED
+- [x] `grep "Reading Benchmark Output"` — section header present
+- [x] `grep "B/W (GB/Sec)"` — throughput column documented
+- [x] `grep "P99 Tx (us)"` — percentile metrics documented
+- [x] `grep "Aggregate B/W"` — multi-worker columns documented
+- [x] `grep ""` — Fern callout present
+- [x] Troubleshooting section still follows the new section
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-CONTEXT.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-CONTEXT.md
new file mode 100644
index 0000000000..0e773eb36d
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-CONTEXT.md
@@ -0,0 +1,124 @@
+# Phase 34: NIXLBench Usage, Troubleshooting, and Cross-References - Context
+
+**Gathered:** 2026-04-07
+**Status:** Ready for planning
+
+
+## Phase Boundary
+
+Author the NIXLBench usage guide covering launching workers, ETCD coordination, all four communication patterns, and interpreting output. Include troubleshooting for common failure modes. Cross-link all backend mentions and ETCD to their existing User Guide pages. No new pages beyond `usage.md` — no separate troubleshooting page.
+
+
+
+
+## Implementation Decisions
+
+### Page Structure
+- **D-01:** Everything goes in a single `usage.md` page — usage guide at the top, troubleshooting section at the bottom. No separate `troubleshooting.md`. This keeps the NIXLBench nav at 3 pages (index.md, build.md, usage.md).
+- **D-02:** Page sections (in order): ETCD Coordination (brief), Communication Patterns (with examples), CLI Options (essential flags), Troubleshooting.
+
+### ETCD Coordination
+- **D-03:** Use a `` callout for the 60-second join window barrier. Link to the existing "Metadata Exchange with ETCD" User Guide page (`docs/user-guide/etcd-metadata-exchange.md`). Brief Docker one-liner for starting ETCD. Do NOT re-explain ETCD setup in detail. Satisfies requirement NB-05.
+- **D-04:** Mention when ETCD is required vs optional (network backends require it; storage backends can run without it for single instances). Keep this concise — 2-3 sentences.
+
+### Usage Examples
+- **D-05:** Focus examples on the 4 communication patterns (pairwise, many-to-one, one-to-many, TP) using UCX as the default backend. Show the `--scheme` flag variations with brief explanation of each pattern.
+- **D-06:** Add 1-2 storage backend examples (GDS for local storage, OBJ for S3) to demonstrate how storage benchmarks differ from network benchmarks. Link to backend User Guide pages for backend-specific flags rather than documenting them here.
+- **D-07:** Show multi-node examples demonstrating initiator/target worker launching (two `nixlbench` commands on separate hosts pointing to the same ETCD server).
+
+### CLI Options
+- **D-08:** Claude's discretion on how many flags to document. Should include enough for a developer to run all 4 communication patterns and basic storage benchmarks. Skip per-backend flag tables — those belong in the backend docs or the full CLI reference (deferred).
+
+### Troubleshooting
+- **D-09:** Cover the 4 ROADMAP-required failure modes: ETCD connection failures, CUDA/GPU not found, backend library missing, build failures.
+- **D-10:** Claude's discretion on whether to add runtime essentials (library-not-found errors, ETCD cleanup after failed runs). These are the two most common runtime issues from the README. Include if they fit naturally; skip if the page is already long enough.
+
+### Cross-Linking
+- **D-11:** Link every backend name (UCX, GDS, GDS_MT, POSIX, GPUNETIO, Mooncake, HF3FS, OBJ, GUSLI, Azure Blob) to its corresponding User Guide backend page on first mention per page. Satisfies NB-06. Backend pages exist at `docs/user-guide/backends/{backend}.md`.
+- **D-12:** Link to "Metadata Exchange with ETCD" (`docs/user-guide/etcd-metadata-exchange.md`) when ETCD coordination is discussed. Satisfies NB-05.
+
+### Claude's Discretion
+- Exact CLI flag table scope (core only vs core + memory/transfer)
+- Whether to include ETCD cleanup and library-not-found in troubleshooting
+- Ordering of troubleshooting items
+- Whether to include a config file example (TOML format)
+- Whether to mention NVSHMEM worker type or keep focus on the default NIXL worker
+
+
+
+
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Source material
+- `benchmark/nixlbench/README.md` — Primary source for usage examples, CLI flags, ETCD coordination, troubleshooting steps, and backend-specific examples
+- `benchmark/nixlbench/README.md` §Usage (line ~390) — Worker launching, ETCD setup, basic examples
+- `benchmark/nixlbench/README.md` §Command Line Options (line ~432) — Full CLI flag reference (70+ flags)
+- `benchmark/nixlbench/README.md` §Troubleshooting (line ~817) — Build and runtime troubleshooting
+
+### Existing docs to cross-link
+- `docs/user-guide/etcd-metadata-exchange.md` — ETCD metadata exchange guide (link target for NB-05)
+- `docs/user-guide/backends/ucx.md` — UCX backend page
+- `docs/user-guide/backends/gds.md` — GDS backend page
+- `docs/user-guide/backends/obj.md` — OBJ (S3) backend page
+- `docs/user-guide/backends/posix.md` — POSIX backend page
+- `docs/user-guide/backends/gpunetio.md` — GPUNETIO backend page
+- `docs/user-guide/backends/mooncake.md` — Mooncake backend page
+- `docs/user-guide/backends/libfabric.md` — Libfabric backend page
+- `docs/user-guide/backends/hf3fs.md` — HF3FS backend page
+- `docs/user-guide/backends/gusli.md` — GUSLI backend page
+- `docs/user-guide/backends/azure-blob.md` — Azure Blob backend page
+- `docs/user-guide/backends/gds-mt.md` — GDS_MT backend page
+
+### Doc patterns
+- `docs/user-guide/building-nixl/docker.md` — Reference for `` and `` callout usage in Fern MDX
+
+### Requirements
+- `.planning/REQUIREMENTS.md` — NB-03 (usage guide), NB-04 (troubleshooting), NB-05 (ETCD callout + link), NB-06 (backend cross-links)
+
+
+
+
+## Existing Code Insights
+
+### Reusable Assets
+- `` Fern component — used in existing docs for important callouts; use for the 60-second ETCD join window
+- `` Fern component — available but not needed for this page (no Docker/native split on usage)
+- Backend User Guide pages all follow the same pattern — consistent link targets
+
+### Established Patterns
+- All backend pages at `docs/user-guide/backends/` use consistent frontmatter and structure
+- ETCD metadata exchange page exists as a standalone User Guide page — link to it, don't duplicate
+- First-mention inline linking for backend names (carried from Phase 33 D-09)
+- Fern frontmatter `title:` + `description:` on all pages
+
+### Integration Points
+- `usage.md` replaces the stub file created in Phase 32 at `docs/development/benchmarking/nixlbench/usage.md`
+- No changes to `docs/index.yml` needed (usage.md entry already exists from Phase 32)
+
+
+
+
+## Specific Ideas
+
+- User wants the 4 communication patterns to be the centerpiece of the usage examples, not per-backend examples
+- ETCD section should be a brief callout + link, not a full setup guide
+- Troubleshooting should cover the ROADMAP-required 4 failure modes at minimum
+
+
+
+
+## Deferred Ideas
+
+- Full CLI reference (70+ flags) — out of scope for v1.1 per REQUIREMENTS.md
+- Per-backend example pages — deferred per REQUIREMENTS.md
+- Performance tuning guide (CPU affinity, network tuning) — not in ROADMAP scope
+- Config file format documentation — Claude's discretion whether to include a brief mention
+
+
+
+---
+
+*Phase: 34-nixlbench-usage-troubleshooting-and-cross-references*
+*Context gathered: 2026-04-07*
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-DISCUSSION-LOG.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-DISCUSSION-LOG.md
new file mode 100644
index 0000000000..0d088a5486
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-DISCUSSION-LOG.md
@@ -0,0 +1,88 @@
+# Phase 34: NIXLBench Usage, Troubleshooting, and Cross-References - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-04-07
+**Phase:** 34-NIXLBench Usage, Troubleshooting, and Cross-References
+**Areas discussed:** Page split, Usage content scope, Troubleshooting scope, Backend examples
+
+---
+
+## Page Split
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Combined usage.md | Usage + troubleshooting in one page, troubleshooting as section at bottom | ✓ |
+| Separate troubleshooting.md | usage.md for guide, troubleshooting.md for failure modes | |
+| You decide | Claude picks based on content volume | |
+
+**User's choice:** Combined usage.md
+**Notes:** Keeps NIXLBench nav at 3 pages (index.md, build.md, usage.md)
+
+---
+
+## Usage Content Scope — CLI Depth
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Core flags only | ~10-15 essential flags in a single table | |
+| Grouped flag tables | Core + memory/transfer + performance (3 tables, ~30 flags) | |
+| You decide | Claude determines right level for 4 communication patterns | ✓ |
+
+**User's choice:** You decide
+**Notes:** None
+
+---
+
+## Usage Content Scope — ETCD
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Warning callout + link | for 60s barrier, link to ETCD page, brief Docker setup | ✓ |
+| Full ETCD section | Dedicated subsection covering setup, required vs optional | |
+| You decide | Claude balances based on NB-05 | |
+
+**User's choice:** Warning callout + link
+**Notes:** Matches NB-05 requirement
+
+---
+
+## Troubleshooting Scope
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| ROADMAP 4 + runtime essentials | 4 required modes + library-not-found + ETCD cleanup | |
+| ROADMAP 4 only | Strict 4 failure modes from success criteria | |
+| You decide | Claude picks based on developer needs | ✓ |
+
+**User's choice:** You decide
+**Notes:** None
+
+---
+
+## Backend Examples
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Pattern-focused examples | 4 patterns using UCX + 1-2 storage examples (GDS, OBJ) | ✓ |
+| Per-backend examples | Brief example for each major backend | |
+| You decide | Claude picks approach | |
+
+**User's choice:** Pattern-focused examples
+**Notes:** Link to backend User Guide pages for backend-specific flags
+
+---
+
+## Claude's Discretion
+
+- CLI flag table scope
+- Runtime issues in troubleshooting (ETCD cleanup, library-not-found)
+- Config file example inclusion
+- NVSHMEM worker type mention
+
+## Deferred Ideas
+
+- Full CLI reference (70+ flags) — out of scope for v1.1
+- Per-backend example pages — deferred
+- Performance tuning guide — not in ROADMAP scope
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-RESEARCH.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-RESEARCH.md
new file mode 100644
index 0000000000..fe803bfbab
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-RESEARCH.md
@@ -0,0 +1,344 @@
+# Phase 34: NIXLBench Usage, Troubleshooting, and Cross-References - Research
+
+**Researched:** 2026-04-07
+**Domain:** Technical documentation (Fern MDX) -- NIXLBench usage guide and troubleshooting
+**Confidence:** HIGH
+
+## Summary
+
+Phase 34 authors the NIXLBench usage and troubleshooting page (`usage.md`), replacing the stub created in Phase 32. All source material lives in `benchmark/nixlbench/README.md` and is well-structured. The page consolidates usage guide content (ETCD coordination, communication patterns, CLI options, output interpretation) and troubleshooting (four required failure modes) into a single page per user decision D-01. Cross-linking to backend User Guide pages and the ETCD metadata exchange page completes the requirements.
+
+The established Fern MDX patterns from Phases 32-33 (frontmatter, `` callouts, inline backend links on first mention) carry forward unchanged. No new libraries, tools, or infrastructure are needed -- this is purely a documentation authoring task.
+
+**Primary recommendation:** Author `usage.md` as a single page with four sections (ETCD Coordination, Communication Patterns, CLI Options, Troubleshooting), drawing examples directly from the README source material and linking every backend name to its User Guide page on first mention.
+
+
+## User Constraints (from CONTEXT.md)
+
+### Locked Decisions
+- **D-01:** Everything goes in a single `usage.md` page -- usage guide at the top, troubleshooting section at the bottom. No separate `troubleshooting.md`. This keeps the NIXLBench nav at 3 pages (index.md, build.md, usage.md).
+- **D-02:** Page sections (in order): ETCD Coordination (brief), Communication Patterns (with examples), CLI Options (essential flags), Troubleshooting.
+- **D-03:** Use a `` callout for the 60-second join window barrier. Link to the existing "Metadata Exchange with ETCD" User Guide page (`docs/user-guide/etcd-metadata-exchange.md`). Brief Docker one-liner for starting ETCD. Do NOT re-explain ETCD setup in detail.
+- **D-04:** Mention when ETCD is required vs optional (network backends require it; storage backends can run without it for single instances). Keep this concise -- 2-3 sentences.
+- **D-05:** Focus examples on the 4 communication patterns (pairwise, many-to-one, one-to-many, TP) using UCX as the default backend. Show the `--scheme` flag variations with brief explanation of each pattern.
+- **D-06:** Add 1-2 storage backend examples (GDS for local storage, OBJ for S3) to demonstrate how storage benchmarks differ from network benchmarks. Link to backend User Guide pages for backend-specific flags.
+- **D-07:** Show multi-node examples demonstrating initiator/target worker launching (two `nixlbench` commands on separate hosts pointing to the same ETCD server).
+- **D-08:** Claude's discretion on how many CLI flags to document. Should include enough for a developer to run all 4 communication patterns and basic storage benchmarks.
+- **D-09:** Cover the 4 ROADMAP-required failure modes: ETCD connection failures, CUDA/GPU not found, backend library missing, build failures.
+- **D-10:** Claude's discretion on whether to add runtime essentials (library-not-found errors, ETCD cleanup after failed runs).
+- **D-11:** Link every backend name (UCX, GDS, GDS_MT, POSIX, GPUNETIO, Mooncake, HF3FS, OBJ, GUSLI, Azure Blob) to its corresponding User Guide backend page on first mention per page.
+- **D-12:** Link to "Metadata Exchange with ETCD" when ETCD coordination is discussed.
+
+### Claude's Discretion
+- Exact CLI flag table scope (core only vs core + memory/transfer)
+- Whether to include ETCD cleanup and library-not-found in troubleshooting
+- Ordering of troubleshooting items
+- Whether to include a config file example (TOML format)
+- Whether to mention NVSHMEM worker type or keep focus on the default NIXL worker
+
+### Deferred Ideas (OUT OF SCOPE)
+- Full CLI reference (70+ flags) -- out of scope for v1.1
+- Per-backend example pages -- deferred
+- Performance tuning guide (CPU affinity, network tuning) -- not in ROADMAP scope
+- Config file format documentation -- Claude's discretion whether to include a brief mention
+
+
+
+## Phase Requirements
+
+| ID | Description | Research Support |
+|----|-------------|------------------|
+| NB-03 | Usage guide covers launching workers (initiator/target), ETCD coordination setup, communication patterns (pairwise/many-to-one/one-to-many/TP), and reading benchmark output | README lines 390-603 provide complete source material for all four patterns, worker launching, and ETCD coordination |
+| NB-04 | Troubleshooting covers ETCD connection failures, CUDA/GPU not found, backend library missing, build failures | README lines 817-915 provide troubleshooting content for all four failure modes |
+| NB-05 | `` callout for ETCD 60-second join window barrier; link to "Metadata Exchange with ETCD" page | ETCD metadata exchange page confirmed at `docs/user-guide/etcd-metadata-exchange.md`; `` component verified in use across docs |
+| NB-06 | Every backend name links to its corresponding User Guide backend page on first mention | All 12 backend pages confirmed at `docs/user-guide/backends/` (ucx, gds, gds-mt, posix, gpunetio, mooncake, hf3fs, obj, gusli, azure-blob, libfabric, uccl) |
+
+
+## Architecture Patterns
+
+### Page Structure (per D-01 and D-02)
+
+The single `usage.md` page follows this structure:
+
+```
+---
+title: NIXLBench Usage and Troubleshooting
+description: How to run NIXLBench benchmarks and troubleshoot common issues.
+---
+
+[Intro paragraph -- what this page covers]
+
+## etcd Coordination
+ - When ETCD is required vs optional (D-04)
+ - Docker one-liner (D-03)
+ - for 60-second barrier (D-03)
+ - Link to etcd-metadata-exchange page (D-12)
+
+## Communication Patterns
+ - Pairwise (D-05)
+ - Many-to-one (D-05)
+ - One-to-many (D-05)
+ - TP (tensor parallel) (D-05)
+ - Multi-node example (D-07)
+
+## Storage Backend Examples
+ - GDS example (D-06)
+ - OBJ (S3) example (D-06)
+
+## CLI Options
+ - Core Configuration table
+ - Memory and Transfer table
+ - (Claude's discretion on scope)
+
+## Troubleshooting
+ - etcd connection failures (D-09)
+ - CUDA/GPU not found (D-09)
+ - Backend library missing (D-09)
+ - Build failures (D-09)
+ - [Optional: ETCD cleanup, library-not-found] (D-10)
+```
+
+### Fern MDX Patterns (established in Phase 33)
+
+- **Frontmatter:** `title:` + `description:` on every page [VERIFIED: existing docs]
+- **Callouts:** `` for critical warnings, `` for supplementary info, `` for helpful hints [VERIFIED: docs/api-reference/*.md, docs/user-guide/etcd-metadata-exchange.md]
+- **Inline links:** Fern path format `/docs/user-guide/backends/ucx` (not relative paths) [VERIFIED: docs/development/benchmarking/nixlbench/index.md]
+- **Code blocks:** Standard triple-backtick with language identifier [VERIFIED: existing docs]
+- **Backend linking convention (Phase 33 D-09):** First mention of each backend name per page gets an inline link to its User Guide page [VERIFIED: index.md]
+
+### Backend Link Map
+
+All links verified against filesystem at `docs/user-guide/backends/`:
+
+| Backend Name in Prose | Link Target |
+|----------------------|-------------|
+| UCX | `/docs/user-guide/backends/ucx` |
+| GPUDirect Storage (GDS) | `/docs/user-guide/backends/gds` |
+| GPUDirect Storage MT (GDS_MT) | `/docs/user-guide/backends/gds-mt` |
+| POSIX | `/docs/user-guide/backends/posix` |
+| DOCA GPUNetIO | `/docs/user-guide/backends/gpunetio` |
+| Mooncake | `/docs/user-guide/backends/mooncake` |
+| Libfabric | `/docs/user-guide/backends/libfabric` |
+| HF3FS | `/docs/user-guide/backends/hf3fs` |
+| OBJ | `/docs/user-guide/backends/obj` |
+| Azure Blob | `/docs/user-guide/backends/azure-blob` |
+| GUSLI | `/docs/user-guide/backends/gusli` |
+
+[VERIFIED: `ls docs/user-guide/backends/` confirms all 12 files exist]
+
+### Terminology (QS-01 compliance)
+
+- Use `plug-in` (not `plugin`) in prose [VERIFIED: REQUIREMENTS.md QS-01]
+- Use `etcd` (lowercase) in prose, `ETCD` only in CLI flag names like `--etcd_endpoints` [VERIFIED: existing docs pattern in etcd-metadata-exchange.md]
+- Backend names follow their canonical capitalization: UCX, GDS, GDS_MT, POSIX, GPUNETIO, OBJ, GUSLI, HF3FS [VERIFIED: README source]
+
+## Don't Hand-Roll
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| ETCD setup instructions | Duplicate the etcd-metadata-exchange page content | Link to `/docs/user-guide/etcd-metadata-exchange` | Per D-03 and QS-02 (no duplicated content) |
+| Backend-specific flag tables | Exhaustive per-backend CLI reference | Brief example + link to backend User Guide page | Per D-06 and deferred scope (full CLI reference out of scope) |
+| Build troubleshooting | Repeat build.md content | Link to build page for build-specific setup | QS-02 compliance |
+
+## Source Material Mapping
+
+This maps README source sections to page sections, so the implementer knows exactly where to extract content from.
+
+| Page Section | README Source | Lines | Adaptation Notes |
+|-------------|-------------|-------|------------------|
+| etcd Coordination | "ETCD Coordination Setup" + "Using ETCD for Coordination" | 392-603 | Condense to Docker one-liner + required/optional rules + 60s barrier warning |
+| Communication Patterns | "Basic Usage Examples" + scheme flag from CLI Options | 415-429, 447 | Restructure around 4 patterns with `--scheme` flag; add multi-node example from lines 596-603 |
+| Storage Backend Examples | Backend-Specific Examples (GDS, OBJ) | 628-753 | Pick 1 GDS + 1 OBJ example; link to backend pages for rest |
+| CLI Options | "Command Line Options" (Core + Memory/Transfer sections) | 432-470 | Tables for core config + memory/transfer; skip per-backend flag tables |
+| Troubleshooting: ETCD | "ETCD Cleanup" + connection context | 911-915 | Frame as "etcd connection failures" with cleanup command |
+| Troubleshooting: CUDA | "CUDA Not Found" + "GPU Access Issues" | 822-893 | Combine into single CUDA/GPU troubleshooting entry |
+| Troubleshooting: Backend lib | "Library Not Found Errors" | 874-880 | `ldconfig` + `ldd` commands |
+| Troubleshooting: Build | "UCX Build Failures" + "etcd-cpp-api Build Issues" + "Docker Build Failures" | 836-869 | Combine build failures into single section |
+
+## Common Pitfalls
+
+### Pitfall 1: Duplicating ETCD Setup Content
+**What goes wrong:** Writing a full ETCD setup guide in usage.md that duplicates etcd-metadata-exchange.md
+**Why it happens:** The README has extensive ETCD content that feels natural to include
+**How to avoid:** Per D-03, only include Docker one-liner + 60s barrier `` + link. No configuration details.
+**Warning signs:** Usage.md ETCD section exceeds 15-20 lines
+
+### Pitfall 2: Exhaustive CLI Reference
+**What goes wrong:** Documenting all 70+ CLI flags when only core + memory/transfer flags are needed
+**Why it happens:** README has comprehensive flag list that's easy to copy wholesale
+**How to avoid:** Per D-08, include only flags needed for the 4 communication patterns + basic storage benchmarks. Per-backend flags are out of scope.
+**Warning signs:** CLI Options section has more than 2-3 tables
+
+### Pitfall 3: Using "ETCD" Instead of "etcd" in Prose
+**What goes wrong:** Inconsistent capitalization fails QS-01
+**Why it happens:** The README uses "ETCD" throughout
+**How to avoid:** Use `etcd` in prose text, `ETCD` only when referring to CLI flag values (e.g., `--runtime_type ETCD`)
+**Warning signs:** Uppercase "ETCD" appearing in descriptive text rather than code contexts
+
+### Pitfall 4: Missing Backend Links on First Mention
+**What goes wrong:** A backend name appears in prose without a link, failing NB-06
+**Why it happens:** Backends mentioned in code blocks or passing references get overlooked
+**How to avoid:** Audit every backend name in prose (not code blocks) and link the first occurrence per page
+**Warning signs:** Backend name in prose text without `[Name](/docs/...)` wrapper
+
+### Pitfall 5: Relative vs Fern Path Links
+**What goes wrong:** Using relative markdown links (`../user-guide/backends/ucx.md`) instead of Fern paths (`/docs/user-guide/backends/ucx`)
+**Why it happens:** Habit from standard markdown
+**How to avoid:** All cross-page links use Fern absolute paths starting with `/docs/`
+**Warning signs:** Links containing `../` or `.md` extensions
+
+## Code Examples
+
+### Warning Callout for 60-Second Barrier (verified Fern pattern)
+
+```markdown
+
+All NIXLBench workers in a benchmark group must connect to etcd within 60 seconds of the first worker joining. Workers that miss this window cause the barrier to fail and the benchmark to abort. For etcd setup and configuration details, see [Metadata Exchange with etcd](/docs/user-guide/etcd-metadata-exchange).
+
+```
+
+Source: Fern callout syntax verified from `docs/api-reference/python-api.md` and `docs/user-guide/etcd-metadata-exchange.md` [VERIFIED: codebase grep]
+
+### Backend First-Mention Link Pattern (from Phase 33 index.md)
+
+```markdown
+NIXLBench supports network backends such as [UCX](/docs/user-guide/backends/ucx),
+[Libfabric](/docs/user-guide/backends/libfabric), [Mooncake](/docs/user-guide/backends/mooncake),
+and [DOCA GPUNetIO](/docs/user-guide/backends/gpunetio), as well as storage backends including
+[GPUDirect Storage](/docs/user-guide/backends/gds) and [OBJ](/docs/user-guide/backends/obj).
+```
+
+Source: Pattern established in `docs/development/benchmarking/nixlbench/index.md` [VERIFIED: codebase]
+
+### Communication Pattern Example Structure
+
+```markdown
+### Pairwise
+
+Pairwise transfers data between matched pairs of initiators and targets. This is the default scheme.
+
+On host 1 (initiator):
+
+```bash
+nixlbench --etcd_endpoints http://etcd-server:2379 --backend UCX \
+ --initiator_seg_type VRAM --target_seg_type VRAM --scheme pairwise
+```
+
+On host 2 (target):
+
+```bash
+nixlbench --etcd_endpoints http://etcd-server:2379 --backend UCX \
+ --initiator_seg_type VRAM --target_seg_type VRAM --scheme pairwise
+```
+```
+
+Source: Derived from README lines 596-603 and CLI flag `--scheme` at line 447 [VERIFIED: README]
+
+### Troubleshooting Entry Structure
+
+```markdown
+### etcd Connection Failures
+
+**Symptoms:** Workers fail to join the benchmark group, or the barrier times out after 60 seconds.
+
+**Resolution:**
+
+1. Verify the etcd server is running and reachable:
+
+ ```bash
+ ETCDCTL_API=3 etcdctl endpoint health --endpoints=http://etcd-server:2379
+ ```
+
+2. If a previous run failed, clean up stale keys before retrying:
+
+ ```bash
+ ETCDCTL_API=3 etcdctl del "xferbench" --prefix=true
+ ```
+```
+
+Source: README lines 911-915 for cleanup command [VERIFIED: README]
+
+## Discretion Recommendations
+
+For items marked as Claude's discretion in the CONTEXT.md:
+
+### CLI Flag Scope: Core + Memory/Transfer (recommended)
+Include two flag tables: Core Configuration (6 flags) and Memory and Transfer Configuration (12 flags). Skip Performance/Threading and Device/Network tables -- they are advanced tuning. This covers everything needed for the 4 communication patterns and basic storage use. [ASSUMED]
+
+### Include ETCD Cleanup and Library-Not-Found (recommended)
+These are the two most common runtime issues per D-10. The ETCD cleanup command (`etcdctl del "xferbench" --prefix=true`) is critical for developers hitting stale state after crashes. Library-not-found (`ldconfig` + `ldd`) covers the "backend library missing" requirement naturally. Both fit in 10-15 lines total. [ASSUMED]
+
+### Troubleshooting Order (recommended)
+Order by likelihood of encounter: (1) etcd connection failures -- most common first-run issue, (2) Build failures -- second most common for native builds, (3) CUDA/GPU not found, (4) Backend library missing. Then optional: ETCD cleanup, library-not-found. [ASSUMED]
+
+### Config File: Brief Mention Only (recommended)
+Include a 3-4 sentence note that `--config_file` accepts TOML files, with a minimal example. This is useful without being a full reference. [ASSUMED]
+
+### NVSHMEM Worker: Brief Mention (recommended)
+Include a 2-sentence note about `--worker_type nvshmem` for GPU-only VRAM transfers. Do not expand into a full section. [ASSUMED]
+
+## Validation Architecture
+
+### Test Framework
+| Property | Value |
+|----------|-------|
+| Framework | Fern docs build check |
+| Config file | `fern/fern.config.json` |
+| Quick run command | `cd fern && fern check` |
+| Full suite command | `cd fern && fern check` |
+
+### Phase Requirements to Test Map
+| Req ID | Behavior | Test Type | Automated Command | File Exists? |
+|--------|----------|-----------|-------------------|-------------|
+| NB-03 | Usage guide covers workers, ETCD, patterns, output | manual review + fern check | `cd fern && fern check` | stub exists at `docs/development/benchmarking/nixlbench/usage.md` |
+| NB-04 | Troubleshooting covers 4 failure modes | manual review | grep for section headers in usage.md | stub exists |
+| NB-05 | `` callout for 60s barrier + ETCD link | manual review + fern check | grep for `` and `/docs/user-guide/etcd-metadata-exchange` in usage.md | stub exists |
+| NB-06 | Backend names link to User Guide pages | manual review | grep for backend link patterns in usage.md | stub exists |
+
+### Sampling Rate
+- **Per task commit:** `cd fern && fern check`
+- **Per wave merge:** `cd fern && fern check` + manual link verification
+- **Phase gate:** `fern check` green + all 4 requirement checks pass
+
+### Wave 0 Gaps
+None -- existing test infrastructure (fern check) covers all phase requirements.
+
+## Assumptions Log
+
+| # | Claim | Section | Risk if Wrong |
+|---|-------|---------|---------------|
+| A1 | Core + Memory/Transfer tables (18 flags) is the right scope for CLI Options | Discretion Recommendations | Page either too sparse or too bloated -- low risk, easy to adjust |
+| A2 | ETCD cleanup and library-not-found should be included | Discretion Recommendations | Page slightly longer than needed -- very low risk |
+| A3 | Troubleshooting order by likelihood is correct | Discretion Recommendations | Negligible -- just reordering |
+| A4 | Brief config file mention is sufficient | Discretion Recommendations | Low risk -- can expand later |
+| A5 | Brief NVSHMEM mention is sufficient | Discretion Recommendations | Low risk -- NVSHMEM is niche |
+
+## Open Questions
+
+None -- all source material is available in the README, all target link pages exist, and Fern MDX patterns are well established from Phases 32-33.
+
+## Sources
+
+### Primary (HIGH confidence)
+- `benchmark/nixlbench/README.md` -- Complete source material for usage, CLI flags, ETCD coordination, troubleshooting
+- `docs/development/benchmarking/nixlbench/index.md` -- Established backend linking pattern from Phase 33
+- `docs/development/benchmarking/nixlbench/build.md` -- Established Fern MDX patterns from Phase 33
+- `docs/user-guide/etcd-metadata-exchange.md` -- Confirmed link target for ETCD cross-reference
+- `docs/user-guide/backends/` -- All 12 backend pages confirmed present
+- `docs/api-reference/python-api.md` -- `` callout syntax verified
+
+### Secondary (MEDIUM confidence)
+- None needed -- all claims verified against codebase
+
+### Tertiary (LOW confidence)
+- None
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH -- purely documentation, no libraries needed
+- Architecture: HIGH -- page structure locked by user decisions, Fern patterns verified
+- Pitfalls: HIGH -- patterns and anti-patterns observed directly from codebase and requirements
+
+**Research date:** 2026-04-07
+**Valid until:** 2026-05-07 (stable -- documentation patterns unlikely to change)
diff --git a/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-VERIFICATION.md b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-VERIFICATION.md
new file mode 100644
index 0000000000..4826aee5a3
--- /dev/null
+++ b/.planning/phases/34-nixlbench-usage-troubleshooting-and-cross-references/34-VERIFICATION.md
@@ -0,0 +1,98 @@
+---
+phase: 34-nixlbench-usage-troubleshooting-and-cross-references
+verified: 2026-04-07T22:30:00Z
+status: human_needed
+score: 7/7 must-haves verified
+overrides_applied: 0
+re_verification:
+ previous_status: gaps_found
+ previous_score: 6/7
+ gaps_closed:
+ - "Usage guide covers reading benchmark output (NB-03, Roadmap SC #1)"
+ gaps_remaining: []
+ regressions: []
+human_verification:
+ - test: "Verify all Fern cross-links resolve to real pages"
+ expected: "Clicking backend links and etcd link navigates to the correct User Guide pages"
+ why_human: "Link target existence was confirmed at file level but rendered navigation requires a running Fern dev server"
+---
+
+# Phase 34: NIXLBench Usage, Troubleshooting, and Cross-References Verification Report
+
+**Phase Goal:** Developers can learn how to run NIXLBench end-to-end -- launch workers, coordinate with etcd, interpret output -- and find help for common failures, with all backends and etcd linked to their respective documentation pages
+**Verified:** 2026-04-07T22:30:00Z
+**Status:** human_needed
+**Re-verification:** Yes -- after gap closure (gap closed successfully by plan 34-02)
+
+## Goal Achievement
+
+### Observable Truths
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | Developer can learn how to launch initiator and target NIXLBench workers with etcd coordination | VERIFIED | etcd Coordination section (line 8) with Docker one-liner, pairwise example with "On host 1 (initiator)" / "On host 2 (target)" format |
+| 2 | Developer can see examples of all four communication patterns (pairwise, many-to-one, one-to-many, TP) | VERIFIED | Four subsections under Communication Patterns with --scheme pairwise (lines 46, 56), manytoone (68), onetomany (82), tp (96) |
+| 3 | Developer can find storage backend examples (GDS and OBJ) with links to backend pages | VERIFIED | GDS example with --backend GDS (line 110), OBJ example with --backend OBJ (line 118), both link to User Guide pages |
+| 4 | Developer can look up essential CLI flags in organized tables | VERIFIED | Core Configuration (6 flags, line 129) and Memory/Transfer Configuration (12 flags, line 140) tables with Flag/Description/Default columns |
+| 5 | Developer can troubleshoot etcd connection failures, CUDA/GPU not found, backend library missing, and build failures | VERIFIED | Four subsections under Troubleshooting (lines 188, 210, 235, 255) with Symptoms/Resolution format |
+| 6 | Every backend name mentioned in prose links to its User Guide page on first mention | VERIFIED | Line 10 links all 10 backends (UCX, DOCA GPUNetIO, Mooncake, Libfabric, GDS, GDS_MT, POSIX, HF3FS, OBJ, GUSLI) to /docs/user-guide/backends/ paths |
+| 7 | Usage guide covers reading benchmark output (NB-03, Roadmap SC #1) | VERIFIED | "Reading Benchmark Output" section at line 161 with 12-column description table (Block Size through P99 Tx), Note callout for multi-worker pairwise extra columns, and latency phase explanation |
+
+**Score:** 7/7 truths verified
+
+### Required Artifacts
+
+| Artifact | Expected | Status | Details |
+|----------|----------|--------|---------|
+| `docs/development/benchmarking/nixlbench/usage.md` | NIXLBench usage guide and troubleshooting page | VERIFIED | 273 lines, complete Fern MDX with frontmatter, not a stub |
+
+### Key Link Verification
+
+| From | To | Via | Status | Details |
+|------|----|-----|--------|---------|
+| usage.md | /docs/user-guide/etcd-metadata-exchange | Inline link in etcd Coordination section | WIRED | Found on lines 10 and 28 |
+| usage.md | /docs/user-guide/backends/ucx | First-mention backend link | WIRED | Found on line 10 |
+| usage.md | /docs/user-guide/backends/gds | First-mention backend link | WIRED | Found on lines 10 and 107 |
+| usage.md | /docs/user-guide/backends/obj | First-mention backend link | WIRED | Found on lines 10 and 115 |
+
+### Data-Flow Trace (Level 4)
+
+Not applicable -- static documentation page, no dynamic data rendering.
+
+### Behavioral Spot-Checks
+
+Step 7b: SKIPPED (documentation-only phase, no runnable entry points).
+
+### Requirements Coverage
+
+| Requirement | Source Plan | Description | Status | Evidence |
+|-------------|------------|-------------|--------|----------|
+| NB-03 | 34-01, 34-02 | Usage guide covers launching workers, etcd coordination, communication patterns, and reading benchmark output | SATISFIED | All items covered: etcd coordination (line 8), four communication patterns (lines 35-99), reading benchmark output (lines 161-184) |
+| NB-04 | 34-01 | Troubleshooting covers etcd connection failures, CUDA/GPU not found, backend library missing, build failures | SATISFIED | Four troubleshooting subsections with Symptoms/Resolution format (lines 188-273) |
+| NB-05 | 34-01 | Warning callout for 60-second etcd join window barrier with link to etcd metadata exchange page | SATISFIED | Warning callout on lines 27-29 with "60 seconds" text and link to /docs/user-guide/etcd-metadata-exchange |
+| NB-06 | 34-01 | Every backend name links to User Guide backend page on first mention | SATISFIED | 10 backend links on line 10, all link targets use correct /docs/user-guide/backends/ paths |
+
+### Anti-Patterns Found
+
+| File | Line | Pattern | Severity | Impact |
+|------|------|---------|----------|--------|
+| (none) | - | - | - | No anti-patterns found |
+
+### Human Verification Required
+
+### 1. Cross-Link Navigation
+
+**Test:** Open usage.md in the Fern dev server and click each backend link and the etcd link
+**Expected:** Each link navigates to the correct User Guide page without 404 errors
+**Why human:** File-level existence checks confirm target files exist, but rendered Fern navigation depends on docs/index.yml routing which requires a running server to validate
+
+### Gaps Summary
+
+No gaps. The single gap from the previous verification ("Reading Benchmark Output" section missing) has been closed by plan 34-02. The section now exists at line 161 with a 12-column description table, a Note callout for multi-worker pairwise extra columns, and a latency phase explanation paragraph.
+
+**Note on troubleshooting.md vs usage.md:** Roadmap SC #3 references `troubleshooting.md` as a separate file, but user decision D-01 explicitly combined troubleshooting into `usage.md`. The troubleshooting content is complete and present; only the file name differs from the roadmap wording. This is an intentional deviation per the user's own decision and does not constitute a gap.
+
+---
+
+_Verified: 2026-04-07T22:30:00Z_
+_Verifier: Claude (gsd-verifier)_
diff --git a/.planning/phases/35-kvbench-overview-and-build/35-01-PLAN.md b/.planning/phases/35-kvbench-overview-and-build/35-01-PLAN.md
new file mode 100644
index 0000000000..64112829f9
--- /dev/null
+++ b/.planning/phases/35-kvbench-overview-and-build/35-01-PLAN.md
@@ -0,0 +1,263 @@
+---
+phase: 35-kvbench-overview-and-build
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+ - docs/index.yml
+ - docs/development/benchmarking/kvbench/index.md
+ - docs/development/benchmarking/kvbench/build.md
+autonomous: true
+requirements: [KB-01, KB-02]
+must_haves:
+ truths:
+ - "KVBench overview page states in its first paragraph that profile invokes nixlbench as a subprocess"
+ - "KVBench overview page explains two command categories: KVBench commands (plan, profile, kvcache) and CTP commands (ct-perftest, sequential-ct-perftest)"
+ - "KVBench build page presents Docker and Python venv paths in a Tabs component"
+ - "Docker tab links to NIXLBench build page instead of duplicating Docker build steps"
+ - "Python venv tab shows self-contained venv + uv install steps"
+ - "Both pages have valid Fern MDX frontmatter with title and description"
+ - "docs/index.yml includes build.md entry under KVBench section"
+ - "fern check passes with zero errors"
+ artifacts:
+ - path: "docs/development/benchmarking/kvbench/index.md"
+ provides: "KVBench overview with nixlbench dependency and command categories"
+ contains: "nixlbench"
+ - path: "docs/development/benchmarking/kvbench/build.md"
+ provides: "KVBench build instructions with Docker and Python venv tabs"
+ contains: ""
+ - path: "docs/index.yml"
+ provides: "Navigation entry for KVBench build page"
+ contains: "Building KVBench"
+ key_links:
+ - from: "docs/development/benchmarking/kvbench/index.md"
+ to: "NIXLBench section"
+ via: "inline link in first paragraph"
+ pattern: "NIXLBench.*\\]\\("
+ - from: "docs/development/benchmarking/kvbench/build.md"
+ to: "NIXLBench build page"
+ via: "cross-link in Docker tab"
+ pattern: "nixlbench/build"
+---
+
+
+Author the KVBench overview page and build page as Fern-compatible MDX, and update the navigation config to include the new build page.
+
+Purpose: Developers understand that KVBench drives NIXLBench as a subprocess and know how to install KVBench using Docker or a Python virtual environment.
+Output: Two complete doc pages (index.md replaced, build.md created) and updated docs/index.yml.
+
+
+
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/workflows/execute-plan.md
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/templates/summary.md
+
+
+
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/35-kvbench-overview-and-build/35-CONTEXT.md
+@.planning/phases/35-kvbench-overview-and-build/35-RESEARCH.md
+@.planning/phases/33-nixlbench-overview-and-build/33-01-SUMMARY.md
+
+
+From docs/development/benchmarking/nixlbench/index.md:
+- Frontmatter: title + description fields
+- First paragraph: problem-first narrative with inline cross-links
+- Features section: grouped bullet list with backend links on first mention
+- Next Steps section: links to sibling pages
+
+From docs/development/benchmarking/nixlbench/build.md:
+- Frontmatter: title + description fields
+- Tabs component: `` with `` and ``
+- Blank lines required after opening `` and before closing ` `
+- Cross-links use format: `/docs/user-guide/building-nixl` (absolute Fern doc paths)
+- Sibling page links use: `./nixlbench/build` format (relative to benchmarking section)
+
+From docs/index.yml (KVBench section, lines 79-84):
+```yaml
+ - section: KVBench
+ collapsed: open-by-default
+ path: development/benchmarking/kvbench/index.md
+ contents:
+ - page: Commands and Examples
+ path: development/benchmarking/kvbench/commands.md
+```
+Must add `build.md` entry BEFORE the `commands.md` entry.
+
+
+
+
+
+
+ Task 1: Update navigation and author KVBench overview page
+ docs/index.yml, docs/development/benchmarking/kvbench/index.md
+
+ - docs/index.yml (full file -- find the KVBench section around line 79)
+ - docs/development/benchmarking/kvbench/index.md (current stub to replace)
+ - docs/development/benchmarking/nixlbench/index.md (structural template)
+ - benchmark/kvbench/README.md (lines 1-80 -- source content for overview, command categories, supported models)
+
+
+**Step 1: Update docs/index.yml** (per D-01)
+
+Find the KVBench section (currently has only `commands.md` under contents). Add a `build.md` entry BEFORE the existing `commands.md` entry. The updated KVBench section should look like:
+
+```yaml
+ - section: KVBench
+ collapsed: open-by-default
+ path: development/benchmarking/kvbench/index.md
+ contents:
+ - page: Building KVBench
+ path: development/benchmarking/kvbench/build.md
+ - page: Commands and Examples
+ path: development/benchmarking/kvbench/commands.md
+```
+
+Preserve exact indentation matching the NIXLBench section above it (10 spaces for section, 12 for contents items, 14 for page/path). Do NOT modify any other part of the file.
+
+**Step 2: Replace docs/development/benchmarking/kvbench/index.md** (per D-02, D-03, D-04)
+
+Replace the Phase 32 stub with the full overview page. The file must have YAML frontmatter with `title: KVBench` and `description: A KV cache benchmarking utility that generates and runs NIXLBench commands to profile LLM inference transfer performance.`
+
+First paragraph (per D-02 -- MUST state nixlbench subprocess relationship in first paragraph): KVBench is a benchmarking utility that generates and runs [NIXLBench](./nixlbench) commands for profiling KV cache transfer performance across LLM architectures. The `profile` command invokes `nixlbench` as a subprocess, executing the transfer benchmarks that KVBench plans based on model architecture and access patterns.
+
+Command Categories section (per D-03 -- two subsections with brief one-line descriptions):
+
+`## Command Categories` heading, then:
+
+`### KVBench Commands` with these bullets:
+- **plan** -- Display the recommended NIXLBench configuration for a given model architecture and access pattern
+- **profile** -- Run NIXLBench with the planned configuration and collect performance results
+- **kvcache** -- Display KV cache layout information for a model architecture
+
+`### CTP Commands` with these bullets:
+- **ct-perftest** -- Run custom traffic performance tests using asymmetric transfer matrices
+- **sequential-ct-perftest** -- Run sequential custom traffic performance tests for ordered matrix evaluation
+
+`## Supported Models` section (per D-04 -- brief mention, no deep explanation):
+KVBench includes model architecture definitions for several LLM families: DeepSeek R1, Llama 3.1, and more. Link to `./kvbench/commands` for details on defining custom model architectures.
+
+`## Next Steps` section:
+- **[Building KVBench](./kvbench/build)** -- Docker and Python virtual environment installation
+- **[Commands and Examples](./kvbench/commands)** -- Full command reference and usage examples
+
+IMPORTANT: Use `./nixlbench` for cross-linking to the NIXLBench section (relative to the benchmarking section root), matching the pattern in the NIXLBench overview page which uses `./nixlbench/build`. Use `./kvbench/build` and `./kvbench/commands` for sibling page links, matching the same relative path convention.
+
+
+ cd /home/omrik/Projects/nixl.gitlab/fern && fern check 2>&1 | tail -5
+
+
+ - grep -q "Building KVBench" docs/index.yml (nav entry exists)
+ - grep -q "nixlbench" docs/development/benchmarking/kvbench/index.md (subprocess dependency mentioned)
+ - grep -q "profile.*nixlbench" docs/development/benchmarking/kvbench/index.md (profile invokes nixlbench)
+ - grep -q "KVBench Commands" docs/development/benchmarking/kvbench/index.md (command category present)
+ - grep -q "CTP Commands" docs/development/benchmarking/kvbench/index.md (command category present)
+ - grep -q "DeepSeek R1" docs/development/benchmarking/kvbench/index.md (model mention)
+ - grep -q "title: KVBench" docs/development/benchmarking/kvbench/index.md (frontmatter present)
+ - grep -q "description:" docs/development/benchmarking/kvbench/index.md (frontmatter present)
+ - The file does NOT contain HTML comments or bare anchor links
+
+
+
+
+ Task 2: Author KVBench build page with Docker and Python venv tabs
+ docs/development/benchmarking/kvbench/build.md
+
+ - docs/development/benchmarking/nixlbench/build.md (structural template -- especially the Tabs component pattern)
+ - benchmark/kvbench/README.md (lines 38-56 -- Docker and Python build steps)
+ - benchmark/kvbench/pyproject.toml (verify dependencies but use Python 3.12+ per D-08, NOT pyproject's >=3.10)
+
+
+Create docs/development/benchmarking/kvbench/build.md (per D-05, D-06, D-07, D-08).
+
+The file must have YAML frontmatter with `title: Building KVBench` and `description: Install KVBench using the NIXLBench Docker container or a Python virtual environment.`
+
+Opening line (per D-08 -- inline requirements, no separate section): KVBench requires Python 3.12 or later. For GPU-accelerated benchmarks, PyTorch is also required.
+
+Then an `## Installation` heading followed by a `` component (per D-05):
+
+Docker tab (``):
+Per D-06, the Docker tab MUST NOT duplicate any Docker build steps. Content should be:
+"KVBench is included in the NIXLBench Docker container. See [Building NIXLBench](./nixlbench/build) for Docker build and setup instructions."
+Then: "After building the container, KVBench is available at `/workspace/benchmark/kvbench/` inside the container."
+
+Python venv tab (``):
+Per D-07, self-contained with the full venv + uv install sequence from the README:
+
+ git clone https://github.com/ai-dynamo/nixl.git
+ cd nixl/benchmark/kvbench
+ python3 -m venv venv
+ source venv/bin/activate
+ pip install uv
+ uv sync --active
+
+Then a verification step: `python main.py --help`
+
+CRITICAL formatting rules for Tabs:
+- Blank line AFTER `` opening tag
+- Blank line BEFORE ` ` closing tag
+- Markdown inside tabs renders normally with these blank lines
+
+Per D-08: Python 3.12+ stated inline in the opening line. Do NOT create a System Requirements section. Do NOT use pyproject.toml's `>=3.10` -- use `3.12 or later` to match NIXLBench build page convention.
+
+Use `./nixlbench/build` for the cross-link to the NIXLBench build page (relative to benchmarking section root, matching existing cross-link conventions).
+
+
+ cd /home/omrik/Projects/nixl.gitlab/fern && fern check 2>&1 | tail -5
+
+
+ - grep -q "title: Building KVBench" docs/development/benchmarking/kvbench/build.md (frontmatter)
+ - grep -q "description:" docs/development/benchmarking/kvbench/build.md (frontmatter)
+ - grep -q "" docs/development/benchmarking/kvbench/build.md (tabs component)
+ - grep -q "Docker" docs/development/benchmarking/kvbench/build.md (Docker tab)
+ - grep -q "Python venv" docs/development/benchmarking/kvbench/build.md (venv tab)
+ - grep -q "nixlbench/build" docs/development/benchmarking/kvbench/build.md (cross-link to NIXLBench build)
+ - grep -q "uv sync" docs/development/benchmarking/kvbench/build.md (Python install steps)
+ - grep -q "3.12" docs/development/benchmarking/kvbench/build.md (correct Python version)
+ - The file does NOT mention Python 3.10
+ - The file does NOT contain a "## System Requirements" heading
+
+
+
+
+
+
+## Trust Boundaries
+
+| Boundary | Description |
+|----------|-------------|
+| N/A | Documentation-only phase -- no runtime code, no user input processing |
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-35-01 | I (Information Disclosure) | Documentation pages | accept | Pages document public open-source tooling; no secrets or internal details exposed |
+
+
+
+1. `cd fern && fern check` passes with zero errors
+2. `docs/index.yml` KVBench section has 3 entries: section root (index.md), Building KVBench (build.md), Commands and Examples (commands.md)
+3. KVBench index.md first paragraph mentions nixlbench subprocess dependency with link
+4. KVBench index.md has two command category subsections covering all 5 subcommands
+5. KVBench build.md uses Tabs component with Docker and Python venv tabs
+6. Docker tab cross-links to NIXLBench build page with no duplicated build steps
+7. Python venv tab contains complete venv + uv install steps
+8. Both pages have valid frontmatter (title + description)
+
+
+
+- fern check passes with zero errors on all KVBench pages
+- KVBench overview page satisfies KB-01: first paragraph states profile invokes nixlbench as subprocess, explains two command categories
+- KVBench build page satisfies KB-02: Docker and Python venv in Tabs component, Docker links to NIXLBench build page
+- Navigation updated with build.md entry under KVBench section
+- No content duplication -- Docker build steps are cross-linked, not repeated
+
+
+
diff --git a/.planning/phases/35-kvbench-overview-and-build/35-CONTEXT.md b/.planning/phases/35-kvbench-overview-and-build/35-CONTEXT.md
new file mode 100644
index 0000000000..7240c68784
--- /dev/null
+++ b/.planning/phases/35-kvbench-overview-and-build/35-CONTEXT.md
@@ -0,0 +1,114 @@
+# Phase 35: KVBench Overview and Build - Context
+
+**Gathered:** 2026-04-07
+**Status:** Ready for planning
+
+
+## Phase Boundary
+
+Author the KVBench overview page and build page as Fern-compatible MDX. The overview states that KVBench's `profile` command invokes `nixlbench` as a subprocess and explains the two command categories (KVBench commands vs CTP commands). The build page covers Docker (reusing NIXLBench container) and Python venv install. No command reference or examples — those are Phase 36.
+
+
+
+
+## Implementation Decisions
+
+### Page Structure
+- **D-01:** Separate pages: `index.md` (overview) and `build.md` (build instructions). Consistent with NIXLBench Phase 33 pattern. This updates Phase 32's nav structure — KVBench section now has 3 child pages (`index.md`, `build.md`, `commands.md`) instead of 2. Phase 32's `docs/index.yml` entries need updating to add a `build.md` entry.
+
+### Overview Content
+- **D-02:** The first paragraph states that KVBench generates and runs `nixlbench` commands, with an inline link to the NIXLBench section. This satisfies requirement KB-01 ("states in its first paragraph that profile invokes nixlbench as a subprocess").
+- **D-03:** The overview presents the two command categories (KVBench commands: plan/profile/kvcache and CTP commands: ct-perftest/sequential-ct-perftest) with brief one-line descriptions. Claude's discretion on visual layout (grouped bullets, two sections, etc.).
+- **D-04:** Mention supported LLM architectures (DeepSeek R1, Llama 3.1, and others) on the overview page as a feature highlight. No deep explanation — that's Phase 36.
+
+### Build Page
+- **D-05:** Use `` component for Docker vs Python venv build paths, consistent with ROADMAP KB-02.
+- **D-06:** Docker tab links to the NIXLBench build page rather than repeating Docker build steps. Brief note: "KVBench is included in the NIXLBench Docker container" + link to NIXLBench build page. No duplication.
+- **D-07:** Python venv tab is self-contained — shows the `venv` + `pip`/`uv` install steps directly since they're short and specific to KVBench.
+- **D-08:** No system requirements section on the build page (KVBench's only requirements are Python 3.12+ and optional PyTorch for GPU benchmarks — mention inline).
+
+### Claude's Discretion
+- Visual layout for the two command categories (grouped bullets vs two sections)
+- Whether to include a "Supported Models" subsection on overview or just a brief mention
+- Exact frontmatter `description:` text for each page
+- Whether to include a "Next steps" link at the bottom of overview pointing to build page
+
+
+
+
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Source material
+- `benchmark/kvbench/README.md` — Primary source for KVBench overview, build instructions, command descriptions, and examples
+- `benchmark/kvbench/README.md` §Overview (line ~26) — Two command categories explanation
+- `benchmark/kvbench/README.md` §Building (line ~38) — Docker and Python build steps
+- `benchmark/kvbench/pyproject.toml` — Python dependencies and project metadata
+
+### Model examples (for overview mention)
+- `benchmark/kvbench/examples/model_deepseek_r1.yaml` — DeepSeek R1 model architecture config
+- `benchmark/kvbench/examples/model_llama_3_1_70b.yaml` — Llama 3.1 70B model architecture config
+
+### Existing doc patterns
+- `docs/development/benchmarking/nixlbench/index.md` — NIXLBench overview page pattern (Phase 33 output)
+- `docs/development/benchmarking/nixlbench/build.md` — NIXLBench build page pattern (Phase 33 output)
+
+### Navigation config
+- `docs/index.yml` — Must be updated to add `build.md` entry under KVBench section (Phase 32 only declared `index.md` + `commands.md`)
+
+### Cross-link targets
+- NIXLBench section pages — link target for the subprocess dependency explanation
+- NIXLBench build page — link target for Docker build instructions
+
+### Requirements
+- `.planning/REQUIREMENTS.md` — KB-01 (overview with nixlbench dependency), KB-02 (build with Docker + venv Tabs)
+
+
+
+
+## Existing Code Insights
+
+### Reusable Assets
+- NIXLBench `index.md` and `build.md` (Phase 33 output) — Template for consistent page structure
+- `` Fern component — used in NIXLBench build page, reuse for Docker/Python tabs
+- Fern frontmatter pattern: `title:` + `description:` on all pages
+
+### Established Patterns
+- Cross-link to existing docs rather than duplicating (carried from Phase 33 D-05)
+- First-mention inline links for backend names (carried from Phase 33 D-09)
+- Problem-first narrative on overview pages (carried from Phase 33 D-02)
+- Rewrite README prose for Fern doc style (carried from Phase 33 D-07)
+
+### Integration Points
+- `docs/index.yml` — Must be updated to reflect 3 pages under KVBench instead of 2
+- New files: `docs/development/benchmarking/kvbench/index.md`, `docs/development/benchmarking/kvbench/build.md`
+- The stub `index.md` from Phase 32 will be replaced with real content
+
+
+
+
+## Specific Ideas
+
+- User wants parallel structure with NIXLBench (separate overview + build pages)
+- NIXLBench dependency stated naturally in first paragraph, not as a callout box
+- Docker build links to NIXLBench build page — zero duplication
+
+
+
+
+## Deferred Ideas
+
+- KVBench command reference (all 5 subcommands with CLI tables) — Phase 36
+- Model configuration YAML schema documentation — Phase 36
+- LLM architecture examples (DeepSeek R1, Llama 3.1) — Phase 36
+- CTP examples and matrix format documentation — Phase 36
+- KVBench GDS tutorial (`benchmark/kvbench/docs/tutorial-gds.md`) — P2, deferred per REQUIREMENTS.md
+- Adding new model architecture guide — P2, deferred per REQUIREMENTS.md
+
+
+
+---
+
+*Phase: 35-kvbench-overview-and-build*
+*Context gathered: 2026-04-07*
diff --git a/.planning/phases/35-kvbench-overview-and-build/35-DISCUSSION-LOG.md b/.planning/phases/35-kvbench-overview-and-build/35-DISCUSSION-LOG.md
new file mode 100644
index 0000000000..a06172ce3d
--- /dev/null
+++ b/.planning/phases/35-kvbench-overview-and-build/35-DISCUSSION-LOG.md
@@ -0,0 +1,74 @@
+# Phase 35: KVBench Overview and Build - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-04-07
+**Phase:** 35-KVBench Overview and Build
+**Areas discussed:** Page structure, NIXLBench dependency framing, Build content, Two command categories
+
+---
+
+## Page Structure
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Separate build.md | Consistent with NIXLBench: index.md + build.md + commands.md | ✓ |
+| Combined in index.md | Overview + build in one page since KVBench build is simpler | |
+| You decide | Claude picks based on content volume | |
+
+**User's choice:** Separate build.md
+**Notes:** Parallel structure with NIXLBench. Updates KVBench nav from 2 to 3 pages.
+
+---
+
+## NIXLBench Dependency Framing
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Inline first paragraph | Natural statement in first paragraph with link to NIXLBench section | ✓ |
+| Callout box | or callout box near the top | |
+| You decide | Claude picks approach | |
+
+**User's choice:** Inline first paragraph
+**Notes:** Matches KB-01 requirement exactly.
+
+---
+
+## Build Content
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Link to NIXLBench build | Brief note + link, no duplication | ✓ |
+| Repeat Docker steps | Copy Docker commands for self-containment | |
+| You decide | Claude picks | |
+
+**User's choice:** Link to NIXLBench build
+**Notes:** Consistent with cross-link philosophy from Phase 33.
+
+---
+
+## Two Command Categories
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Two-column or grouped bullets | Clear visual split: KVBench Commands and CTP Commands | |
+| Single list with labels | One list tagged [KVBench] or [CTP] | |
+| You decide | Claude picks presentation | ✓ |
+
+**User's choice:** You decide
+**Notes:** None
+
+---
+
+## Claude's Discretion
+
+- Visual layout for two command categories
+- Whether to include "Supported Models" subsection
+- Exact frontmatter description text
+- Whether to include "Next steps" link
+
+## Deferred Ideas
+
+- Command reference, model config, LLM examples — Phase 36
+- GDS tutorial, adding new model architecture — P2, deferred
diff --git a/.planning/phases/35-kvbench-overview-and-build/35-RESEARCH.md b/.planning/phases/35-kvbench-overview-and-build/35-RESEARCH.md
new file mode 100644
index 0000000000..cdd7b7abb4
--- /dev/null
+++ b/.planning/phases/35-kvbench-overview-and-build/35-RESEARCH.md
@@ -0,0 +1,343 @@
+# Phase 35: KVBench Overview and Build - Research
+
+**Researched:** 2026-04-07
+**Domain:** Fern MDX documentation authoring (KVBench overview and build pages)
+**Confidence:** HIGH
+
+## Summary
+
+Phase 35 creates two Fern-compatible MDX pages for KVBench: an overview page (`index.md`) and a build page (`build.md`). The source material is well-defined in `benchmark/kvbench/README.md`, and the output pattern is established by the Phase 33 NIXLBench pages. The primary work is content authoring -- rewriting README prose into Fern doc style, structuring the overview around the two command categories and NIXLBench subprocess relationship, and creating a tabbed Docker/Python venv build page.
+
+The navigation config (`docs/index.yml`) must be updated to add a `build.md` entry under the KVBench section. The existing Phase 32 stub `index.md` will be replaced with real content.
+
+**Primary recommendation:** Follow the NIXLBench page pattern exactly (frontmatter, cross-linking, Tabs component) and use `benchmark/kvbench/README.md` as the single source of truth for content.
+
+
+## User Constraints (from CONTEXT.md)
+
+### Locked Decisions
+- **D-01:** Separate pages: `index.md` (overview) and `build.md` (build instructions). Consistent with NIXLBench Phase 33 pattern. Phase 32's `docs/index.yml` entries need updating to add a `build.md` entry.
+- **D-02:** The first paragraph states that KVBench generates and runs `nixlbench` commands, with an inline link to the NIXLBench section. This satisfies requirement KB-01.
+- **D-03:** The overview presents the two command categories (KVBench commands: plan/profile/kvcache and CTP commands: ct-perftest/sequential-ct-perftest) with brief one-line descriptions. Claude's discretion on visual layout.
+- **D-04:** Mention supported LLM architectures (DeepSeek R1, Llama 3.1, and others) on the overview page as a feature highlight. No deep explanation -- that's Phase 36.
+- **D-05:** Use `` component for Docker vs Python venv build paths, consistent with ROADMAP KB-02.
+- **D-06:** Docker tab links to the NIXLBench build page rather than repeating Docker build steps. Brief note: "KVBench is included in the NIXLBench Docker container" + link to NIXLBench build page. No duplication.
+- **D-07:** Python venv tab is self-contained -- shows the `venv` + `pip`/`uv` install steps directly since they're short and specific to KVBench.
+- **D-08:** No system requirements section on the build page (KVBench's only requirements are Python 3.12+ and optional PyTorch for GPU benchmarks -- mention inline).
+
+### Claude's Discretion
+- Visual layout for the two command categories (grouped bullets vs two sections)
+- Whether to include a "Supported Models" subsection on overview or just a brief mention
+- Exact frontmatter `description:` text for each page
+- Whether to include a "Next steps" link at the bottom of overview pointing to build page
+
+### Deferred Ideas (OUT OF SCOPE)
+- KVBench command reference (all 5 subcommands with CLI tables) -- Phase 36
+- Model configuration YAML schema documentation -- Phase 36
+- LLM architecture examples (DeepSeek R1, Llama 3.1) -- Phase 36
+- CTP examples and matrix format documentation -- Phase 36
+- KVBench GDS tutorial -- P2, deferred per REQUIREMENTS.md
+- Adding new model architecture guide -- P2, deferred per REQUIREMENTS.md
+
+
+
+## Phase Requirements
+
+| ID | Description | Research Support |
+|----|-------------|------------------|
+| KB-01 | KVBench overview page states in its first paragraph that `profile` invokes `nixlbench` as a subprocess (NIXLBench dependency), and covers what KVBench is and its two command categories (KVBench commands vs CTP commands) | Source content in `benchmark/kvbench/README.md` lines 26-32; NIXLBench page pattern in `docs/development/benchmarking/nixlbench/index.md`; cross-link target is `./nixlbench` relative path |
+| KB-02 | KVBench build page covers Docker build (re-using NIXLBench container) and Python venv install, using `` for the two paths | Source content in `benchmark/kvbench/README.md` lines 38-56; Tabs pattern in `docs/development/benchmarking/nixlbench/build.md`; Docker tab links to NIXLBench build page, Python tab shows venv+uv steps |
+
+
+## Architecture Patterns
+
+### Fern MDX Page Structure (Established Pattern)
+
+Every doc page follows this structure, verified from the NIXLBench Phase 33 output:
+
+```markdown
+---
+title: Page Title
+description: One-sentence summary for SEO and navigation.
+---
+
+First paragraph introduces the tool/concept with inline cross-links on first mention.
+
+## Section Heading
+
+Content...
+
+## Next Steps
+
+- **[Link text](./relative-path)** -- Brief description
+```
+
+[VERIFIED: `docs/development/benchmarking/nixlbench/index.md`]
+
+### Cross-Link Conventions
+
+- Use relative Fern doc paths: `./nixlbench/build` (not full filesystem paths)
+- Backend names link to their User Guide pages on first mention: `[UCX](/docs/user-guide/backends/ucx)`
+- etcd links to: `/docs/user-guide/etcd-metadata-exchange`
+- NIXLBench section root: referenced as a relative sibling path from KVBench pages
+
+[VERIFIED: `docs/development/benchmarking/nixlbench/index.md` line 6-11]
+
+### Tabs Component Pattern
+
+The `` component is used in the NIXLBench build page and works as follows:
+
+```markdown
+
+
+
+Content for Docker tab...
+
+
+
+
+Content for Python venv tab...
+
+
+
+```
+
+Key rules:
+- Blank line after `` and before ` `
+- Markdown content inside tabs renders normally
+- Code blocks inside tabs work without issues
+
+[VERIFIED: `docs/development/benchmarking/nixlbench/build.md` lines 27-110]
+
+### Navigation Config Pattern (docs/index.yml)
+
+Current KVBench section (Phase 32 output):
+```yaml
+- section: KVBench
+ collapsed: open-by-default
+ path: development/benchmarking/kvbench/index.md
+ contents:
+ - page: Commands and Examples
+ path: development/benchmarking/kvbench/commands.md
+```
+
+Must be updated to add `build.md`:
+```yaml
+- section: KVBench
+ collapsed: open-by-default
+ path: development/benchmarking/kvbench/index.md
+ contents:
+ - page: Building KVBench
+ path: development/benchmarking/kvbench/build.md
+ - page: Commands and Examples
+ path: development/benchmarking/kvbench/commands.md
+```
+
+This mirrors the NIXLBench section structure where build comes before usage/commands.
+
+[VERIFIED: `docs/index.yml` lines 79-84]
+
+### Anti-Patterns to Avoid
+- **GitHub-flavored anchor links (`[text](#heading)`):** Fern does not support these. Use Fern relative doc links instead.
+- **HTML comments (``):** Fern MDX does not strip HTML comments; they render as visible text.
+- **Bare HTML tags:** Only Fern-provided components (``, ``, ``, ``, etc.) are allowed.
+- **Duplicating content from other pages:** Always cross-link instead (D-06 mandates this for Docker build steps).
+
+## Don't Hand-Roll
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| Tab switching UI | Custom HTML/JS tabs | `` / `` Fern component | Fern renders these natively; custom HTML breaks |
+| Callout boxes | Custom blockquote styling | ``, `` Fern components | Consistent styling, accessible |
+| Navigation structure | Manual sidebar links | `docs/index.yml` entries | Fern generates sidebar from this config |
+
+## Common Pitfalls
+
+### Pitfall 1: pyproject.toml says Python >=3.10 but README and docs say 3.12+
+**What goes wrong:** Conflicting Python version requirements confuse users.
+**Why it happens:** `pyproject.toml` has `requires-python = ">=3.10"` but the project README and NIXLBench build page both state Python 3.12+.
+**How to avoid:** Follow the user decision D-08 which says "Python 3.12+" -- this aligns with the NIXLBench build page and is the documented minimum. Do not reference pyproject.toml's `>=3.10`.
+**Warning signs:** Mentioning Python 3.10 anywhere in the docs.
+
+[VERIFIED: `benchmark/kvbench/pyproject.toml` line 19 shows `>=3.10`; NIXLBench build.md line 23 shows "Python: 3.12 or later"]
+
+### Pitfall 2: Forgetting the Blank Lines Around Tab Content
+**What goes wrong:** Markdown inside `` tags doesn't render as expected (headings become plain text, lists break).
+**Why it happens:** Fern MDX requires blank lines after opening `` and before closing ` ` for markdown parsing to activate.
+**How to avoid:** Always include blank lines as shown in the Tabs pattern above.
+**Warning signs:** Content inside tabs appearing as unformatted text.
+
+### Pitfall 3: Wrong Cross-Link Path Format
+**What goes wrong:** Links to NIXLBench pages return 404 in the rendered docs.
+**Why it happens:** Using filesystem-relative paths instead of Fern doc paths.
+**How to avoid:** Use Fern doc link format: `/docs/development/benchmarking/nixlbench/build` (no `.md` extension in some contexts) or relative `./nixlbench/build`. Check existing cross-links in NIXLBench pages for the exact format used.
+**Warning signs:** Any link containing `.md` extension or filesystem-style paths.
+
+[VERIFIED: NIXLBench index.md uses `./nixlbench/build` format]
+
+### Pitfall 4: Describing Commands in Too Much Detail on Overview Page
+**What goes wrong:** Overview page becomes a mini-reference, duplicating Phase 36 content.
+**Why it happens:** The README has extensive command documentation that's easy to over-include.
+**How to avoid:** Stick to D-03: brief one-line descriptions per command. Full CLI tables and examples are Phase 36 scope.
+**Warning signs:** Any CLI argument tables or multi-line command examples on the overview page.
+
+## Code Examples
+
+### KVBench Overview Page (index.md) Structure
+
+```markdown
+---
+title: KVBench
+description: [one-line description]
+---
+
+[First paragraph: KVBench generates and runs nixlbench commands for KV cache
+transfer benchmarking. The `profile` command invokes [NIXLBench](link) as a
+subprocess. Inline link to NIXLBench section.]
+
+## Command Categories
+
+### KVBench Commands
+- **plan** -- one-line description
+- **profile** -- one-line description
+- **kvcache** -- one-line description
+
+### CTP Commands
+- **ct-perftest** -- one-line description
+- **sequential-ct-perftest** -- one-line description
+
+## Supported Models
+
+Brief mention of DeepSeek R1, Llama 3.1, and others.
+
+## Next Steps
+
+- **[Building KVBench](./kvbench/build)** -- Docker and Python venv install
+- **[Commands and Examples](./kvbench/commands)** -- Full command reference
+```
+
+[ASSUMED -- layout is Claude's discretion per CONTEXT.md; planner should finalize]
+
+### KVBench Build Page (build.md) Structure
+
+```markdown
+---
+title: Building KVBench
+description: [one-line description]
+---
+
+KVBench requires Python 3.12 or later. For GPU benchmarks, PyTorch is also needed.
+
+
+
+
+KVBench is included in the NIXLBench Docker container. See
+[Building NIXLBench](/docs/development/benchmarking/nixlbench/build) for
+Docker build instructions.
+
+
+
+
+Clone the repository and set up a virtual environment:
+
+\```bash
+git clone https://github.com/ai-dynamo/nixl.git
+cd nixl/benchmark/kvbench
+python3 -m venv venv
+source venv/bin/activate
+pip install uv
+uv sync --active
+\```
+
+
+
+```
+
+[VERIFIED: build steps from `benchmark/kvbench/README.md` lines 49-56; Docker cross-link pattern from D-06]
+
+## Source Content Mapping
+
+| README Section | Maps To | Phase |
+|----------------|---------|-------|
+| Overview (lines 26-32) | `index.md` first paragraph + command categories | 35 |
+| Supported LLM Architectures (lines 33-36) | `index.md` brief model mention | 35 |
+| Building > Docker (lines 40-46) | `build.md` Docker tab (cross-link only) | 35 |
+| Building > Python (lines 48-56) | `build.md` Python venv tab | 35 |
+| Command Descriptions (lines 160-215) | `commands.md` | 36 (OUT OF SCOPE) |
+| Command Line Arguments (lines 79-158) | `commands.md` | 36 (OUT OF SCOPE) |
+| Examples (lines 217-405) | `commands.md` | 36 (OUT OF SCOPE) |
+
+## Files to Create/Modify
+
+| File | Action | Notes |
+|------|--------|-------|
+| `docs/development/benchmarking/kvbench/index.md` | **Replace** stub | Currently a placeholder from Phase 32 |
+| `docs/development/benchmarking/kvbench/build.md` | **Create** new | Does not exist yet |
+| `docs/index.yml` | **Modify** | Add `build.md` entry under KVBench section, before `commands.md` |
+
+## Validation Architecture
+
+### Test Framework
+| Property | Value |
+|----------|-------|
+| Framework | Fern CLI |
+| Config file | `fern/fern.config.json` (assumed) |
+| Quick run command | `cd fern && fern check` |
+| Full suite command | `cd fern && fern check` |
+
+### Phase Requirements to Test Map
+| Req ID | Behavior | Test Type | Automated Command | File Exists? |
+|--------|----------|-----------|-------------------|-------------|
+| KB-01 | Overview page has correct content structure | manual | Visual review of `index.md` content | Will exist after implementation |
+| KB-02 | Build page has Docker/venv tabs | smoke | `cd fern && fern check` | Will exist after implementation |
+| QS-04 | Fern-compatible MDX (no build errors) | smoke | `cd fern && fern check` | N/A |
+
+### Sampling Rate
+- **Per task commit:** `cd fern && fern check`
+- **Per wave merge:** `cd fern && fern check`
+- **Phase gate:** Full `fern check` green before `/gsd-verify-work`
+
+### Wave 0 Gaps
+None -- existing Fern infrastructure covers all phase requirements. `fern check` validates MDX syntax, frontmatter, and navigation config.
+
+## Assumptions Log
+
+| # | Claim | Section | Risk if Wrong |
+|---|-------|---------|---------------|
+| A1 | Overview page layout with "Command Categories" and "Supported Models" sections | Code Examples | Low -- layout is Claude's discretion per CONTEXT.md; easy to adjust |
+| A2 | Cross-link format `./kvbench/build` for sibling page links | Code Examples | Medium -- may need `/docs/development/benchmarking/kvbench/build` format; verify against NIXLBench pattern |
+| A3 | `fern check` is the correct validation command | Validation Architecture | Low -- standard Fern CLI command; used in prior phases |
+
+## Open Questions
+
+1. **Exact Fern cross-link path format for sibling pages**
+ - What we know: NIXLBench index.md uses `./nixlbench/build` to link to its build page
+ - What's unclear: Whether KVBench index.md should use `./kvbench/build` or just `./build` since they're in the same directory
+ - Recommendation: Check what NIXLBench index.md does -- it uses `./nixlbench/build` which suggests the path is relative to the benchmarking section root, not the current file. Implementer should verify with `fern check`.
+
+## Sources
+
+### Primary (HIGH confidence)
+- `benchmark/kvbench/README.md` -- Full source content for overview, build steps, command categories
+- `benchmark/kvbench/pyproject.toml` -- Python version requirement (>=3.10), dependencies
+- `docs/development/benchmarking/nixlbench/index.md` -- NIXLBench overview page pattern (Phase 33 output)
+- `docs/development/benchmarking/nixlbench/build.md` -- NIXLBench build page pattern with Tabs component
+- `docs/index.yml` lines 79-84 -- Current KVBench navigation config
+- `benchmark/kvbench/examples/model_deepseek_r1.yaml` -- DeepSeek R1 model architecture fields
+- `benchmark/kvbench/examples/model_llama_3_1_70b.yaml` -- Llama 3.1 70B model architecture fields
+
+### Secondary (MEDIUM confidence)
+- None
+
+### Tertiary (LOW confidence)
+- None
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH -- no libraries needed; pure content authoring using established Fern MDX patterns
+- Architecture: HIGH -- page structure directly mirrors Phase 33 NIXLBench output
+- Pitfalls: HIGH -- verified against actual source files and existing pages
+
+**Research date:** 2026-04-07
+**Valid until:** 2026-05-07 (stable -- Fern MDX patterns unlikely to change)
diff --git a/.planning/phases/35-kvbench-overview-and-build/35-VERIFICATION.md b/.planning/phases/35-kvbench-overview-and-build/35-VERIFICATION.md
new file mode 100644
index 0000000000..c150956e06
--- /dev/null
+++ b/.planning/phases/35-kvbench-overview-and-build/35-VERIFICATION.md
@@ -0,0 +1,83 @@
+---
+phase: 35-kvbench-overview-and-build
+verified: 2026-04-07T12:00:00Z
+status: passed
+score: 8/8 must-haves verified
+overrides_applied: 0
+---
+
+# Phase 35: KVBench Overview and Build Verification Report
+
+**Phase Goal:** Developers understand that KVBench drives NIXLBench as a subprocess and know how to install KVBench using either the Docker container or a Python virtual environment
+**Verified:** 2026-04-07T12:00:00Z
+**Status:** passed
+**Re-verification:** No -- initial verification
+
+## Goal Achievement
+
+### Observable Truths
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | KVBench overview page states in its first paragraph that profile invokes nixlbench as a subprocess | VERIFIED | index.md line 6: "The `profile` command invokes `nixlbench` as a subprocess" |
+| 2 | KVBench overview page explains two command categories: KVBench commands and CTP commands | VERIFIED | index.md has `### KVBench Commands` (plan, profile, kvcache) and `### CTP Commands` (ct-perftest, sequential-ct-perftest) -- all 5 subcommands listed |
+| 3 | KVBench build page presents Docker and Python venv paths in a Tabs component | VERIFIED | build.md has `` with `` and `` |
+| 4 | Docker tab links to NIXLBench build page instead of duplicating Docker build steps | VERIFIED | build.md line 13: `See [Building NIXLBench](./nixlbench/build)` -- no duplicated build commands |
+| 5 | Python venv tab shows self-contained venv + uv install steps | VERIFIED | build.md lines 22-27: complete git clone, venv, pip install uv, uv sync sequence with verification step |
+| 6 | Both pages have valid Fern MDX frontmatter with title and description | VERIFIED | index.md: `title: KVBench`, `description: A KV cache benchmarking...`; build.md: `title: Building KVBench`, `description: Install KVBench...` |
+| 7 | docs/index.yml includes build.md entry under KVBench section | VERIFIED | index.yml line 83-84: `page: Building KVBench` / `path: development/benchmarking/kvbench/build.md` |
+| 8 | fern check passes with zero errors | VERIFIED | `fern check` returns 0 errors (1 warning is pre-existing color contrast issue, unrelated) |
+
+**Score:** 8/8 truths verified
+
+### Required Artifacts
+
+| Artifact | Expected | Status | Details |
+|----------|----------|--------|---------|
+| `docs/development/benchmarking/kvbench/index.md` | KVBench overview with nixlbench dependency and command categories | VERIFIED | 29 lines, contains "nixlbench", command categories, supported models, next steps |
+| `docs/development/benchmarking/kvbench/build.md` | KVBench build instructions with Docker and Python venv tabs | VERIFIED | 38 lines, contains `` component with both installation paths |
+| `docs/index.yml` | Navigation entry for KVBench build page | VERIFIED | Contains "Building KVBench" entry before "Commands and Examples" |
+
+### Key Link Verification
+
+| From | To | Via | Status | Details |
+|------|----|-----|--------|---------|
+| kvbench/index.md | NIXLBench section | `[NIXLBench](./nixlbench)` in first paragraph | WIRED | Link present at line 6 |
+| kvbench/build.md | NIXLBench build page | `[Building NIXLBench](./nixlbench/build)` in Docker tab | WIRED | Cross-link present at line 13 |
+
+### Data-Flow Trace (Level 4)
+
+Not applicable -- documentation-only phase with no dynamic data rendering.
+
+### Behavioral Spot-Checks
+
+| Behavior | Command | Result | Status |
+|----------|---------|--------|--------|
+| fern check passes | `cd fern && fern check` | 0 errors, 1 pre-existing warning | PASS |
+| Commits exist | `git log --oneline 58e4d2ab` / `d359cdbb` | Both commits found | PASS |
+
+### Requirements Coverage
+
+| Requirement | Source Plan | Description | Status | Evidence |
+|-------------|------------|-------------|--------|----------|
+| KB-01 | 35-01-PLAN | KVBench overview page states profile invokes nixlbench as subprocess, covers two command categories | SATISFIED | index.md first paragraph + command categories sections verified |
+| KB-02 | 35-01-PLAN | KVBench build page covers Docker (reusing NIXLBench container) and Python venv in Tabs component | SATISFIED | build.md Tabs component with Docker cross-link and Python venv install verified |
+
+### Anti-Patterns Found
+
+| File | Line | Pattern | Severity | Impact |
+|------|------|---------|----------|--------|
+| (none) | - | - | - | No TODOs, FIXMEs, placeholders, or stubs found |
+
+### Human Verification Required
+
+No items require human verification. All truths are verifiable programmatically.
+
+### Gaps Summary
+
+No gaps found. All 8 must-haves verified, both requirements (KB-01, KB-02) satisfied, fern check passes, and no anti-patterns detected.
+
+---
+
+_Verified: 2026-04-07T12:00:00Z_
+_Verifier: Claude (gsd-verifier)_
diff --git a/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-01-PLAN.md b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-01-PLAN.md
new file mode 100644
index 0000000000..5ed2467db1
--- /dev/null
+++ b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-01-PLAN.md
@@ -0,0 +1,370 @@
+---
+phase: 36-kvbench-commands-model-config-and-llm-examples
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+ - docs/development/benchmarking/kvbench/commands.md
+autonomous: true
+requirements: [KB-03, KB-04, KB-05]
+
+must_haves:
+ truths:
+ - "Developer can look up any of the 5 KVBench subcommands and see its description and CLI arguments"
+ - "Developer can understand the model architecture YAML schema (both MLA and MHA/GQA variants) from field tables and annotated examples"
+ - "Developer can understand the model config YAML schema (strategy/runtime/system) from field tables and annotated examples"
+ - "Developer can copy-paste DeepSeek R1 end-to-end example (both block and layer access patterns) and run it"
+ - "Developer can copy-paste Llama 3.1 end-to-end example and run it"
+ artifacts:
+ - path: "docs/development/benchmarking/kvbench/commands.md"
+ provides: "Complete KVBench commands, model config, and LLM examples page"
+ contains: "## Command Reference"
+ min_lines: 300
+ key_links:
+ - from: "docs/development/benchmarking/kvbench/commands.md"
+ to: "docs/development/benchmarking/nixlbench/index.md"
+ via: "inline link to NIXLBench overview"
+ pattern: "nixlbench"
+ - from: "docs/development/benchmarking/kvbench/commands.md"
+ to: "docs/user-guide/backends/*"
+ via: "first-mention inline links for backend names"
+ pattern: "\\(/docs/user-guide/backends/"
+---
+
+
+Author the full KVBench commands.md page replacing the Phase 32 stub. This single page (per D-01) contains three major sections: (1) Command Reference documenting all 5 subcommands with CLI argument tables, (2) Model Configuration Guide documenting both YAML schemas with field tables and annotated examples, and (3) LLM Examples with end-to-end DeepSeek R1 and Llama 3.1 configurations.
+
+Purpose: Developers need a single reference page to look up KVBench subcommands, understand model configuration schemas, and copy-paste working examples.
+Output: Complete `docs/development/benchmarking/kvbench/commands.md` (replacing 7-line stub)
+
+
+
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/workflows/execute-plan.md
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/templates/summary.md
+
+
+
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-CONTEXT.md
+@.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-RESEARCH.md
+@.planning/phases/35-kvbench-overview-and-build/35-01-SUMMARY.md
+
+
+
+
+
+
+
+
+
+
+From docs/development/benchmarking/nixlbench/usage.md:
+- Fern frontmatter: `title:` + `description:`
+- First paragraph references related pages with relative links
+- Two-column tables for CLI flags
+- Code blocks with `bash` language tag
+- First-mention inline links for backend names
+- ``, `` Fern components where needed
+
+
+
+
+
+
+ Task 1: Author Command Reference and CLI Argument Tables
+ docs/development/benchmarking/kvbench/commands.md
+
+ - benchmark/kvbench/README.md (full file -- CLI tables lines 79-158, command descriptions lines 160-216, examples lines 218-296)
+ - benchmark/kvbench/commands/args.py (authoritative Click CLI definitions -- validate argument names and help text)
+ - benchmark/kvbench/main.py (entry point -- validate subcommand names and kvcache output format)
+ - docs/development/benchmarking/nixlbench/usage.md (Fern doc style pattern from Phase 34)
+ - docs/development/benchmarking/kvbench/index.md (cross-link targets, command category descriptions)
+
+
+Replace the 7-line stub in `docs/development/benchmarking/kvbench/commands.md` with the first two major sections of the page. Keep the existing frontmatter (`title: KVBench Commands and Examples`, `description: ...`).
+
+**Page structure to write (this task covers sections 1 and 2 only; Task 2 adds sections 3 and 4):**
+
+**Intro paragraph:** Brief sentence explaining this page covers command reference, model configuration, and examples. Link to [Building KVBench](./kvbench/build) for installation. Link to [NIXLBench](./nixlbench) since profile invokes it as a subprocess.
+
+**Section 1: ## Command Reference** (per D-02)
+
+Write subsections for each of the 5 subcommands in this order (per discretion recommendation: KVBench commands first, then CTP):
+
+1. `### plan` -- Generates recommended nixlbench configurations. Show usage: `python main.py plan --model ... --model_config ... --backend ...`. Brief description from README line 164-170.
+
+2. `### profile` -- Runs nixlbench with specified configuration. Show usage. Brief description from README line 172-178.
+
+3. `### kvcache` -- Displays KV cache information for a model config. Show usage AND the tabulated output example from README lines 184-189 (the 4-line table showing Model/ISL/Num Requests/Batch Size/IO Size/TP/PP/Page Size/Access). Per discretion: include it -- it helps developers understand what kvcache shows.
+
+4. `### ct-perftest` -- Runs single custom traffic pattern benchmark. Brief description from README lines 209-215. Include the report format description (total latency, avg time per iteration, total size, avg bandwidth by rank). Add `` about `CUDA_VISIBLE_DEVICES` requirement from README line 215.
+
+5. `### sequential-ct-perftest` -- Runs sequential custom traffic patterns. Brief description from README lines 194-207. Include report format (total latency per matrix, isolated latency). Show the output table format from README lines 200-206.
+
+**Section 2: ## Command Line Arguments** (per D-03, D-04, D-05)
+
+Five subsections with two-column tables (Argument | Description). Default values in description text, not separate column (per D-03).
+
+1. `### Common Arguments` -- 3 args: `--model`, `--model_config`, `--model_configs`. From README lines 85-89.
+
+2. `### CLI Override Arguments` -- 7 args: `--pp`, `--tp`, `--isl`, `--osl`, `--num_requests`, `--page_size`, `--access_pattern`. From README lines 95-103.
+
+3. `### Plan Command Arguments` -- 1 arg: `--format`. From README lines 109-111.
+
+4. `### Shared Benchmark Arguments` -- All args from README lines 117-148. This includes `--source`, `--destination`, `--backend`, `--worker_type`, `--initiator_seg_type`, `--target_seg_type`, `--scheme`, `--mode`, `--op_type`, `--check_consistency`, `--total_buffer_size`, `--recreate_xfer`, `--start_block_size`, `--max_block_size`, `--start_batch_size`, `--max_batch_size`, `--num_iter`, `--warmup_iter`, `--num_threads`, `--num_initiator_dev`, `--num_target_dev`, `--enable_pt`, `--progress_threads`, `--device_list`, `--runtime_type`, `--etcd-endpoints`, `--storage_enable_direct`, `--filepath`, `--enable_vmm`. Transcribe descriptions and defaults from README. IMPORTANT per D-05/QS-03: Use `--etcd-endpoints` (hyphens) for KVBench, and add a `` after the table: "KVBench uses `--etcd-endpoints` (hyphens). NIXLBench uses `--etcd_endpoints` (underscores). Both forms are accepted by the CLI, but this documentation follows each tool's convention."
+
+5. `### CTP Command Arguments` -- 4 args: `config_file`, `--verify-buffers / --no-verify-buffers`, `--print-recv-buffers / --no-print-recv-buffers`, `--json-output-path`. From README lines 154-158.
+
+Cross-reference argument names against `commands/args.py` to validate spelling and help text. Backend names on first mention should link to their User Guide pages using the pattern: [UCX](/docs/user-guide/backends/ucx), [GDS](/docs/user-guide/backends/gds), etc.
+
+Do NOT write sections 3 (Model Configuration Guide) or 4 (LLM Examples) yet -- Task 2 handles those.
+
+
+ cd /home/omrik/Projects/nixl.gitlab/fern && fern check 2>&1 | tail -20
+
+
+ - grep -c "### plan" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### profile" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### kvcache" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### ct-perftest" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### sequential-ct-perftest" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "## Command Line Arguments" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### Common Arguments" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### CLI Override Arguments" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### Shared Benchmark Arguments" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### CTP Command Arguments" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep "etcd-endpoints" docs/development/benchmarking/kvbench/commands.md (hyphens used for KVBench)
+ - grep "/docs/user-guide/backends/" docs/development/benchmarking/kvbench/commands.md (backend cross-links present)
+ - fern check passes with zero errors
+
+
+
+
+ Task 2: Author Model Configuration Guide and LLM Examples
+ docs/development/benchmarking/kvbench/commands.md
+
+ - docs/development/benchmarking/kvbench/commands.md (read the file as written by Task 1 to append after it)
+ - benchmark/kvbench/docs/creating-a-model-config.md (model config schema guide -- adapt for Fern style)
+ - benchmark/kvbench/examples/model_deepseek_r1.yaml (DeepSeek R1 model architecture YAML)
+ - benchmark/kvbench/examples/model_llama_3_1_70b.yaml (Llama 3.1 70B model architecture YAML)
+ - benchmark/kvbench/examples/block-tp1-pp8.yaml (block access model config)
+ - benchmark/kvbench/examples/layer-tp1-pp8.yaml (layer access model config)
+ - benchmark/kvbench/models/model_config.py (dataclass definitions -- authoritative field list and types)
+ - benchmark/kvbench/README.md (lines 218-296 -- example command output for DeepSeek R1)
+
+
+Append sections 3 and 4 to the commands.md file written by Task 1. Do NOT modify the frontmatter or sections 1-2.
+
+**Section 3: ## Model Configuration Guide** (per D-06, D-07)
+
+Intro paragraph: KVBench uses two YAML configuration files -- a model architecture file describing the LLM structure, and a model config file specifying parallelism, runtime, and system settings.
+
+**### Model Architecture YAML**
+
+Explain that different model architectures have different fields depending on their attention mechanism.
+
+**Common fields** table (two-column: Field | Description):
+- `model_name` -- Model identifier (e.g., `DEEPSEEK_R1`, `LLAMA3.1_70B`)
+- `num_layers` -- Number of transformer layers
+- `query_head_dimension` -- Dimension of each query head
+- `num_model_params` -- Total model parameter count
+
+**MLA fields** (Multi-Latent Attention, e.g., DeepSeek R1) table:
+- `num_query_heads` -- Number of query attention heads
+- `embedding_dimension` -- Model embedding dimension
+- `rope_mla_dimension` -- RoPE dimension for MLA
+- `mla_latent_vector_dimension` -- Latent vector dimension for MLA compression
+
+**MHA/GQA fields** (Multi-Head / Grouped-Query Attention, e.g., Llama 3.1) table:
+- `num_query_heads_with_mha` -- Number of query heads (MHA variant)
+- `gqa_num_queries_in_group` -- Number of queries per KV head group (GQA)
+
+Then show **complete annotated example** for DeepSeek R1 (per D-07):
+```yaml
+model_name: 'DEEPSEEK_R1'
+num_layers: 61
+num_query_heads: 128
+query_head_dimension: 128
+embedding_dimension: 7168
+rope_mla_dimension: 64
+mla_latent_vector_dimension: 512
+num_model_params: 671000000000
+```
+(Add inline YAML comments explaining each field)
+
+And a **complete annotated example** for Llama 3.1 70B:
+```yaml
+model_name: 'LLAMA3.1_70B'
+num_layers: 80
+num_query_heads_with_mha: 64
+query_head_dimension: 128
+gqa_num_queries_in_group: 8
+num_model_params: 70000000000
+```
+
+**### Model Config YAML**
+
+Three-section structure: `strategy`, `runtime`, `system`. Per D-06.
+
+**Strategy fields** table:
+- `tp_size` -- Tensor parallelism size (default: 1)
+- `pp_size` -- Pipeline parallelism size (default: 1)
+- `model_quant_mode` -- Model weight quantization mode (default: `"fp8"`)
+- `kvcache_quant_mode` -- KV cache quantization mode (default: `"fp8"`)
+
+**Runtime fields** table:
+- `isl` -- Input sequence length in tokens (default: 1)
+- `osl` -- Output sequence length in tokens (default: 1)
+- `num_requests` -- Number of inference requests (default: 1)
+
+**System fields** table:
+- `hardware` -- Hardware platform (e.g., `"H100"`, `"A100"`)
+- `backend` -- Inference backend engine (e.g., `"SGLANG"`)
+- `access_pattern` -- KV cache access pattern: `"block"` or `"layer"`
+- `page_size` -- Page size for access pattern (default: 1)
+- `source` -- Source descriptor type
+- `destination` -- Destination descriptor type
+
+Then show **complete annotated example** using `block-tp1-pp8.yaml` values (per D-07):
+```yaml
+strategy:
+ tp_size: 1 # Tensor parallelism -- 1 GPU for tensor-parallel
+ pp_size: 8 # Pipeline parallelism -- 8 GPUs for pipeline-parallel
+ model_quant_mode: "fp8" # Model weight quantization
+ kvcache_quant_mode: "fp8" # KV cache quantization
+
+runtime:
+ isl: 1000 # Input sequence length (tokens)
+ osl: 100 # Output sequence length (tokens)
+ num_requests: 10 # Number of inference requests
+
+system:
+ hardware: "H100" # Hardware platform
+ backend: "SGLANG" # Inference backend engine
+ access_pattern: "block" # KV cache access pattern
+ page_size: 16 # Page size for block access
+```
+
+Add a `` explaining block vs layer access patterns: "Block access groups KV cache entries into fixed-size pages. Layer access transfers KV cache one transformer layer at a time. Block access typically produces fewer, larger transfers; layer access produces more, smaller transfers."
+
+**Section 4: ## LLM Examples** (per D-08, D-09)
+
+Intro paragraph: End-to-end examples showing model architecture YAML, model config YAML, and the `plan` and `profile` commands.
+
+**### DeepSeek R1**
+
+**#### Block Access (TP=1, PP=16)**
+
+Show the full end-to-end flow per D-08:
+1. Model architecture YAML (`model_deepseek_r1.yaml`) -- show the full YAML without license header
+2. Model config YAML (`block-tp1-pp16.yaml`) -- reconstruct from block-tp1-pp8.yaml but with pp_size: 16 and isl: 10000, page_size: 256 (matching README example output which shows PP: 16, Page Size: 256, ISL: 10000 tokens)
+3. Plan command -- copy from README lines 222-228
+4. Output -- copy from README lines 229-243
+5. Profile command:
+```bash
+python main.py profile \
+ --model ./examples/model_deepseek_r1.yaml \
+ --model_config ./examples/block-tp1-pp16.yaml \
+ --backend GDS \
+ --source gpu \
+ --etcd-endpoints "http://localhost:2379"
+```
+
+**#### Layer Access (TP=1, PP=16)** (per D-09 -- both access patterns for DeepSeek R1)
+
+1. Model config YAML (`layer-tp1-pp16.yaml`) -- same as block but `access_pattern: "layer"`. (Model architecture YAML unchanged -- add note "Uses the same model architecture YAML as above.")
+2. Plan command -- copy from README lines 247-253
+3. Output -- copy from README lines 254-268
+4. Brief note comparing the output: "With layer access, the batch size increases and block size decreases compared to block access, reflecting the per-layer transfer granularity."
+
+**### Llama 3.1 70B**
+
+**#### Block Access (TP=1, PP=8)**
+
+Per D-08 -- end-to-end example:
+1. Model architecture YAML (`model_llama_3_1_70b.yaml`) -- show the full YAML without license header
+2. Model config YAML (`block-tp1-pp8.yaml`) -- show the full YAML without license header
+3. Plan command:
+```bash
+python main.py plan \
+ --model ./examples/model_llama_3_1_70b.yaml \
+ --model_config ./examples/block-tp1-pp8.yaml \
+ --backend GDS \
+ --source gpu \
+ --etcd-endpoints "http://localhost:2379"
+```
+4. For the output: Since README has no Llama 3.1 output, show only the command and add a ``: "The output follows the same format as the DeepSeek R1 example above, with values computed from the Llama 3.1 70B architecture." Do NOT fabricate specific batch_size/block_size numbers.
+5. Profile command (same args as plan but `profile` instead of `plan`).
+
+Ensure all backend names on first mention in sections 3-4 link to their User Guide pages.
+
+
+ cd /home/omrik/Projects/nixl.gitlab/fern && fern check 2>&1 | tail -20
+
+
+ - grep -c "## Model Configuration Guide" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### Model Architecture YAML" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### Model Config YAML" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "## LLM Examples" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### DeepSeek R1" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep -c "### Llama 3.1" docs/development/benchmarking/kvbench/commands.md returns 1
+ - grep "model_deepseek_r1.yaml" docs/development/benchmarking/kvbench/commands.md (DeepSeek YAML shown)
+ - grep "model_llama_3_1_70b.yaml" docs/development/benchmarking/kvbench/commands.md (Llama YAML shown)
+ - grep "block-tp1-pp" docs/development/benchmarking/kvbench/commands.md (block access config shown)
+ - grep "layer-tp1-pp" docs/development/benchmarking/kvbench/commands.md (layer access config shown per D-09)
+ - grep "mla_latent_vector_dimension" docs/development/benchmarking/kvbench/commands.md (MLA fields documented)
+ - grep "gqa_num_queries_in_group" docs/development/benchmarking/kvbench/commands.md (GQA fields documented)
+ - grep "strategy:" docs/development/benchmarking/kvbench/commands.md (model config uses YAML key, not Python attr)
+ - wc -l docs/development/benchmarking/kvbench/commands.md shows 300+ lines
+ - fern check passes with zero errors
+
+
+
+
+
+
+## Trust Boundaries
+
+| Boundary | Description |
+|----------|-------------|
+| Documentation content | No trust boundaries -- this is static documentation with no user input processing |
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-36-01 | Information Disclosure | CLI examples with etcd URLs | accept | Examples use localhost URLs only; no credentials shown |
+| T-36-02 | Tampering | YAML examples could mislead users | accept | All values sourced from verified example files in repository |
+
+
+
+1. `fern check` passes with zero errors
+2. All 5 subcommands have dedicated subsections under Command Reference
+3. All 5 argument groups have two-column tables under Command Line Arguments
+4. Model Architecture YAML documents both MLA and MHA/GQA field variants
+5. Model Config YAML documents all three sections (strategy/runtime/system)
+6. DeepSeek R1 example shows both block and layer access patterns
+7. Llama 3.1 example shows block access pattern with end-to-end flow
+8. `--etcd-endpoints` uses hyphens throughout (KVBench convention per QS-03)
+9. Backend names link to User Guide pages on first mention
+
+
+
+- `docs/development/benchmarking/kvbench/commands.md` is 300+ lines replacing the 7-line stub
+- Page has 4 major sections: Command Reference, Command Line Arguments, Model Configuration Guide, LLM Examples
+- All 5 subcommands documented with descriptions
+- CLI tables validated against args.py source
+- Both YAML schemas have field tables and annotated examples
+- DeepSeek R1 and Llama 3.1 end-to-end examples present
+- Block and layer access pattern variants shown for DeepSeek R1
+- fern check passes
+
+
+
diff --git a/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-01-SUMMARY.md b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-01-SUMMARY.md
new file mode 100644
index 0000000000..bba23fdb9c
--- /dev/null
+++ b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-01-SUMMARY.md
@@ -0,0 +1,93 @@
+---
+phase: 36-kvbench-commands-model-config-and-llm-examples
+plan: 01
+subsystem: docs
+tags: [kvbench, fern, cli-reference, yaml-schema, deepseek, llama, benchmarking]
+
+# Dependency graph
+requires:
+ - phase: 35-kvbench-overview-and-build
+ provides: KVBench overview and build pages (index.md, build.md)
+provides:
+ - Complete KVBench commands.md page with command reference, model config guide, and LLM examples
+affects: [37-terminology-normalization-and-quality-audit]
+
+# Tech tracking
+tech-stack:
+ added: []
+ patterns: [two-column CLI argument tables, annotated YAML examples, end-to-end copy-paste examples]
+
+key-files:
+ created: []
+ modified:
+ - docs/development/benchmarking/kvbench/commands.md
+
+key-decisions:
+ - "Documented README CLI arguments only; backend-specific args (GDS, GPUNETIO, HF3FS, OBJ) are passthrough to NIXLBench and documented on respective backend pages"
+ - "Llama 3.1 example shows command invocation without fabricated output values, with note referencing DeepSeek R1 output format"
+ - "Used --etcd-endpoints (hyphens) throughout KVBench docs per README convention, with cross-tool note about NIXLBench underscores"
+
+patterns-established:
+ - "MLA vs MHA/GQA field table pattern: common fields first, then architecture-specific fields in separate tables"
+ - "End-to-end LLM example pattern: model arch YAML + model config YAML + plan command + output + profile command"
+
+requirements-completed: [KB-03, KB-04, KB-05]
+
+# Metrics
+duration: 3min
+completed: 2026-04-07
+---
+
+# Phase 36 Plan 01: KVBench Commands, Model Config, and LLM Examples Summary
+
+**Complete KVBench commands.md page (494 lines) with all 5 subcommand references, CLI argument tables, dual YAML schema documentation, and DeepSeek R1 / Llama 3.1 end-to-end examples**
+
+## Performance
+
+- **Duration:** 3 min
+- **Started:** 2026-04-07T21:10:52Z
+- **Completed:** 2026-04-07T21:13:55Z
+- **Tasks:** 2
+- **Files modified:** 1
+
+## Accomplishments
+- Replaced 7-line stub with 494-line complete page covering all 4 major sections
+- Documented all 5 KVBench subcommands (plan, profile, kvcache, ct-perftest, sequential-ct-perftest) with usage examples and output formats
+- Created 5 CLI argument group tables validated against args.py Click definitions
+- Documented both model architecture YAML schemas (MLA for DeepSeek R1, MHA/GQA for Llama 3.1) and model config YAML schema (strategy/runtime/system)
+- Provided DeepSeek R1 end-to-end examples with both block and layer access patterns showing output differences
+- Provided Llama 3.1 70B end-to-end example with block access pattern
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Author Command Reference and CLI Argument Tables** - `5b503326` (feat)
+2. **Task 2: Author Model Configuration Guide and LLM Examples** - `ed7bc08f` (feat)
+
+## Files Created/Modified
+- `docs/development/benchmarking/kvbench/commands.md` - Complete KVBench commands, model config, and LLM examples page (494 lines, replacing 7-line stub)
+
+## Decisions Made
+- Documented only the CLI arguments present in the README tables (matching D-04 grouping). Backend-specific arguments from args.py (GDS batch settings, GPUNETIO device list, HF3FS pool size, OBJ/S3 credentials) are passthrough to NIXLBench and documented on respective backend pages.
+- For Llama 3.1, showed only the command invocation without fabricated output values. Added a Note component explaining the output follows the same format as DeepSeek R1. This avoids misleading developers with unverified numbers.
+- Used the actual YAML file values from the examples/ directory for model config examples. The README examples show different ISL/page_size values achieved via CLI overrides.
+
+## Deviations from Plan
+
+None - plan executed exactly as written.
+
+## Issues Encountered
+None.
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- KVBench documentation is complete across all 3 pages (index.md, build.md, commands.md)
+- Ready for Phase 37 terminology normalization and quality audit
+- fern check passes with 0 errors
+
+---
+*Phase: 36-kvbench-commands-model-config-and-llm-examples*
+*Completed: 2026-04-07*
diff --git a/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-CONTEXT.md b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-CONTEXT.md
new file mode 100644
index 0000000000..e2a3f9defc
--- /dev/null
+++ b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-CONTEXT.md
@@ -0,0 +1,113 @@
+# Phase 36: KVBench Commands, Model Config, and LLM Examples - Context
+
+**Gathered:** 2026-04-07
+**Status:** Ready for planning
+
+
+## Phase Boundary
+
+Author the KVBench command reference (all 5 subcommands with CLI argument tables), model configuration guide (both YAML schemas), and LLM architecture examples (DeepSeek R1 and Llama 3.1 end-to-end). All content goes in a single `commands.md` page. No CTP-specific examples — those are deferred.
+
+
+
+
+## Implementation Decisions
+
+### Page Structure
+- **D-01:** All content in a single `commands.md` page. Sections in order: (1) Command Reference (all 5 subcommands), (2) Model Configuration Guide (both YAML schemas), (3) LLM Examples (DeepSeek R1 and Llama 3.1). No additional pages — KVBench nav stays at 3 pages (index.md, build.md, commands.md).
+
+### Command Reference
+- **D-02:** Document all 5 subcommands: `plan`, `profile`, `kvcache`, `ct-perftest`, `sequential-ct-perftest`. Each subcommand gets a brief description and its CLI argument table.
+- **D-03:** CLI argument tables use two-column format: Argument | Description. Default values included in the description text (not a separate column). Matches the README's existing format.
+- **D-04:** Arguments grouped by: Common Arguments, CLI Override Arguments, Plan Command Arguments, Shared Benchmark Arguments, CTP Command Arguments. This satisfies REQUIREMENTS KB-03 grouping.
+- **D-05:** CLI tables must be validated against `python main.py [cmd] --help` output. Note that KVBench uses `--etcd-endpoints` (hyphens) while NIXLBench uses `--etcd_endpoints` (underscores) — per REQUIREMENTS QS-03.
+
+### Model Configuration Guide
+- **D-06:** Document both YAML schemas with field description tables + annotated examples:
+ - **Model architecture YAML** (e.g., `model_deepseek_r1.yaml`): fields like `model_name`, `num_layers`, `num_query_heads`, `query_head_dimension`, etc.
+ - **Model config YAML** (e.g., `block-tp1-pp8.yaml`): three sections — `strategy` (tp/pp/quant), `runtime` (isl/osl/requests), `system` (hardware/backend/access_pattern/page_size).
+- **D-07:** Each schema gets a field table followed by a complete annotated YAML example showing real values from the `examples/` directory.
+
+### LLM Examples
+- **D-08:** Full end-to-end examples for DeepSeek R1 and Llama 3.1. Each example shows: (1) the model architecture YAML, (2) the model config YAML, (3) the `plan` command with its output (generated nixlbench command), (4) the `profile` command. Developers can copy-paste and run.
+- **D-09:** Include both block and layer access pattern variants for at least one model to demonstrate the difference.
+
+### Claude's Discretion
+- Whether to include a CTP commands section beyond the CLI table (basic description vs detailed usage)
+- Ordering of subcommands within the reference section
+- Whether to show `kvcache` command output example
+- How many model config variants to include (just tp1-pp8, or also tp8-pp16)
+
+
+
+
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Source material — CLI reference
+- `benchmark/kvbench/README.md` §Command Line Arguments (line ~79) — All CLI argument tables grouped by category
+- `benchmark/kvbench/README.md` §Command Descriptions (line ~162) — Subcommand descriptions and usage
+- `benchmark/kvbench/main.py` — Entry point; validate `--help` output against documented arguments
+
+### Source material — Model config
+- `benchmark/kvbench/docs/creating-a-model-config.md` — Existing developer guide for model config YAML schema
+- `benchmark/kvbench/examples/model_deepseek_r1.yaml` — DeepSeek R1 model architecture config
+- `benchmark/kvbench/examples/model_llama_3_1_70b.yaml` — Llama 3.1 70B model architecture config
+- `benchmark/kvbench/examples/block-tp1-pp8.yaml` — Block access pattern model config
+- `benchmark/kvbench/examples/layer-tp1-pp8.yaml` — Layer access pattern model config
+
+### Source material — Examples
+- `benchmark/kvbench/README.md` §Examples (line ~218) — KVBench and CTP example commands with output
+
+### Doc patterns
+- `docs/development/benchmarking/nixlbench/usage.md` — NIXLBench usage page pattern (Phase 34 output)
+
+### Requirements
+- `.planning/REQUIREMENTS.md` — KB-03 (command reference), KB-04 (model config guide), KB-05 (LLM examples)
+
+
+
+
+## Existing Code Insights
+
+### Reusable Assets
+- `benchmark/kvbench/docs/creating-a-model-config.md` — Existing model config guide; can be adapted (rewrite for Fern style, don't copy verbatim)
+- `benchmark/kvbench/examples/` — 27 example YAML files covering various tp/pp/access pattern combinations
+- Fern frontmatter pattern: `title:` + `description:` on all pages
+
+### Established Patterns
+- Two-column CLI argument tables (Argument | Description) used in the README
+- Three-section model config structure (strategy / runtime / system) is well-established
+- First-mention inline links for backend names (carried from Phase 33 D-09)
+- Rewrite README prose for Fern doc style (carried from Phase 33 D-07)
+
+### Integration Points
+- `commands.md` replaces the stub file created in Phase 32 at `docs/development/benchmarking/kvbench/commands.md`
+- No changes to `docs/index.yml` needed (commands.md entry already exists from Phase 32)
+
+
+
+
+## Specific Ideas
+
+- User wants full end-to-end examples with command output — developers should be able to copy-paste
+- Field tables + annotated examples for both YAML schemas — structured documentation
+- Two-column CLI tables matching README format
+
+
+
+
+## Deferred Ideas
+
+- CTP examples with traffic matrices — deferred per REQUIREMENTS.md
+- KVBench GDS tutorial (`benchmark/kvbench/docs/tutorial-gds.md`) — P2, deferred
+- Adding new model architecture guide (`benchmark/kvbench/docs/adding-a-new-model-architecture.md`) — P2, deferred
+- Additional model examples beyond DeepSeek R1 and Llama 3.1
+
+
+
+---
+
+*Phase: 36-kvbench-commands-model-config-and-llm-examples*
+*Context gathered: 2026-04-07*
diff --git a/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-DISCUSSION-LOG.md b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-DISCUSSION-LOG.md
new file mode 100644
index 0000000000..81a938531a
--- /dev/null
+++ b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-DISCUSSION-LOG.md
@@ -0,0 +1,75 @@
+# Phase 36: KVBench Commands, Model Config, and LLM Examples - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-04-07
+**Phase:** 36-KVBench Commands, Model Config, and LLM Examples
+**Areas discussed:** Page structure, CLI table format, Model config documentation, LLM examples depth
+
+---
+
+## Page Structure
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Single commands.md | All content in one page: commands + model config + examples | ✓ |
+| Two pages | commands.md + examples.md | |
+| Three pages | commands.md + model-config.md + llm-examples.md | |
+
+**User's choice:** Single commands.md
+**Notes:** KVBench content is less voluminous than NIXLBench. Keeps nav at 3 pages.
+
+---
+
+## CLI Table Format
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Argument + description | Two-column table, defaults in description text | ✓ |
+| Argument + description + default | Three-column table with explicit defaults column | |
+| You decide | Claude picks matching existing patterns | |
+
+**User's choice:** Argument + description (two-column)
+**Notes:** Matches README format.
+
+---
+
+## Model Config Documentation
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Field tables + annotated example | Table per schema + complete annotated YAML | ✓ |
+| Annotated examples only | YAML with inline comments | |
+| You decide | Claude picks | |
+
+**User's choice:** Field tables + annotated example
+**Notes:** Two schemas: model architecture and model config.
+
+---
+
+## LLM Examples Depth
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Full end-to-end with output | Plan command + output + profile command, YAML configs inline | ✓ |
+| Commands only | Plan and profile commands with file paths, no output | |
+| You decide | Claude determines completeness | |
+
+**User's choice:** Full end-to-end with output
+**Notes:** Developers can copy-paste and run.
+
+---
+
+## Claude's Discretion
+
+- CTP commands section depth
+- Subcommand ordering
+- kvcache command output example
+- Number of model config variants
+
+## Deferred Ideas
+
+- CTP examples — deferred per REQUIREMENTS.md
+- GDS tutorial — P2, deferred
+- Adding new model architecture guide — P2, deferred
diff --git a/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-RESEARCH.md b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-RESEARCH.md
new file mode 100644
index 0000000000..fae98803bf
--- /dev/null
+++ b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-RESEARCH.md
@@ -0,0 +1,395 @@
+# Phase 36: KVBench Commands, Model Config, and LLM Examples - Research
+
+**Researched:** 2026-04-07
+**Domain:** Technical documentation authoring (Fern MDX, CLI reference, YAML schema docs)
+**Confidence:** HIGH
+
+## Summary
+
+This phase writes the body of `docs/development/benchmarking/kvbench/commands.md`, replacing the stub created in Phase 32. All content -- command reference for 5 subcommands, model configuration guide for both YAML schemas, and end-to-end LLM examples for DeepSeek R1 and Llama 3.1 -- goes on this single page per user decision D-01.
+
+The source material is well-defined: the README contains all CLI argument tables and example outputs, `commands/args.py` is the authoritative Click source for argument definitions, `docs/creating-a-model-config.md` documents the model config schema, and the `examples/` directory provides 27 real YAML files. The NIXLBench usage page (Phase 34 output) establishes the Fern doc style pattern.
+
+**Primary recommendation:** Transcribe CLI tables and YAML schemas directly from source code (`args.py`, `model_config.py`, example YAMLs), rewrite README prose in Fern doc style, and construct end-to-end examples from README examples section. Validate all CLI flags match `args.py` definitions.
+
+
+## User Constraints (from CONTEXT.md)
+
+### Locked Decisions
+- **D-01:** All content in a single `commands.md` page. Sections in order: (1) Command Reference (all 5 subcommands), (2) Model Configuration Guide (both YAML schemas), (3) LLM Examples (DeepSeek R1 and Llama 3.1). No additional pages -- KVBench nav stays at 3 pages (index.md, build.md, commands.md).
+- **D-02:** Document all 5 subcommands: `plan`, `profile`, `kvcache`, `ct-perftest`, `sequential-ct-perftest`. Each subcommand gets a brief description and its CLI argument table.
+- **D-03:** CLI argument tables use two-column format: Argument | Description. Default values included in the description text (not a separate column). Matches the README's existing format.
+- **D-04:** Arguments grouped by: Common Arguments, CLI Override Arguments, Plan Command Arguments, Shared Benchmark Arguments, CTP Command Arguments. This satisfies REQUIREMENTS KB-03 grouping.
+- **D-05:** CLI tables must be validated against `python main.py [cmd] --help` output. Note that KVBench uses `--etcd-endpoints` (hyphens) while NIXLBench uses `--etcd_endpoints` (underscores) -- per REQUIREMENTS QS-03.
+- **D-06:** Document both YAML schemas with field description tables + annotated examples: Model architecture YAML and Model config YAML.
+- **D-07:** Each schema gets a field table followed by a complete annotated YAML example showing real values from the `examples/` directory.
+- **D-08:** Full end-to-end examples for DeepSeek R1 and Llama 3.1. Each example shows: (1) the model architecture YAML, (2) the model config YAML, (3) the `plan` command with its output, (4) the `profile` command. Developers can copy-paste and run.
+- **D-09:** Include both block and layer access pattern variants for at least one model to demonstrate the difference.
+
+### Claude's Discretion
+- Whether to include a CTP commands section beyond the CLI table (basic description vs detailed usage)
+- Ordering of subcommands within the reference section
+- Whether to show `kvcache` command output example
+- How many model config variants to include (just tp1-pp8, or also tp8-pp16)
+
+### Deferred Ideas (OUT OF SCOPE)
+- CTP examples with traffic matrices -- deferred per REQUIREMENTS.md
+- KVBench GDS tutorial -- P2, deferred
+- Adding new model architecture guide -- P2, deferred
+- Additional model examples beyond DeepSeek R1 and Llama 3.1
+
+
+
+## Phase Requirements
+
+| ID | Description | Research Support |
+|----|-------------|------------------|
+| KB-03 | KVBench command reference covers all 5 subcommands with CLI argument tables validated against `--help` output; arguments grouped by Common / CLI Override / Per-command | README CLI tables + `commands/args.py` Click decorators provide authoritative source; grouping matches README exactly |
+| KB-04 | KVBench model configuration guide documents model architecture YAML schema and model config YAML schema with field descriptions and examples | `models/model_config.py` dataclasses + `docs/creating-a-model-config.md` + example YAMLs provide complete schema coverage |
+| KB-05 | KVBench LLM examples page covers DeepSeek R1 and Llama 3.1 example configurations with end-to-end `plan` and `profile` command examples | README examples section has DeepSeek R1 block/layer examples with full command output; Llama 3.1 YAML available in examples/ |
+
+
+## Architecture Patterns
+
+### Target File Structure
+```
+docs/development/benchmarking/kvbench/commands.md # Replace stub (7 lines) with full content
+```
+
+No other files created or modified. The `docs/index.yml` entry already exists (line 86: `path: development/benchmarking/kvbench/commands.md`). [VERIFIED: codebase grep]
+
+### Page Structure Pattern (from D-01)
+```markdown
+---
+title: KVBench Commands and Examples
+description: ...
+---
+
+[Intro paragraph]
+
+## Command Reference
+### plan
+### profile
+### kvcache
+### ct-perftest
+### sequential-ct-perftest
+
+## Command Line Arguments
+### Common Arguments [table]
+### CLI Override Arguments [table]
+### Plan Command Arguments [table]
+### Shared Benchmark Arguments [table]
+### CTP Command Arguments [table]
+
+## Model Configuration Guide
+### Model Architecture YAML [schema table + example]
+### Model Config YAML [schema table + example]
+
+## LLM Examples
+### DeepSeek R1 [end-to-end]
+### Llama 3.1 [end-to-end]
+```
+
+### Fern Doc Style Pattern
+From `docs/development/benchmarking/nixlbench/usage.md` (Phase 34): [VERIFIED: codebase read]
+- Fern frontmatter: `title:` + `description:`
+- First paragraph references related pages with relative links
+- Two-column or three-column tables for CLI flags
+- Code blocks with `bash` language tag
+- First-mention inline links for backend names (Phase 33 D-09 pattern)
+- ``, `` Fern components where needed
+- No HTML comments, no bare anchor links (QS-04)
+
+### Anti-Patterns to Avoid
+- **Duplicating build instructions:** Link to `./kvbench/build` instead
+- **Re-explaining etcd setup:** Link to `/docs/user-guide/etcd-metadata-exchange`
+- **Copying README verbatim:** Rewrite prose for Fern doc style (D-07 from Phase 33)
+- **Ignoring the etcd flag discrepancy:** KVBench uses `--etcd-endpoints` (hyphens), NIXLBench uses `--etcd_endpoints` (underscores) -- must document per QS-03
+
+## Source Material Inventory
+
+### CLI Argument Tables (for KB-03)
+
+Five argument groups from `benchmark/kvbench/README.md` lines 79-158 and `commands/args.py`: [VERIFIED: codebase read]
+
+| Group | Defined In | Arg Count | Used By Commands |
+|-------|-----------|-----------|-----------------|
+| Common Arguments | `common_args()` decorator | 3 | plan, profile, kvcache |
+| CLI Override Arguments | `cli_args()` decorator | 9 (includes `--source`, `--destination`) | plan, profile, kvcache |
+| Plan Command Arguments | `plan_args()` decorator | 1 | plan only |
+| Shared Benchmark Arguments | `nixl_bench_args()` decorator | 40+ | plan, profile |
+| CTP Command Arguments | inline in main.py | 3-4 | ct-perftest, sequential-ct-perftest |
+
+**Critical finding -- args.py has MORE arguments than the README documents.** The README's "Shared Benchmark Arguments" table lists ~30 arguments, but `args.py:nixl_bench_args()` defines additional arguments not in the README: [VERIFIED: codebase read]
+- `--posix_api_type` (POSIX backend API type)
+- `--benchmark_group` (parallel benchmark group name)
+- `--num_files` (file count for storage)
+- `--large_blk_iter_ftr` (large block iteration factor)
+- `--gds_batch_pool_size`, `--gds_batch_limit` (GDS backend)
+- `--gds_mt_num_threads` (GDS MT backend)
+- `--gpunetio_device_list`, `--gpunetio_oob_list` (GPUNETIO backend)
+- `--hf3fs_iopool_size` (HF3FS backend)
+- `--obj_access_key`, `--obj_secret_key`, `--obj_session_token`, `--obj_bucket_name`, `--obj_scheme`, `--obj_region`, `--obj_use_virtual_addressing`, `--obj_endpoint_override`, `--obj_req_checksum`, `--obj_ca_bundle` (OBJ/S3 backend)
+
+**Recommendation:** Document the arguments from the README tables (which match the user-facing documentation scope). Backend-specific arguments (GDS, GPUNETIO, HF3FS, OBJ) are passed through to nixlbench and are documented on respective backend pages. The planner should note this scope boundary.
+
+### etcd Flag Discrepancy (QS-03)
+- `args.py` line 162: `--etcd_endpoints` (underscores) -- Click accepts both forms [VERIFIED: codebase read]
+- README examples: `--etcd-endpoints` (hyphens) [VERIFIED: codebase read]
+- NIXLBench: `--etcd_endpoints` (underscores) [VERIFIED: Phase 34 output]
+- Click's behavior: underscored option names accept both `--etcd_endpoints` and `--etcd-endpoints` on the command line [ASSUMED]
+- **Action:** Document as `--etcd-endpoints` in KVBench docs (matching README convention) and note the difference from NIXLBench's `--etcd_endpoints`
+
+### Model Architecture YAML Schema (for KB-04)
+
+Two distinct schema patterns based on attention mechanism: [VERIFIED: codebase read]
+
+**DeepSeek R1 (MLA -- Multi-Latent Attention):**
+```yaml
+model_name: 'DEEPSEEK_R1'
+num_layers: 61
+num_query_heads: 128
+query_head_dimension: 128
+embedding_dimension: 7168
+rope_mla_dimension: 64
+mla_latent_vector_dimension: 512
+num_model_params: 671000000000
+```
+
+**Llama 3.1 70B (MHA/GQA -- Multi-Head / Grouped-Query Attention):**
+```yaml
+model_name: 'LLAMA3.1_70B'
+num_layers: 80
+num_query_heads_with_mha: 64
+query_head_dimension: 128
+gqa_num_queries_in_group: 8
+num_model_params: 70000000000
+```
+
+Key schema differences:
+- DeepSeek R1 uses `num_query_heads` + MLA-specific fields (`rope_mla_dimension`, `mla_latent_vector_dimension`, `embedding_dimension`)
+- Llama 3.1 uses `num_query_heads_with_mha` + GQA field (`gqa_num_queries_in_group`)
+- Both share: `model_name`, `num_layers`, `query_head_dimension`, `num_model_params`
+
+### Model Config YAML Schema (for KB-04)
+
+Three sections with dataclass definitions in `models/model_config.py`: [VERIFIED: codebase read]
+
+| Section | Field | Type | Default | Description |
+|---------|-------|------|---------|-------------|
+| strategy | tp_size | int | 1 | Tensor parallelism size |
+| strategy | pp_size | int | 1 | Pipeline parallelism size |
+| strategy | model_quant_mode | str | "fp8" | Model weight quantization mode |
+| strategy | kvcache_quant_mode | str | "fp8" | KV cache quantization mode |
+| runtime | num_requests | int | 1 | Number of inference requests |
+| runtime | isl | int | 1 | Input sequence length |
+| runtime | osl | int | 1 | Output sequence length |
+| system | backend | str | "SGLANG" | Inference backend engine |
+| system | hardware | str | None | Hardware platform (e.g., "H100") |
+| system | page_size | int | 1 | Page size for access pattern |
+| system | access_pattern | str | None | KV cache access pattern ("block" or "layer") |
+| system | source | str | None | Source descriptor type |
+| system | destination | str | None | Destination descriptor type |
+
+Note: The YAML key is `strategy` but `ModelConfig` stores it as `model` attribute (line 81: `model: StrategyConfig`). The YAML-to-Python mapping is: `strategy` -> `config.model`, `runtime` -> `config.runtime`, `system` -> `config.system`. [VERIFIED: model_config.py lines 162-173]
+
+### LLM Examples (for KB-05)
+
+The README provides three DeepSeek R1 examples with full command output: [VERIFIED: codebase read]
+1. DeepSeek R1 with Block Access (TP=1, PP=16) -- lines 222-244
+2. DeepSeek R1 with Layer Access (TP=1, PP=16) -- lines 246-269
+3. CLI Override example (TP=1, PP=8, overriding to PP=32) -- lines 271-296
+
+**Missing from README:** Llama 3.1 examples. The YAML exists (`model_llama_3_1_70b.yaml`) but no example command output in README.
+
+**Recommendation for D-08/D-09:**
+- DeepSeek R1: Use block-tp1-pp8 example with plan+profile, then show layer-tp1-pp8 variant (satisfies D-09: both access patterns)
+- Llama 3.1: Construct a plan example using `model_llama_3_1_70b.yaml` + `block-tp1-pp8.yaml`. The command output cannot be verified without running the tool, so adapt the output format pattern from the DeepSeek examples.
+
+## Don't Hand-Roll
+
+| Problem | Don't Build | Use Instead | Why |
+|---------|-------------|-------------|-----|
+| CLI argument descriptions | Writing descriptions from scratch | Copy from `args.py` help strings and README tables | Authoritative source, avoids drift |
+| Model config field descriptions | Inventing descriptions | Copy from `model_config.py` docstrings and `creating-a-model-config.md` | Already documented accurately |
+| Example command output | Guessing nixlbench output format | Copy from README examples section (lines 218-296) | Exact output format matters for copy-paste |
+| YAML schema structure | Describing from memory | Read actual YAML files from `examples/` directory | Real files are the schema specification |
+
+## Common Pitfalls
+
+### Pitfall 1: README CLI Table Drift from Source Code
+**What goes wrong:** README documents `--etcd-endpoints` but `args.py` defines `--etcd_endpoints`. Some arguments in `args.py` are missing from README entirely.
+**Why it happens:** README and code evolve independently.
+**How to avoid:** Use `args.py` as the authoritative source for argument names and help text. Cross-reference with README for grouping and defaults. Flag any discrepancies.
+**Warning signs:** Argument not found in `--help` output during QS-03 validation.
+
+### Pitfall 2: Model Architecture Schema is Model-Specific
+**What goes wrong:** Documenting a single "model architecture YAML schema" when different models have different fields.
+**Why it happens:** DeepSeek R1 uses MLA fields, Llama uses MHA/GQA fields. They share some common fields but diverge.
+**How to avoid:** Document common fields first, then model-specific fields in separate subsections or examples. Be explicit about which fields apply to which architecture.
+**Warning signs:** A field table that claims to be universal but only matches one model.
+
+### Pitfall 3: Llama 3.1 Example Output Cannot Be Verified
+**What goes wrong:** Writing example `plan` command output for Llama 3.1 that doesn't match actual tool output.
+**Why it happens:** README only has DeepSeek R1 examples. Cannot run the tool in this environment.
+**How to avoid:** Either (a) construct the example with a clear note about the format being based on DeepSeek R1 patterns, or (b) show only the command invocation without output. Phase 37 QS-03 validation will catch output mismatches.
+**Warning signs:** Batch size / block size numbers that don't make sense for Llama 3.1 70B architecture.
+
+### Pitfall 4: YAML Key vs Python Attribute Name Mismatch
+**What goes wrong:** Documenting `model:` as a YAML section when the YAML key is `strategy:`.
+**Why it happens:** `ModelConfig.model` attribute stores strategy data, but YAML uses `strategy:` key.
+**How to avoid:** Document the YAML keys (`strategy`, `runtime`, `system`), not the Python attributes.
+**Warning signs:** Example YAML that uses `model:` instead of `strategy:`.
+
+## Code Examples
+
+### CLI Argument Table Format (from README, two-column per D-03)
+```markdown
+| Argument | Description |
+| -------- | ----------- |
+| `--model` | Path to a model architecture config YAML file |
+| `--model_config` | Path to a model config YAML file |
+```
+Source: `benchmark/kvbench/README.md` lines 85-89 [VERIFIED: codebase read]
+
+### Model Config Annotated YAML (for D-07)
+```yaml
+# Model config: block access, TP=1, PP=8
+strategy:
+ tp_size: 1 # Tensor parallelism -- GPUs for tensor-parallel execution
+ pp_size: 8 # Pipeline parallelism -- GPUs for pipeline-parallel execution
+ model_quant_mode: "fp8" # Model weight quantization (fp8, fp16, int8)
+ kvcache_quant_mode: "fp8" # KV cache quantization (fp8, fp16, int8)
+
+runtime:
+ isl: 1000 # Input sequence length (tokens)
+ osl: 100 # Output sequence length (tokens)
+ num_requests: 10 # Number of inference requests
+
+system:
+ hardware: "H100" # Hardware platform
+ backend: "SGLANG" # Inference backend engine
+ access_pattern: "block" # KV cache access pattern (block or layer)
+ page_size: 16 # Page size for block access pattern
+```
+Source: `benchmark/kvbench/examples/block-tp1-pp8.yaml` + `docs/creating-a-model-config.md` [VERIFIED: codebase read]
+
+### End-to-End Example Pattern (for D-08)
+```markdown
+#### DeepSeek R1 -- Block Access (TP=1, PP=8)
+
+**Model architecture** (`model_deepseek_r1.yaml`):
+[show YAML]
+
+**Model config** (`block-tp1-pp8.yaml`):
+[show YAML]
+
+**Plan command:**
+```bash
+python main.py plan \
+ --model ./examples/model_deepseek_r1.yaml \
+ --model_config ./examples/block-tp1-pp8.yaml \
+ --backend GDS \
+ --source gpu \
+ --etcd-endpoints "http://localhost:2379"
+```
+
+**Output:**
+[show command output from README]
+
+**Profile command:**
+```bash
+python main.py profile \
+ --model ./examples/model_deepseek_r1.yaml \
+ --model_config ./examples/block-tp1-pp8.yaml \
+ --backend GDS \
+ --source gpu \
+ --etcd-endpoints "http://localhost:2379"
+```
+```
+Source: `benchmark/kvbench/README.md` lines 218-296 [VERIFIED: codebase read]
+
+## Discretion Recommendations
+
+Based on the discretion areas from CONTEXT.md:
+
+1. **CTP commands section depth:** Keep it minimal -- brief description for each CTP subcommand plus the CLI table. CTP examples are explicitly deferred. Include the Sequential CT Perftest and CT Perftest report format descriptions from README lines 195-215 as they help users understand what the commands do.
+
+2. **Subcommand ordering:** Follow the README order: plan, profile, kvcache (KVBench commands first), then ct-perftest, sequential-ct-perftest (CTP commands). This matches the two-category structure from the index page.
+
+3. **kvcache command output example:** Yes, include it. The README has a complete example with tabulated output (line 186-189). It's 4 lines and helps developers understand what kvcache shows.
+
+4. **Model config variants:** Include tp1-pp8 as the primary example (simpler, matches build page examples). The LLM examples section uses tp1-pp16 (from README examples) and shows block vs layer variants, which is sufficient variety. No need for tp8-pp16 -- it would add length without new concepts.
+
+## Validation Architecture
+
+### Test Framework
+| Property | Value |
+|----------|-------|
+| Framework | Fern CLI |
+| Config file | `fern/fern.config.json` |
+| Quick run command | `cd fern && fern check` |
+| Full suite command | `cd fern && fern check` |
+
+### Phase Requirements to Test Map
+| Req ID | Behavior | Test Type | Automated Command | File Exists? |
+|--------|----------|-----------|-------------------|-------------|
+| KB-03 | CLI argument tables present for all 5 subcommands | manual review + fern check | `cd fern && fern check` | N/A (content review) |
+| KB-04 | Both YAML schemas documented with field tables and examples | manual review | visual inspection of commands.md | N/A (content review) |
+| KB-05 | DeepSeek R1 and Llama 3.1 end-to-end examples present | manual review | visual inspection of commands.md | N/A (content review) |
+
+### Sampling Rate
+- **Per task commit:** `cd fern && fern check`
+- **Per wave merge:** Full fern check
+- **Phase gate:** Fern check green + manual review of content completeness
+
+### Wave 0 Gaps
+None -- existing Fern infrastructure covers build validation. Content accuracy requires manual review against source material (Phase 37 QS-03 handles formal CLI validation).
+
+## Assumptions Log
+
+| # | Claim | Section | Risk if Wrong |
+|---|-------|---------|---------------|
+| A1 | Click accepts both `--etcd_endpoints` and `--etcd-endpoints` interchangeably when option defined with underscores | Source Material Inventory | Low -- if wrong, examples using hyphens would fail; easily tested |
+| A2 | Llama 3.1 plan command output follows the same format as DeepSeek R1 (header block + nixlbench command) | Common Pitfalls / Pitfall 3 | Medium -- if output format differs, example would be misleading; Phase 37 QS-03 catches this |
+
+## Open Questions
+
+1. **Llama 3.1 example output values**
+ - What we know: The command invocation is straightforward (`python main.py plan --model ./examples/model_llama_3_1_70b.yaml --model_config ./examples/block-tp1-pp8.yaml`)
+ - What's unclear: The exact batch_size and block_size values in the generated nixlbench command
+ - Recommendation: Use the DeepSeek R1 example format but note that specific output values depend on the model architecture. If possible, annotate the example as "representative output" or omit the output block for Llama 3.1.
+
+2. **Backend-specific arguments scope**
+ - What we know: `args.py` has ~15 backend-specific arguments (GDS, GPUNETIO, HF3FS, OBJ) not in README tables
+ - What's unclear: Whether users expect these in KVBench docs or on backend pages
+ - Recommendation: Document only the arguments from the README tables (matching D-04 grouping). Backend-specific args are passthrough to nixlbench and documented elsewhere.
+
+## Sources
+
+### Primary (HIGH confidence)
+- `benchmark/kvbench/README.md` -- CLI tables, command descriptions, examples
+- `benchmark/kvbench/commands/args.py` -- Authoritative Click CLI definitions
+- `benchmark/kvbench/models/model_config.py` -- Model config dataclass schema
+- `benchmark/kvbench/docs/creating-a-model-config.md` -- Model config guide
+- `benchmark/kvbench/examples/*.yaml` -- All 27 example YAML files
+- `benchmark/kvbench/main.py` -- Entry point, command registration, kvcache output
+- `docs/development/benchmarking/nixlbench/usage.md` -- Phase 34 Fern doc style pattern
+- `docs/development/benchmarking/kvbench/commands.md` -- Existing stub (7 lines)
+- `docs/index.yml` line 86 -- Nav entry already registered
+
+### Secondary (MEDIUM confidence)
+- None needed -- all source material is in the codebase
+
+### Tertiary (LOW confidence)
+- None
+
+## Metadata
+
+**Confidence breakdown:**
+- Standard stack: HIGH -- this is documentation authoring, no libraries needed
+- Architecture: HIGH -- page structure locked by user decisions, source material fully inventoried
+- Pitfalls: HIGH -- identified from direct source code analysis (args.py vs README drift, schema differences)
+
+**Research date:** 2026-04-07
+**Valid until:** 2026-05-07 (stable -- source code changes would require re-research)
diff --git a/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-VERIFICATION.md b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-VERIFICATION.md
new file mode 100644
index 0000000000..18106f2301
--- /dev/null
+++ b/.planning/phases/36-kvbench-commands-model-config-and-llm-examples/36-VERIFICATION.md
@@ -0,0 +1,78 @@
+---
+phase: 36-kvbench-commands-model-config-and-llm-examples
+verified: 2026-04-07T22:15:00Z
+status: passed
+score: 5/5
+overrides_applied: 0
+---
+
+# Phase 36: KVBench Commands, Model Config, and LLM Examples Verification Report
+
+**Phase Goal:** Developers can look up any KVBench subcommand, understand the model configuration YAML schema, and run end-to-end examples for DeepSeek R1 and Llama 3.1
+**Verified:** 2026-04-07T22:15:00Z
+**Status:** passed
+**Re-verification:** No -- independent verification (previous VERIFICATION.md existed with passed status, no gaps)
+
+## Goal Achievement
+
+### Observable Truths
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | Developer can look up any of the 5 KVBench subcommands and see its description and CLI arguments | VERIFIED | All 5 subcommands have dedicated `###` subsections: `plan` (L10), `profile` (L24), `kvcache` (L37), `ct-perftest` (L57), `sequential-ct-perftest` (L71). Each has description, bash usage example, and output format where applicable. 5 CLI argument group tables under `## Command Line Arguments` (L93): Common (L95, 3 args), CLI Override (L105, 7 args), Plan Command (L119, 1 arg), Shared Benchmark (L127, 30 args), CTP (L167, 4 args). |
+| 2 | Developer can understand the model architecture YAML schema (both MLA and MHA/GQA variants) from field tables and annotated examples | VERIFIED | `### Model Architecture YAML` (L182): Common fields table (4 fields), MLA fields table (4 fields incl. `mla_latent_vector_dimension` L202), MHA/GQA fields table (2 fields incl. `gqa_num_queries_in_group` L209). Annotated YAML examples for DeepSeek R1 (L213-222) and Llama 3.1 70B (L226-233) with inline comments. DeepSeek R1 values cross-checked against `benchmark/kvbench/examples/model_deepseek_r1.yaml` -- exact match confirmed. |
+| 3 | Developer can understand the model config YAML schema (strategy/runtime/system) from field tables and annotated examples | VERIFIED | `### Model Config YAML` (L235): Strategy (4 fields L241-246), Runtime (3 fields L250-254), System (6 fields L259-265). Annotated block access example (L269-286) with all three YAML sections and inline comments. Note component (L288-290) explains block vs layer access patterns. |
+| 4 | Developer can copy-paste DeepSeek R1 end-to-end example (both block and layer access patterns) and run it | VERIFIED | `### DeepSeek R1` (L296): `#### Block Access (TP=1, PP=16)` (L298) has model arch YAML, model config YAML, plan command, output, and profile command. `#### Layer Access (TP=1, PP=16)` (L376) has model config YAML, plan command, output, and profile command. Both use `--etcd-endpoints` (hyphens). |
+| 5 | Developer can copy-paste Llama 3.1 end-to-end example and run it | VERIFIED | `### Llama 3.1 70B` (L434): `#### Block Access (TP=1, PP=8)` (L436) has model arch YAML, model config YAML, plan command, Note about output format, and profile command. Output not fabricated -- references DeepSeek R1 format instead. |
+
+**Score:** 5/5 truths verified
+
+### Required Artifacts
+
+| Artifact | Expected | Status | Details |
+|----------|----------|--------|---------|
+| `docs/development/benchmarking/kvbench/commands.md` | Complete KVBench commands, model config, and LLM examples page, 300+ lines, contains `## Command Reference` | VERIFIED | 494 lines (L1 exists, L2 substantive, L3 wired). All 4 major sections present: Command Reference, Command Line Arguments, Model Configuration Guide, LLM Examples. Fern frontmatter preserved. No TODOs/FIXMEs/placeholders found. YAML field values cross-checked against source repository files -- exact match. |
+
+### Key Link Verification
+
+| From | To | Via | Status | Details |
+|------|----|-----|--------|---------|
+| `commands.md` L6 | `nixlbench/index.md` | `[NIXLBench](./nixlbench)` | WIRED | Link present at L6. Target file confirmed to exist at `docs/development/benchmarking/nixlbench/index.md`. |
+| `commands.md` L6 | `kvbench/build.md` | `[Building KVBench](./kvbench/build)` | WIRED | Link present at L6. Target file confirmed to exist at `docs/development/benchmarking/kvbench/build.md`. |
+| `commands.md` L135 | `docs/user-guide/backends/*` | Backend name inline links | WIRED | 8 backend links found in `--backend` row: UCX, GDS, GDS_MT, POSIX, GPUNETIO, Mooncake, HF3FS, OBJ -- all using `/docs/user-guide/backends/` prefix. |
+| `commands.md` | `docs/index.yml` | Nav registration | WIRED | Registered per PLAN (index.yml line 85-86). |
+
+### Data-Flow Trace (Level 4)
+
+Not applicable -- static documentation, no dynamic data rendering.
+
+### Behavioral Spot-Checks
+
+Step 7b: SKIPPED (documentation-only phase, no runnable entry points).
+
+### Requirements Coverage
+
+| Requirement | Source Plan | Description | Status | Evidence |
+|-------------|------------|-------------|--------|----------|
+| KB-03 | 36-01-PLAN | KVBench command reference covers all 5 subcommands with CLI argument tables, grouped by Common / CLI Override / Per-command | SATISFIED | All 5 subcommands documented with dedicated subsections. 5 argument group tables with correct grouping. `--etcd-endpoints` uses hyphens with cross-tool note (L163-165). |
+| KB-04 | 36-01-PLAN | KVBench model configuration guide documents model architecture and model config YAML schemas with field descriptions and examples | SATISFIED | Model Architecture YAML: 3 field tables (Common, MLA, MHA/GQA) + 2 annotated examples. Model Config YAML: 3 field tables (Strategy, Runtime, System) + 1 annotated example + block/layer note. |
+| KB-05 | 36-01-PLAN | KVBench LLM examples covers DeepSeek R1 and Llama 3.1 with end-to-end plan and profile command examples | SATISFIED | DeepSeek R1: block + layer access (both with plan, output, profile). Llama 3.1 70B: block access (plan + profile + format note). All examples include model arch YAML + model config YAML + commands. |
+
+### Anti-Patterns Found
+
+| File | Line | Pattern | Severity | Impact |
+|------|------|---------|----------|--------|
+| (none) | - | - | - | No TODOs, FIXMEs, placeholders, or stub patterns found |
+
+### Human Verification Required
+
+No human verification items identified. All truths are verifiable through content inspection.
+
+### Gaps Summary
+
+No gaps found. All 5 observable truths verified. All 3 requirements (KB-03, KB-04, KB-05) satisfied. The single artifact (`commands.md`) is substantive at 494 lines with YAML values cross-checked against source repository files, all key links are wired to confirmed-existing target files, and no anti-patterns were detected.
+
+---
+
+_Verified: 2026-04-07T22:15:00Z_
+_Verifier: Claude (gsd-verifier)_
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-01-PLAN.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-01-PLAN.md
new file mode 100644
index 0000000000..b3ec28891d
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-01-PLAN.md
@@ -0,0 +1,221 @@
+---
+phase: 37-terminology-normalization-and-quality-audit
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+ - docs/user-guide/benchmarking/nixlbench/index.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+ - docs/user-guide/benchmarking/nixlbench/usage.md
+ - docs/user-guide/benchmarking/kvbench/index.md
+ - docs/user-guide/benchmarking/kvbench/build.md
+ - docs/user-guide/benchmarking/kvbench/commands.md
+autonomous: true
+requirements: [QS-01, QS-02, QS-03]
+
+must_haves:
+ truths:
+ - "Zero instances of 'plugin' (without hyphen) in prose across all 6 pages"
+ - "Zero instances of 'ETCD' in prose text (code/CLI contexts excluded)"
+ - "KV cache written as two words in prose, kvcache only in code/CLI"
+ - "NIXLBench and KVBench always camelCase in prose"
+ - "InfiniBand always camelCase (not Infiniband or infiniband)"
+ - "Backend names ALL CAPS in tables/CLI contexts"
+ - "No duplicated content — build steps, etcd setup, backend configs use cross-links"
+ - "NIXLBench CLI tables use --etcd_endpoints (underscores)"
+ - "KVBench CLI tables use --etcd-endpoints (hyphens)"
+ - "All cross-links from new pages to existing pages use valid relative paths"
+ artifacts:
+ - path: "docs/user-guide/benchmarking/nixlbench/index.md"
+ provides: "NIXLBench overview — terminology normalized"
+ - path: "docs/user-guide/benchmarking/nixlbench/build.md"
+ provides: "NIXLBench build — terminology normalized"
+ - path: "docs/user-guide/benchmarking/nixlbench/usage.md"
+ provides: "NIXLBench usage — terminology normalized"
+ - path: "docs/user-guide/benchmarking/kvbench/index.md"
+ provides: "KVBench overview — terminology normalized"
+ - path: "docs/user-guide/benchmarking/kvbench/build.md"
+ provides: "KVBench build — terminology normalized"
+ - path: "docs/user-guide/benchmarking/kvbench/commands.md"
+ provides: "KVBench commands — terminology normalized"
+ key_links:
+ - from: "all 6 pages"
+ to: "docs/user-guide/etcd-metadata-exchange.md"
+ via: "relative link on first etcd mention"
+ pattern: "etcd.*metadata"
+ - from: "all 6 pages"
+ to: "docs/user-guide/backends/*.md"
+ via: "first-mention inline links for backend names"
+ pattern: "backends/"
+---
+
+
+Audit all 6 benchmarking pages for terminology drift, content duplication, CLI flag accuracy, and cross-link validity. Fix all issues inline.
+
+Purpose: Ensure new pages match v1.0 docs conventions before milestone close.
+Output: All 6 pages passing terminology, duplication, CLI, and cross-link checks.
+
+
+
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/workflows/execute-plan.md
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/templates/summary.md
+
+
+
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/37-terminology-normalization-and-quality-audit/37-CONTEXT.md
+
+@docs/user-guide/benchmarking/nixlbench/index.md
+@docs/user-guide/benchmarking/nixlbench/build.md
+@docs/user-guide/benchmarking/nixlbench/usage.md
+@docs/user-guide/benchmarking/kvbench/index.md
+@docs/user-guide/benchmarking/kvbench/build.md
+@docs/user-guide/benchmarking/kvbench/commands.md
+
+
+
+
+
+ Task 1: Terminology grep audit and inline fixes
+
+ - docs/user-guide/benchmarking/nixlbench/index.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+ - docs/user-guide/benchmarking/nixlbench/usage.md
+ - docs/user-guide/benchmarking/kvbench/index.md
+ - docs/user-guide/benchmarking/kvbench/build.md
+ - docs/user-guide/benchmarking/kvbench/commands.md
+ - docs/user-guide/etcd-metadata-exchange.md (terminology baseline)
+ - docs/user-guide/backends/ucx.md (terminology baseline)
+
+
+ docs/user-guide/benchmarking/nixlbench/index.md
+ docs/user-guide/benchmarking/nixlbench/build.md
+ docs/user-guide/benchmarking/nixlbench/usage.md
+ docs/user-guide/benchmarking/kvbench/index.md
+ docs/user-guide/benchmarking/kvbench/build.md
+ docs/user-guide/benchmarking/kvbench/commands.md
+
+
+Run grep passes across all 6 benchmarking pages for each terminology rule (per D-01). Fix all violations inline:
+
+1. **plugin -> plug-in**: Grep for `plugin` (case-insensitive) in prose text. Replace with `plug-in` unless inside a code block, CLI command, or file path. Do NOT change code-context uses like `backend_plugin` or code references.
+
+2. **ETCD -> etcd in prose**: Grep for `ETCD` outside of code blocks, CLI flag values, and environment variable names. Replace with `etcd` in prose. Keep `ETCD` in: CLI flags like `--etcd_endpoints`, environment variables, code blocks, table cells showing CLI values.
+
+3. **KV cache vs kvcache**: Grep for `kvcache` and `KVcache` and `kv cache` in prose. Prose must use `KV cache` (two words, KV uppercase). The literal `kvcache` is correct only when referring to the CLI subcommand name (e.g., `kvcache` subcommand).
+
+4. **NIXLBench casing**: Grep for `NIXL Bench`, `nixlbench` (in prose, not CLI), `Nixlbench`, `NixlBench`. Prose must use `NIXLBench`. CLI command references like `nixlbench` are fine.
+
+5. **KVBench casing**: Same pattern — grep for `KV Bench`, `kvbench` (in prose), `Kvbench`, `KvBench`. Prose must use `KVBench`.
+
+6. **InfiniBand casing**: Grep for `Infiniband`, `infiniband`, `INFINIBAND`. Must be `InfiniBand` in prose. Exception: `ibverbs` and similar compound terms stay lowercase.
+
+7. **Backend names in tables**: Verify backend names (UCX, GDS, POSIX, MOCHIBI, etc.) are ALL CAPS in tables and CLI contexts.
+
+8. **Content duplication check (per D-02)**: Scan for any paragraphs that re-explain etcd setup, NIXL build steps, or backend configuration instead of cross-linking. If found, replace with a cross-link to the source page: etcd setup -> `docs/user-guide/etcd-metadata-exchange.md`, build steps -> `docs/user-guide/building-nixl/index.md`, backends -> `docs/user-guide/backends/{name}.md`.
+
+9. **CLI flag validation (per D-04, D-05)**: Read `benchmark/nixlbench/README.md` and `benchmark/kvbench/README.md` for authoritative flag names. Verify:
+ - NIXLBench pages use `--etcd_endpoints` (underscores) — NOT `--etcd-endpoints`
+ - KVBench pages use `--etcd-endpoints` (hyphens) — NOT `--etcd_endpoints`
+ - Any other flag mismatches between docs and README are fixed
+
+For each grep, use ripgrep or similar to search. Fix each violation by editing the file directly.
+
+
+ - grep -rn "plugin" across all 6 files returns zero prose matches (code-context matches OK)
+ - grep -rn "ETCD" across all 6 files returns zero prose-context matches
+ - grep -rn "kvcache" in prose (outside code blocks) returns zero matches
+ - grep -rn "NIXL Bench\|Nixlbench\|NixlBench" returns zero matches
+ - grep -rn "KV Bench\|Kvbench\|KvBench" returns zero matches
+ - grep -rn "Infiniband\b\|infiniband\b\|INFINIBAND" returns zero matches
+ - NIXLBench files contain --etcd_endpoints (underscores), not --etcd-endpoints
+ - KVBench files contain --etcd-endpoints (hyphens), not --etcd_endpoints
+
+
+ cd /home/omrik/Projects/nixl.gitlab && echo "=== plugin check ===" && grep -rn '\bplugin\b' docs/user-guide/benchmarking/ --include="*.md" | grep -v '```' | grep -v 'backend_plugin' | grep -v '_plugin' || echo "PASS: no plugin drift" && echo "=== ETCD prose check ===" && grep -rn '\bETCD\b' docs/user-guide/benchmarking/ --include="*.md" | grep -v '```' | grep -v '\-\-' | grep -v 'ETCD_' || echo "PASS: no ETCD drift" && echo "=== InfiniBand check ===" && grep -rn 'Infiniband\|infiniband\|INFINIBAND' docs/user-guide/benchmarking/ --include="*.md" || echo "PASS: InfiniBand OK" && echo "=== NIXLBench etcd flag ===" && grep -rn 'etcd.endpoints' docs/user-guide/benchmarking/nixlbench/ --include="*.md" | head -5 && echo "=== KVBench etcd flag ===" && grep -rn 'etcd.endpoints' docs/user-guide/benchmarking/kvbench/ --include="*.md" | head -5
+
+
+
+
+ Task 2: Cross-link audit and verification
+
+ - docs/user-guide/benchmarking/nixlbench/index.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+ - docs/user-guide/benchmarking/nixlbench/usage.md
+ - docs/user-guide/benchmarking/kvbench/index.md
+ - docs/user-guide/benchmarking/kvbench/build.md
+ - docs/user-guide/benchmarking/kvbench/commands.md
+ - docs/index.yml
+
+
+ docs/user-guide/benchmarking/nixlbench/index.md
+ docs/user-guide/benchmarking/nixlbench/build.md
+ docs/user-guide/benchmarking/nixlbench/usage.md
+ docs/user-guide/benchmarking/kvbench/index.md
+ docs/user-guide/benchmarking/kvbench/build.md
+ docs/user-guide/benchmarking/kvbench/commands.md
+
+
+Audit every cross-link in the 6 benchmarking pages (per D-08, D-09, D-10):
+
+1. **Extract all markdown links**: Find every `[text](path)` pattern in each of the 6 files. List them all.
+
+2. **Verify link targets exist**: For each relative link (e.g., `../../user-guide/backends/ucx.md`), resolve the path from the source file's directory and verify the target file exists on disk. If the link uses Fern's path format (e.g., `/docs/user-guide/backends/ucx`) instead of relative markdown paths, verify the corresponding .md file exists under the docs/ tree.
+
+3. **Backend first-mention links (per D-08)**: For each backend name (UCX, GDS, POSIX, etc.) mentioned in prose across the 6 pages, verify the FIRST mention per page is an inline link to the corresponding User Guide backend page. If any first mention is unlinked, add the link.
+
+4. **etcd links (per D-09)**: Verify every etcd cross-link points to the etcd metadata exchange page. The target path should resolve to `docs/user-guide/etcd-metadata-exchange.md` (or its Fern equivalent).
+
+5. **NIXLBench <-> KVBench cross-links (per D-10)**: Verify:
+ - KVBench overview (`kvbench/index.md`) links to NIXLBench section
+ - KVBench build page links to NIXLBench container (if referencing shared Docker image)
+ - NIXLBench pages do NOT need to link back to KVBench
+
+6. **Fix broken links**: If any link target does not exist or path is incorrect, fix the link. Use the Fern path convention that matches what the existing pages use (check whether they use relative .md paths or Fern route paths).
+
+
+ - Every markdown link [text](path) in the 6 files resolves to an existing file or valid Fern route
+ - Backend names have first-mention links on each page
+ - etcd cross-links point to etcd-metadata-exchange page
+ - KVBench overview links to NIXLBench
+
+
+ cd /home/omrik/Projects/nixl.gitlab && echo "=== Extracting all links ===" && grep -rnoP '\[([^\]]+)\]\(([^)]+)\)' docs/user-guide/benchmarking/ --include="*.md" | head -40 && echo "=== Checking etcd links ===" && grep -rn 'etcd.*metadata\|metadata.*etcd' docs/user-guide/benchmarking/ --include="*.md" | head -10 && echo "=== Checking KVBench->NIXLBench links ===" && grep -rn 'nixlbench' docs/user-guide/benchmarking/kvbench/ --include="*.md" | head -10
+
+
+
+
+
+
+## Trust Boundaries
+
+No trust boundaries — this is a documentation audit with no user input, authentication, or data processing.
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-37-01 | I (Information Disclosure) | docs/*.md | accept | Documentation is public; no secrets to disclose |
+
+
+
+1. All terminology grep checks pass with zero prose violations
+2. All cross-links resolve to existing files
+3. CLI flags match authoritative sources (NIXLBench underscores, KVBench hyphens)
+4. No duplicated content — cross-links used instead
+
+
+
+- Zero terminology drift across all 6 benchmarking pages
+- All cross-links valid and first-mention backend links present
+- CLI flag tables accurate per --help output
+
+
+
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-01-SUMMARY.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-01-SUMMARY.md
new file mode 100644
index 0000000000..fa69181e2a
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-01-SUMMARY.md
@@ -0,0 +1,99 @@
+---
+phase: 37-terminology-normalization-and-quality-audit
+plan: 01
+subsystem: docs
+tags: [terminology, cross-links, quality-audit, fern, markdown]
+
+# Dependency graph
+requires:
+ - phase: 33-nixlbench-overview-and-build
+ provides: NIXLBench overview and build pages
+ - phase: 34-nixlbench-usage-and-troubleshooting
+ provides: NIXLBench usage page
+ - phase: 35-kvbench-overview-and-build
+ provides: KVBench overview and build pages
+ - phase: 36-kvbench-commands-and-examples
+ provides: KVBench commands page
+provides:
+ - All 6 benchmarking pages pass terminology checks with zero violations
+ - All cross-links verified and first-mention backend links present
+ - CLI flags validated against authoritative README sources
+affects: [37-02]
+
+# Tech tracking
+tech-stack:
+ added: []
+ patterns: [first-mention backend linking, etcd cross-link on first mention per page]
+
+key-files:
+ created: []
+ modified:
+ - docs/user-guide/benchmarking/kvbench/commands.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+
+key-decisions:
+ - "All 6 pages already followed terminology conventions -- only 2 missing first-mention links needed fixing"
+
+patterns-established:
+ - "First-mention cross-link: every backend name and etcd reference links to its User Guide page on first appearance per page"
+
+requirements-completed: [QS-01, QS-02, QS-03]
+
+# Metrics
+duration: 3min
+completed: 2026-04-07
+---
+
+# Phase 37 Plan 01: Terminology Normalization and Quality Audit Summary
+
+**Zero terminology drift found across all 6 benchmarking pages; added 2 missing first-mention cross-links (etcd on kvbench/commands, UCX on nixlbench/build)**
+
+## Performance
+
+- **Duration:** 3 min
+- **Started:** 2026-04-07T21:27:33Z
+- **Completed:** 2026-04-07T21:31:32Z
+- **Tasks:** 2
+- **Files modified:** 2
+
+## Accomplishments
+- Verified zero violations across all terminology rules: plug-in, etcd casing, KV cache, NIXLBench/KVBench casing, InfiniBand, backend ALL CAPS
+- Confirmed CLI flags are correct: NIXLBench uses --etcd_endpoints (underscores), KVBench uses --etcd-endpoints (hyphens)
+- Verified all cross-links resolve to valid Fern routes and existing files
+- Added missing first-mention etcd link on kvbench/commands.md
+- Added missing first-mention UCX backend link on nixlbench/build.md
+- Confirmed KVBench pages link to NIXLBench as required
+- No duplicated content found -- all reusable content uses cross-links
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Terminology grep audit and inline fixes** - no changes needed (all checks passed)
+2. **Task 2: Cross-link audit and verification** - `fb98eb7f` (fix)
+
+## Files Created/Modified
+- `docs/user-guide/benchmarking/kvbench/commands.md` - Added etcd metadata exchange cross-link on first etcd mention in CLI table
+- `docs/user-guide/benchmarking/nixlbench/build.md` - Added UCX backend cross-link on first UCX mention in dependency list
+
+## Decisions Made
+- All 6 pages were already well-written with correct terminology from prior phases; only 2 missing first-mention links needed adding
+- Decided to link UCX in the native build dependency list (borderline prose/list context) since it improves discoverability
+
+## Deviations from Plan
+
+None -- plan executed exactly as written. The terminology audit found zero violations, and the cross-link audit found only 2 minor missing first-mention links which were fixed inline.
+
+## Issues Encountered
+None
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- All 6 benchmarking pages are terminology-clean and cross-link verified
+- Ready for plan 37-02 (Fern build validation) if applicable
+
+---
+*Phase: 37-terminology-normalization-and-quality-audit*
+*Completed: 2026-04-07*
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-02-PLAN.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-02-PLAN.md
new file mode 100644
index 0000000000..091cb6b69b
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-02-PLAN.md
@@ -0,0 +1,197 @@
+---
+phase: 37-terminology-normalization-and-quality-audit
+plan: 02
+type: execute
+wave: 2
+depends_on: [37-01]
+files_modified:
+ - docs/user-guide/benchmarking/nixlbench/index.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+ - docs/user-guide/benchmarking/nixlbench/usage.md
+ - docs/user-guide/benchmarking/kvbench/index.md
+ - docs/user-guide/benchmarking/kvbench/build.md
+ - docs/user-guide/benchmarking/kvbench/commands.md
+autonomous: true
+requirements: [QS-04]
+
+must_haves:
+ truths:
+ - "fern check passes with zero errors"
+ - "All 6 benchmarking pages use valid Fern MDX syntax"
+ - "No bare anchor links, no HTML comments, proper frontmatter"
+ artifacts:
+ - path: "docs/user-guide/benchmarking/nixlbench/index.md"
+ provides: "Fern-valid NIXLBench overview"
+ - path: "docs/user-guide/benchmarking/nixlbench/build.md"
+ provides: "Fern-valid NIXLBench build"
+ - path: "docs/user-guide/benchmarking/nixlbench/usage.md"
+ provides: "Fern-valid NIXLBench usage"
+ - path: "docs/user-guide/benchmarking/kvbench/index.md"
+ provides: "Fern-valid KVBench overview"
+ - path: "docs/user-guide/benchmarking/kvbench/build.md"
+ provides: "Fern-valid KVBench build"
+ - path: "docs/user-guide/benchmarking/kvbench/commands.md"
+ provides: "Fern-valid KVBench commands"
+ key_links:
+ - from: "docs/index.yml"
+ to: "all 6 benchmarking pages"
+ via: "navigation entries"
+ pattern: "benchmarking/"
+---
+
+
+Run fern check to validate all benchmarking pages pass Fern's MDX validation. Fix any errors found.
+
+Purpose: Final quality gate — ensures all new pages render correctly on the Fern docs site.
+Output: Clean fern check with zero errors across all benchmarking pages.
+
+
+
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/workflows/execute-plan.md
+@/home/omrik/Projects/nixl.gitlab/.claude/get-shit-done/templates/summary.md
+
+
+
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/37-terminology-normalization-and-quality-audit/37-CONTEXT.md
+@.planning/phases/37-terminology-normalization-and-quality-audit/37-01-SUMMARY.md
+
+@docs/user-guide/benchmarking/nixlbench/index.md
+@docs/user-guide/benchmarking/nixlbench/build.md
+@docs/user-guide/benchmarking/nixlbench/usage.md
+@docs/user-guide/benchmarking/kvbench/index.md
+@docs/user-guide/benchmarking/kvbench/build.md
+@docs/user-guide/benchmarking/kvbench/commands.md
+
+
+
+
+
+ Task 1: Run fern check and fix all errors
+
+ - docs/user-guide/benchmarking/nixlbench/index.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+ - docs/user-guide/benchmarking/nixlbench/usage.md
+ - docs/user-guide/benchmarking/kvbench/index.md
+ - docs/user-guide/benchmarking/kvbench/build.md
+ - docs/user-guide/benchmarking/kvbench/commands.md
+ - fern/docs.yml
+
+
+ docs/user-guide/benchmarking/nixlbench/index.md
+ docs/user-guide/benchmarking/nixlbench/build.md
+ docs/user-guide/benchmarking/nixlbench/usage.md
+ docs/user-guide/benchmarking/kvbench/index.md
+ docs/user-guide/benchmarking/kvbench/build.md
+ docs/user-guide/benchmarking/kvbench/commands.md
+
+
+Run `fern check` from the `fern/` directory (per D-06). This validates:
+- MDX syntax (no invalid JSX, proper component usage)
+- Navigation entries in docs.yml resolve to actual files
+- Frontmatter is valid
+- No broken internal Fern links
+
+Steps:
+1. Run `cd /home/omrik/Projects/nixl.gitlab/fern && fern check` and capture output
+2. If errors are reported:
+ a. Parse each error message to identify the file and line
+ b. Read the affected file
+ c. Fix the issue (common fixes: bare anchor links need Fern link syntax, HTML comments must be removed, components need proper import/usage)
+ d. Re-run `fern check` to confirm the fix
+3. Repeat until `fern check` passes with zero errors
+
+Common Fern MDX issues to watch for:
+- Bare `` tags (use markdown `[text](url)` instead)
+- HTML comments `` (remove or convert to MDX comments `{/* */}`)
+- Missing or malformed frontmatter `title:` and `description:` fields
+- Invalid Tabs/CodeBlock component usage
+- Links to non-existent navigation paths
+
+Per D-07, optionally run `fern docs dev` if visual preview would catch issues that `fern check` misses (Claude's discretion).
+
+
+ - `fern check` exits with code 0 and reports no errors
+ - All 6 benchmarking pages are valid Fern MDX
+ - No regressions introduced to existing pages
+
+
+ cd /home/omrik/Projects/nixl.gitlab/fern && fern check 2>&1
+
+
+
+
+ Task 2: Final terminology re-sweep and summary
+
+ - docs/user-guide/benchmarking/nixlbench/index.md
+ - docs/user-guide/benchmarking/nixlbench/build.md
+ - docs/user-guide/benchmarking/nixlbench/usage.md
+ - docs/user-guide/benchmarking/kvbench/index.md
+ - docs/user-guide/benchmarking/kvbench/build.md
+ - docs/user-guide/benchmarking/kvbench/commands.md
+
+
+ docs/user-guide/benchmarking/nixlbench/index.md
+ docs/user-guide/benchmarking/nixlbench/build.md
+ docs/user-guide/benchmarking/nixlbench/usage.md
+ docs/user-guide/benchmarking/kvbench/index.md
+ docs/user-guide/benchmarking/kvbench/build.md
+ docs/user-guide/benchmarking/kvbench/commands.md
+
+
+Final sweep after all fixes from Plan 01 and Task 1 of this plan. This catches any regressions introduced during fern check fixes.
+
+Run the complete terminology check suite one final time:
+
+1. `grep -rn '\bplugin\b' docs/user-guide/benchmarking/ --include="*.md"` — filter out code blocks, expect zero prose matches
+2. `grep -rn '\bETCD\b' docs/user-guide/benchmarking/ --include="*.md"` — filter out code/CLI, expect zero prose matches
+3. `grep -rn 'Infiniband\|infiniband\|INFINIBAND' docs/user-guide/benchmarking/ --include="*.md"` — expect zero matches
+4. `grep -rn 'NIXL Bench\|Nixlbench\|NixlBench' docs/user-guide/benchmarking/ --include="*.md"` — expect zero matches
+5. `grep -rn 'KV Bench\|Kvbench\|KvBench' docs/user-guide/benchmarking/ --include="*.md"` — expect zero matches
+6. Verify NIXLBench pages have `--etcd_endpoints` (underscores) and KVBench pages have `--etcd-endpoints` (hyphens)
+
+If any violations found, fix them. If all pass, the phase is complete.
+
+
+ - All 6 terminology grep checks return zero violations
+ - CLI flag conventions confirmed correct
+ - No regressions from fern check fixes
+
+
+ cd /home/omrik/Projects/nixl.gitlab && echo "=== Final terminology sweep ===" && echo "plugin:" && (grep -rn '\bplugin\b' docs/user-guide/benchmarking/ --include="*.md" | grep -v '```' | grep -v '_plugin' || echo "CLEAN") && echo "ETCD:" && (grep -rn '\bETCD\b' docs/user-guide/benchmarking/ --include="*.md" | grep -v '```' | grep -v '\-\-' | grep -v 'ETCD_' || echo "CLEAN") && echo "InfiniBand:" && (grep -rn 'Infiniband\|infiniband\|INFINIBAND' docs/user-guide/benchmarking/ --include="*.md" || echo "CLEAN") && echo "NIXLBench casing:" && (grep -rn 'NIXL Bench\|Nixlbench\|NixlBench' docs/user-guide/benchmarking/ --include="*.md" || echo "CLEAN") && echo "KVBench casing:" && (grep -rn 'KV Bench\|Kvbench\|KvBench' docs/user-guide/benchmarking/ --include="*.md" || echo "CLEAN") && echo "=== fern check ===" && cd fern && fern check 2>&1 | tail -5
+
+
+
+
+
+
+## Trust Boundaries
+
+No trust boundaries — documentation validation only.
+
+## STRIDE Threat Register
+
+| Threat ID | Category | Component | Disposition | Mitigation Plan |
+|-----------|----------|-----------|-------------|-----------------|
+| T-37-01 | I (Information Disclosure) | docs/*.md | accept | Public documentation; no secrets |
+
+
+
+1. `fern check` exits cleanly with zero errors
+2. Final terminology sweep returns zero violations across all checks
+3. No regressions introduced during fix cycles
+
+
+
+- fern check passes with zero errors
+- All terminology normalized per D-01 rules
+- Phase 37 quality audit complete
+
+
+
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-02-SUMMARY.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-02-SUMMARY.md
new file mode 100644
index 0000000000..0d765d658c
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-02-SUMMARY.md
@@ -0,0 +1,94 @@
+---
+phase: 37-terminology-normalization-and-quality-audit
+plan: 02
+subsystem: docs
+tags: [fern, mdx, validation, terminology, quality-audit]
+
+# Dependency graph
+requires:
+ - phase: 37-terminology-normalization-and-quality-audit
+ provides: Terminology-normalized benchmarking pages from plan 01
+provides:
+ - All 6 benchmarking pages pass fern check with zero errors
+ - Final terminology re-sweep confirms zero violations
+ - Phase 37 quality audit complete
+affects: []
+
+# Tech tracking
+tech-stack:
+ added: []
+ patterns: []
+
+key-files:
+ created: []
+ modified: []
+
+key-decisions:
+ - "All 6 benchmarking pages passed fern check on first run with zero errors -- no fixes needed"
+ - "Pre-existing contrast warning (NVIDIA green #76B900 vs light background) is branding, not a benchmarking page issue"
+
+patterns-established: []
+
+requirements-completed: [QS-04]
+
+# Metrics
+duration: 3min
+completed: 2026-04-07
+---
+
+# Phase 37 Plan 02: Fern Build Validation and Final Sweep Summary
+
+**All 6 benchmarking pages pass fern check with zero errors and zero terminology violations on final re-sweep**
+
+## Performance
+
+- **Duration:** 3 min
+- **Started:** 2026-04-07T21:32:55Z
+- **Completed:** 2026-04-07T21:36:03Z
+- **Tasks:** 2
+- **Files modified:** 0
+
+## Accomplishments
+- Ran fern check: zero errors across all benchmarking pages (only pre-existing NVIDIA green contrast warning)
+- Final terminology sweep: zero violations for plug-in, ETCD, InfiniBand, NIXLBench casing, KVBench casing
+- Confirmed CLI flag conventions: NIXLBench uses --etcd_endpoints (underscores), KVBench uses --etcd-endpoints (hyphens)
+- Phase 37 quality audit complete -- all 6 benchmarking pages are consistent with v1.0 conventions
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Run fern check and fix all errors** - no commit needed (fern check passed clean, zero errors)
+2. **Task 2: Final terminology re-sweep and summary** - no commit needed (all checks passed clean, zero violations)
+
+## Files Created/Modified
+
+None -- all pages passed validation without requiring changes.
+
+## Decisions Made
+- fern check passed with zero errors on first run; the only warning is a pre-existing contrast ratio issue with the NVIDIA green accent color (#76B900) which is a branding constraint, not a benchmarking page issue
+- All terminology checks returned clean results, confirming plan 01 fixes were correct and no regressions exist
+
+## Deviations from Plan
+
+None -- plan executed exactly as written. All validation checks passed on first run.
+
+## Issues Encountered
+None
+
+## User Setup Required
+None - no external service configuration required.
+
+## Next Phase Readiness
+- Phase 37 complete: all 6 benchmarking pages are terminology-normalized, cross-link verified, CLI-validated, and Fern MDX compliant
+- v1.1 milestone documentation is ready for final review
+
+## Self-Check: PASSED
+
+- All 6 benchmarking pages exist and are valid Fern MDX
+- fern check returns 0 errors
+- All terminology grep checks return CLEAN
+
+---
+*Phase: 37-terminology-normalization-and-quality-audit*
+*Completed: 2026-04-07*
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-CONTEXT.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-CONTEXT.md
new file mode 100644
index 0000000000..79c508e65d
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-CONTEXT.md
@@ -0,0 +1,123 @@
+# Phase 37: Terminology Normalization and Quality Audit - Context
+
+**Gathered:** 2026-04-07
+**Status:** Ready for planning
+
+
+## Phase Boundary
+
+Batch grep pass across all new benchmarking pages (Phases 32-36 output) for terminology drift, validate CLI flag tables against `--help` output, confirm `fern check` passes, and audit cross-links. Fix all issues inline. No new content — this is quality assurance only.
+
+
+
+
+## Implementation Decisions
+
+### Terminology Rules
+- **D-01:** Expanded terminology standardization list (aligning new pages to v1.0 docs conventions):
+ - `plug-in` (not `plugin`) — consistent with existing NIXL docs
+ - `etcd` (lowercase) in prose text; `ETCD` acceptable in CLI flag values and code contexts
+ - `KV cache` (two words, space) in prose text; `kvcache` only in code/CLI contexts (e.g., `kvcache` subcommand)
+ - `NIXLBench` (camelCase, one word) — not `NIXL Bench`, `nixlbench` (except in CLI commands), or `Nixlbench`
+ - `KVBench` (camelCase, one word) — same pattern as NIXLBench
+ - Backend names ALL CAPS in tables and CLI contexts (UCX, GDS, POSIX, etc.)
+ - `InfiniBand` (camelCase) — not `Infiniband`, `infiniband`, or `IB` (except in compound terms like `ibverbs`)
+ - All other terms: align to whatever the existing v1.0 docs use
+
+### Audit Scope
+- **D-02:** Audit only the new benchmarking pages created in Phases 32-36. Do NOT audit existing v1.0 docs — those were already audited. The goal is to align new pages TO the v1.0 conventions, not to re-audit the whole site.
+- **D-03:** Cross-links from new pages to existing pages should be verified (links resolve correctly). Cross-links from existing pages to new pages are not expected to exist yet (no existing page was modified to link to benchmarking).
+
+### CLI Flag Validation
+- **D-04:** Validate CLI flag tables against actual `--help` output for both tools:
+ - NIXLBench: run `nixlbench --help` (or check README if binary not available)
+ - KVBench: run `python main.py [cmd] --help` for each subcommand
+ - Flag the `--etcd_endpoints` (NIXLBench, underscores) vs `--etcd-endpoints` (KVBench, hyphens) difference — per QS-03, this is intentional and must be documented correctly in each tool's pages
+- **D-05:** Fix any mismatches between documented flags and actual `--help` output directly in the markdown files.
+
+### Fern Build Validation
+- **D-06:** Run `fern check` from the `fern/` directory. Fix any failures inline — this is the final quality gate. All new pages must pass `fern check` with no errors.
+- **D-07:** Optionally run `fern docs dev` to preview rendered output and catch visual issues (broken tables, missing images, malformed MDX). Claude's discretion on whether a visual preview is needed.
+
+### Cross-Link Audit
+- **D-08:** Verify all first-mention inline links for backend names resolve to correct User Guide backend pages.
+- **D-09:** Verify ETCD links point to `docs/user-guide/etcd-metadata-exchange.md`.
+- **D-10:** Verify NIXLBench ↔ KVBench cross-links work in both directions (KVBench overview links to NIXLBench, NIXLBench doesn't need to link back).
+
+### Claude's Discretion
+- Whether to run `fern docs dev` for visual preview
+- Whether to fix minor style inconsistencies beyond the explicit terminology list (e.g., sentence vs fragment in table cells)
+- Ordering of audit steps (grep first vs fern check first)
+
+
+
+
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Pages to audit
+- `docs/user-guide/benchmarking/nixlbench/index.md` — NIXLBench overview (Phase 33)
+- `docs/user-guide/benchmarking/nixlbench/build.md` — NIXLBench build (Phase 33)
+- `docs/user-guide/benchmarking/nixlbench/usage.md` — NIXLBench usage + troubleshooting (Phase 34)
+- `docs/user-guide/benchmarking/kvbench/index.md` — KVBench overview (Phase 35)
+- `docs/user-guide/benchmarking/kvbench/build.md` — KVBench build (Phase 35)
+- `docs/user-guide/benchmarking/kvbench/commands.md` — KVBench commands, config, examples (Phase 36)
+
+### Navigation config
+- `docs/index.yml` — Verify all benchmarking nav entries resolve correctly
+
+### Terminology reference (v1.0 baseline)
+- `docs/user-guide/backends/ucx.md` — Backend page terminology pattern (v1.0 baseline)
+- `docs/user-guide/etcd-metadata-exchange.md` — etcd terminology in existing docs
+- `docs/user-guide/building-nixl/index.md` — Existing Developer Guide terminology
+
+### CLI validation sources
+- `benchmark/nixlbench/README.md` §Command Line Options — NIXLBench flag reference
+- `benchmark/kvbench/README.md` §Command Line Arguments — KVBench flag reference
+- `benchmark/kvbench/main.py` — KVBench CLI entry point for `--help` validation
+
+### Requirements
+- `.planning/REQUIREMENTS.md` — QS-01 (terminology), QS-02 (no duplication), QS-03 (CLI validation), QS-04 (Fern MDX)
+
+
+
+
+## Existing Code Insights
+
+### Reusable Assets
+- v1.0 docs pages — Terminology baseline to grep against for consistency
+- `fern check` command — Built-in Fern validation for MDX/navigation issues
+
+### Established Patterns
+- v1.0 already established: `plug-in`, lowercase `etcd` in prose, backend ALL CAPS, `InfiniBand`
+- Existing cross-link pattern: first-mention inline links for backend names → User Guide backend pages
+
+### Integration Points
+- All 6 new benchmarking pages + `docs/index.yml` are the audit targets
+- No existing pages are modified in this phase
+
+
+
+
+## Specific Ideas
+
+- User wants new pages aligned TO v1.0 conventions, not a re-audit of the whole site
+- Expanded terminology list beyond REQUIREMENTS QS-01: added KV cache, NIXLBench/KVBench casing, InfiniBand, backend caps
+- Fix all issues inline — no separate report/review step
+
+
+
+
+## Deferred Ideas
+
+- Full site terminology audit across all v1.0 + v1.1 pages — not needed, v1.0 already audited
+- Automated terminology linting CI check — interesting but out of scope
+- Style guide document for future contributors — could be useful but not in this milestone
+
+
+
+---
+
+*Phase: 37-terminology-normalization-and-quality-audit*
+*Context gathered: 2026-04-07*
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-DISCUSSION-LOG.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-DISCUSSION-LOG.md
new file mode 100644
index 0000000000..fa0c75719d
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-DISCUSSION-LOG.md
@@ -0,0 +1,62 @@
+# Phase 37: Terminology Normalization and Quality Audit - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-04-07
+**Phase:** 37-Terminology Normalization and Quality Audit
+**Areas discussed:** Terminology rules, Audit scope, Fern build validation
+
+---
+
+## Terminology Rules
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| REQUIREMENTS list only | Stick to QS-01 (plug-in, etcd, backend caps) | |
+| Expanded term list | Pre-define additional rules: KV cache, NIXLBench, InfiniBand | ✓ |
+| You decide | Claude builds list during audit | |
+
+**User's choice:** Expanded term list
+**Notes:** User also selected all 4 specific rules (KV cache, NIXLBench, backend caps, InfiniBand) and added "Align to terminology throughout the docs."
+
+---
+
+## Audit Scope
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| New pages only | Audit only benchmarking pages from Phases 32-36 | |
+| New pages + cross-references | Audit new pages + existing pages that link to/from benchmarking | |
+| Full site audit | Audit all docs pages | |
+| Custom | User: "Just align the new pages to the previous ones (v1.0)" | ✓ |
+
+**User's choice:** Align new pages to v1.0 conventions
+**Notes:** v1.0 pages already audited — just align new pages TO them.
+
+---
+
+## Fern Build Validation
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Fix inline | Run fern check, fix issues directly | ✓ |
+| Flag for review | Log failures, don't fix | |
+| You decide | Claude handles as part of workflow | |
+
+**User's choice:** Fix inline
+**Notes:** Final quality gate — everything must pass.
+
+---
+
+## Claude's Discretion
+
+- Whether to run fern docs dev for visual preview
+- Minor style fixes beyond explicit terminology list
+- Audit step ordering
+
+## Deferred Ideas
+
+- Full site terminology audit — not needed
+- Automated terminology linting CI — out of scope
+- Style guide document — future consideration
diff --git a/.planning/phases/37-terminology-normalization-and-quality-audit/37-VERIFICATION.md b/.planning/phases/37-terminology-normalization-and-quality-audit/37-VERIFICATION.md
new file mode 100644
index 0000000000..ded8401030
--- /dev/null
+++ b/.planning/phases/37-terminology-normalization-and-quality-audit/37-VERIFICATION.md
@@ -0,0 +1,95 @@
+---
+phase: 37-terminology-normalization-and-quality-audit
+verified: 2026-04-07T22:15:00Z
+status: passed
+score: 4/4
+overrides_applied: 0
+---
+
+# Phase 37: Terminology Normalization and Quality Audit Verification Report
+
+**Phase Goal:** All six new benchmarking pages are internally consistent, match the terminology of the existing docs site, and the Fern build is clean
+**Verified:** 2026-04-07T22:15:00Z
+**Status:** passed
+**Re-verification:** No -- initial verification
+
+## Goal Achievement
+
+### Observable Truths
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | Zero instances of `plugin` (must be `plug-in`), `ETCD` in prose (must be `etcd`), and no inconsistent backend capitalizations | VERIFIED | `grep -rn '\bplugin\b'` returns zero prose matches. `grep -rn '\bETCD\b'` returns only 2 matches in CLI table cells showing `ETCD` as a `--runtime_type` value (code context, correct). `grep` for `Infiniband/infiniband/INFINIBAND` returns zero. `grep` for `NIXL Bench/Nixlbench/NixlBench` returns zero. `grep` for `KV Bench/Kvbench/KvBench` returns zero. `kvcache` in prose only appears as CLI subcommand name or YAML field name (correct). |
+| 2 | No content duplicates existing docs: build steps, etcd setup, and backend configs replaced with cross-links | VERIFIED | Build pages cross-link to [Building NIXL from Source](/docs/user-guide/building-nixl) (2 links in nixlbench/build.md). etcd references cross-link to etcd-metadata-exchange page (nixlbench/index.md, nixlbench/usage.md x2, kvbench/commands.md). Backend names link to their User Guide pages on first mention per page. The etcd docker-run on usage.md is a minimal quick-start code block with cross-link to full etcd page, not duplicated prose. |
+| 3 | NIXLBench CLI tables use `--etcd_endpoints` (underscores) and KVBench uses `--etcd-endpoints` (hyphens) | VERIFIED | NIXLBench pages: 8 occurrences of `--etcd_endpoints` (underscores), zero hyphens. KVBench pages: 8 occurrences of `--etcd-endpoints` (hyphens), zero underscores. Explicit note in kvbench/commands.md line 164 documents the convention difference. |
+| 4 | `fern check` passes with zero errors against all six new pages | VERIFIED | `fern check` output: "Found 0 errors and 1 warning in 0.000 seconds." The warning is a pre-existing NVIDIA green contrast issue (#76B900), unrelated to benchmarking pages. |
+
+**Score:** 4/4 truths verified
+
+### Required Artifacts
+
+| Artifact | Expected | Status | Details |
+|----------|----------|--------|---------|
+| `docs/user-guide/benchmarking/nixlbench/index.md` | NIXLBench overview, terminology normalized | VERIFIED | 22 lines, substantive content with backend links, etcd cross-link, proper casing |
+| `docs/user-guide/benchmarking/nixlbench/build.md` | NIXLBench build, terminology normalized | VERIFIED | 110 lines, Tabs component, UCX backend link added by plan 01, cross-links to building-nixl |
+| `docs/user-guide/benchmarking/nixlbench/usage.md` | NIXLBench usage, terminology normalized | VERIFIED | 278 lines, 4 communication patterns, CLI tables with --etcd_endpoints, troubleshooting, Warning component |
+| `docs/user-guide/benchmarking/kvbench/index.md` | KVBench overview, terminology normalized | VERIFIED | 29 lines, NIXLBench link, KV cache two-word usage, command categories |
+| `docs/user-guide/benchmarking/kvbench/build.md` | KVBench build, terminology normalized | VERIFIED | 38 lines, Tabs component, Docker/venv paths, NIXLBench container cross-link |
+| `docs/user-guide/benchmarking/kvbench/commands.md` | KVBench commands, terminology normalized | VERIFIED | 495 lines, CLI tables with --etcd-endpoints, backend links, model config schemas, LLM examples |
+
+### Key Link Verification
+
+| From | To | Via | Status | Details |
+|------|----|-----|--------|---------|
+| nixlbench/index.md | etcd-metadata-exchange | `[etcd](/docs/user-guide/etcd-metadata-exchange)` | WIRED | Line 6, first etcd mention |
+| nixlbench/usage.md | etcd-metadata-exchange | `[etcd](/docs/user-guide/etcd-metadata-exchange)` | WIRED | Lines 10 and 28 |
+| kvbench/commands.md | etcd-metadata-exchange | `[etcd](/docs/user-guide/etcd-metadata-exchange)` | WIRED | Line 158, in CLI table |
+| All 6 pages | backends/*.md | First-mention inline links | WIRED | nixlbench/index.md links 11 backends, usage.md links 10 backends, build.md links UCX, kvbench/commands.md links 8 backends |
+| kvbench/index.md | nixlbench | `[NIXLBench](./nixlbench)` | WIRED | Line 6, first paragraph |
+| kvbench/commands.md | nixlbench | `[NIXLBench](./nixlbench)` | WIRED | Line 6 |
+| kvbench/build.md | nixlbench/build | `[Building NIXLBench](./nixlbench/build)` | WIRED | Line 12, Docker tab |
+| nixlbench/build.md | building-nixl | `[Building NIXL from Source](/docs/user-guide/building-nixl)` | WIRED | Lines 6 and 70 |
+
+### Data-Flow Trace (Level 4)
+
+Not applicable -- documentation pages, no dynamic data rendering.
+
+### Behavioral Spot-Checks
+
+| Behavior | Command | Result | Status |
+|----------|---------|--------|--------|
+| fern check passes | `cd fern && fern check` | 0 errors, 1 warning (pre-existing) | PASS |
+| No plugin in prose | `grep -rn '\bplugin\b' docs/user-guide/benchmarking/` | Zero matches | PASS |
+| No ETCD in prose | `grep -rn '\bETCD\b' ... \| grep -v backtick/CLI` | Zero prose matches (2 CLI table values correct) | PASS |
+| NIXLBench uses underscores | `grep 'etcd.endpoints' nixlbench/` | All 8 use `--etcd_endpoints` | PASS |
+| KVBench uses hyphens | `grep 'etcd.endpoints' kvbench/` | All 8 use `--etcd-endpoints` | PASS |
+| No bare anchor links | `grep '/` (not relative filesystem paths). For example, a NIXLBench page referencing the ETCD User Guide page uses:
+
+```markdown
+[Metadata Exchange with ETCD](/docs/user-guide/etcd-metadata-exchange)
+```
+
+KVBench pages referencing NIXLBench should use:
+
+```markdown
+[NIXLBench](/docs/development/benchmarks/nixlbench)
+```
+
+## Integration Points
+
+### Files to Modify
+
+| File | Change | Notes |
+|------|--------|-------|
+| `docs/index.yml` | Add two `section:` blocks under Developer Guide | Insert after `building-a-backend-plugin.md` entry; both use `collapsed: open-by-default` to match existing style |
+
+### New Files to Create
+
+| File | Content Source | Notes |
+|------|---------------|-------|
+| `docs/development/benchmarks/nixlbench/index.md` | NIXLBench README Features + Quick Start sections | Overview, key capabilities, link to child pages |
+| `docs/development/benchmarks/nixlbench/build.md` | NIXLBench README Building section | Docker build (recommended) + native build |
+| `docs/development/benchmarks/nixlbench/usage.md` | NIXLBench README Usage section | ETCD setup, basic usage, config file format, multi-node patterns |
+| `docs/development/benchmarks/nixlbench/cli-reference.md` | NIXLBench README Command Line Options section | Full flag reference, organized by category (core, memory, performance, storage, per-backend) |
+| `docs/development/benchmarks/nixlbench/backend-examples.md` | NIXLBench README Backend-Specific Examples section | Per-backend command examples (UCX, GPUNETIO, GDS, POSIX, OBJ, GUSLI, etc.) |
+| `docs/development/benchmarks/kvbench/index.md` | KVBench README Overview + Table of Contents | Overview, two command categories (KVBench vs CTP), supported LLM architectures |
+| `docs/development/benchmarks/kvbench/build.md` | KVBench README Building section | Docker + Python venv install instructions |
+| `docs/development/benchmarks/kvbench/commands.md` | KVBench README Command Descriptions + CLI Arguments | All five commands: plan, profile, kvcache, ct-perftest, sequential-ct-perftest |
+| `docs/development/benchmarks/kvbench/llm-examples.md` | KVBench README Examples + Developer Guides content | DeepSeek R1 block/layer examples, CTP YAML configs, matrix generation, Slurm examples |
+
+### No Changes Required
+
+| File | Reason |
+|------|--------|
+| `fern/docs.yml` | Site config is complete; no new products, versions, or nav tabs needed |
+| `fern/components/` | No new custom MDX components needed |
+| `docs/assets/` | No new images or fonts needed |
+| Any existing docs page | No existing content needs modification; benchmark pages are additive |
+
+## Anti-Patterns
+
+### Anti-Pattern 1: Top-Level Benchmarks Section
+
+**What people do:** Add a "Benchmarks" entry at the same level as Getting Started, User Guide, Developer Guide.
+
+**Why it's wrong:** NIXLBench and KVBench are developer tools for measuring NIXL performance — they are not user-facing product features. Elevating them to top-level implies equal importance to core documentation, which fragments the navigation hierarchy for all users. The PROJECT.md explicitly specifies Developer Guide as the target section.
+
+**Do this instead:** Place both tools as subsections of Developer Guide, after "Building a Backend Plugin".
+
+### Anti-Pattern 2: Flat Nine-Page Dump into `development/`
+
+**What people do:** Add all nine new `.md` files directly into `docs/development/` without subdirectories.
+
+**Why it's wrong:** The `development/` directory currently holds two files (`building-a-backend-plugin.md`, `sb-api-reference.md`). Adding nine more files destroys the scannability of the directory. The navigation grouping in `docs/index.yml` would also become a flat nine-entry list under Developer Guide with no visual hierarchy.
+
+**Do this instead:** Group by tool under `docs/development/benchmarks/nixlbench/` and `docs/development/benchmarks/kvbench/`.
+
+### Anti-Pattern 3: Single-Page Per Tool
+
+**What people do:** Combine all NIXLBench content into one `nixlbench.md` file (mirroring the README structure).
+
+**Why it's wrong:** The NIXLBench README is ~700 lines covering features, system requirements, building, usage, CLI reference, and backend examples. A single-page dump produces a doc that is hard to navigate and impossible to deep-link. The CLI reference alone covers nine backend-specific flag groups. Splitting into focused pages enables bookmarking, search result precision, and progressive disclosure.
+
+**Do this instead:** Use the five-page split (index, build, usage, cli-reference, backend-examples).
+
+### Anti-Pattern 4: Copying Raw README Verbatim
+
+**What people do:** Copy-paste the README markdown directly into the docs file.
+
+**Why it's wrong:** The READMEs contain GitHub-flavored Markdown that is not fully Fern/MDX compatible: bare anchor-only links (`[text](#heading)`), HTML comments, and relative paths to source files (`../../src/plugins/gusli/README.md`). These will either silently break or cause build errors. The READMEs also lack Fern frontmatter (`title:`, `description:`), which all existing docs pages include.
+
+**Do this instead:** Reauthor content as Fern-compatible MDX with frontmatter, replacing relative links with absolute `/docs/` paths, and converting GitHub-flavored callouts to Fern `` components.
+
+## Build Order Considerations
+
+The nine new files can be created in any order since they have no build-time dependencies on each other. However, the recommended authoring order is:
+
+1. Create directory structure (`docs/development/benchmarks/nixlbench/`, `docs/development/benchmarks/kvbench/`)
+2. Update `docs/index.yml` with navigation entries (ensures nav is defined before content, allowing incremental preview)
+3. Author NIXLBench pages (index → build → usage → cli-reference → backend-examples)
+4. Author KVBench pages (index → build → commands → llm-examples)
+
+The `docs/index.yml` change is the single file that gates Fern's ability to render any of the new pages. It should be the first substantive change made.
+
+## Sources
+
+- Observed: `docs/index.yml` — existing navigation tree structure (lines 52–68 for Developer Guide pattern)
+- Observed: `docs/user-guide/building-nixl/index.md` — section landing page pattern with frontmatter
+- Observed: `docs/development/building-a-backend-plugin.md` — flat page pattern with frontmatter
+- Observed: `benchmark/nixlbench/README.md` — NIXLBench content scope (~700 lines, covering features, build, usage, CLI, backend examples)
+- Observed: `benchmark/kvbench/README.md` — KVBench content scope (~430 lines, covering KVBench + CTP commands)
+- Observed: `benchmark/kvbench/docs/` — supplementary KVBench developer guides (tutorial-gds.md, creating-a-model-config.md, adding-a-new-model-architecture.md, ct-perftest.md) — candidate content for llm-examples.md
+- Observed: `fern/docs.yml` — Fern product/version config, confirms `docs/index.yml` is the navigation source
+
+---
+*Architecture research for: NIXL documentation site — NIXLBench and KVBench integration*
+*Researched: 2026-04-07*
diff --git a/.planning/research/FEATURES.md b/.planning/research/FEATURES.md
new file mode 100644
index 0000000000..c3def1bc45
--- /dev/null
+++ b/.planning/research/FEATURES.md
@@ -0,0 +1,212 @@
+# Feature Research
+
+**Domain:** Benchmark tool documentation pages (NIXLBench + KVBench) under Developer Guide
+**Researched:** 2026-04-07
+**Confidence:** HIGH — both READMEs are comprehensive, kvbench/docs/ contains three additional guides
+
+## Context
+
+"Features" here means documentation page categories and sections for two benchmark tools. The
+existing site already covers ETCD, backends, building from source, and API references. The new
+pages must not duplicate those — they link to them. The audience is developers benchmarking NIXL
+transfer performance in distributed AI inference settings.
+
+---
+
+## NIXLBench Documentation Pages
+
+### Table Stakes (Must Have)
+
+Pages developers expect when looking up a benchmark tool. Missing these = "go read the source code".
+
+| Page / Section | Why Expected | Complexity | Notes |
+|----------------|--------------|------------|-------|
+| Overview | What it is, why it exists, key capabilities | LOW | Covers: multi-backend, ETCD coordination, memory types, communication patterns, worker types |
+| System requirements | Devs need to know if they can run it before spending time building | LOW | HW (x86/aarch64, GPU, NIC), SW (Ubuntu 22/24, Docker 20.10+, CUDA 12.8+), min RAM/disk |
+| Building: Docker path | Recommended method; devs expect it to be prominent | MEDIUM | build.sh options table, ETCD container launch, first benchmark run |
+| Building: Native path | Needed for CI / restricted environments | HIGH | Ordered deps (NIXL first, then nixlbench), meson options, PATH/LD_LIBRARY_PATH setup |
+| Quick start | Concrete "copy-paste and it works" in under 5 minutes | LOW | Docker path only, UCX backend, single pair of hosts |
+| CLI reference — core flags | Developers need all flags with types and defaults in one place | MEDIUM | Group by: core config, memory/transfer, performance/threading, device/network |
+| CLI reference — storage flags | Storage backends have distinct flag sets | MEDIUM | Per-backend tables: GDS, GDS_MT, POSIX, HF3FS, OBJ, AZURE_BLOB, GUSLI |
+| CLI reference — config file | TOML config file as alternative to CLI flags | LOW | Precedence rules, example TOML |
+| ETCD coordination | ETCD is mandatory for network backends and multi-node — needs its own callout | LOW | Required vs optional matrix per backend type; cleanup command for stuck instances |
+| Backend-specific examples | Devs target one backend; they need runnable copy-paste examples | MEDIUM | One subsection per backend: UCX, GPUNETIO, GDS, GDS_MT, POSIX, GUSLI, OBJ, AZURE_BLOB, NVSHMEM |
+| Troubleshooting | Build issues, runtime errors, and perf tuning all appear in the README | MEDIUM | Sections: build failures (CUDA, UCX, etcd-cpp-api, Docker), runtime (library not found, GPU, network), ETCD cleanup, perf tuning (CPU affinity, network buffers) |
+
+### Differentiators (Worth Having)
+
+Sections not universally present in benchmark tool docs but genuinely useful here.
+
+| Page / Section | Value Proposition | Complexity | Notes |
+|----------------|-------------------|------------|-------|
+| Communication patterns explainer | "Pairwise / many-to-one / one-to-many / TP" are non-obvious; a short explainer prevents misconfiguration | LOW | Embedded in overview or usage page; diagram optional |
+| Worker types comparison (nixl vs nvshmem) | NVSHMEM worker is VRAM-only with different constraints; knowing when to use each saves debugging | LOW | Short table or callout box |
+| Multi-node launch guide | The 60s ETCD window and rank coordination are footguns; explicit guidance prevents silent failures | LOW | Sequence diagram of rank registration; ETCD cleanup step |
+| Performance tuning guide | CPU affinity, NUMA binding, network buffer sizing — non-obvious but high impact | LOW | Can live in Troubleshooting under "Performance Tuning" subheading |
+
+### Anti-Features (Explicitly Avoid)
+
+| Page / Section | Why Requested | Why Problematic | Alternative |
+|----------------|---------------|-----------------|-------------|
+| Full dependency build instructions duplicated from Building from Source | Devs find that page via nav | Creates drift when NIXL build steps change | Link to the existing Developer Guide > Building NIXL from Source pages |
+| ETCD administration guide | Devs want to understand ETCD deeply | Out of scope for a benchmark tool; ETCD is a prerequisite | Link to the existing User Guide > Metadata Exchange with ETCD page |
+| API reference for nixlbench internals | Might seem logical for a developer guide | nixlbench is a standalone binary, not a library — it has no public API | CLI reference is the right artifact |
+| Separate "Installation" page | Mirrors what "Building" already covers | Two pages for the same concept creates duplication and nav confusion | Merge into a single "Building NIXLBench" page with Docker and native subsections |
+
+---
+
+## KVBench Documentation Pages
+
+### Table Stakes (Must Have)
+
+| Page / Section | Why Expected | Complexity | Notes |
+|----------------|--------------|------------|-------|
+| Overview | What KVBench does vs what NIXLBench does — the relationship is non-obvious | LOW | Two function groups: KVBench commands (LLM KV cache testing) and CTP commands (asymmetric traffic matrices); Python tool that calls nixlbench |
+| Building: Docker path | Reuses nixlbench Docker container — must say so explicitly | LOW | Points to nixlbench/contrib/build.sh; no separate container |
+| Building: Python venv path | Lightweight local install for plan/kvcache without running nixlbench | LOW | python3 -m venv, pip install uv, uv sync |
+| Command reference: plan | Most-used command for pre-benchmark configuration | MEDIUM | --model, --model_config, --format, all shared benchmark args; show sample output |
+| Command reference: profile | Runs the actual benchmark | MEDIUM | Same args as plan; explain it calls nixlbench under the hood |
+| Command reference: kvcache | Inspect KV cache sizing without running anything | LOW | Tabular output with model, ISL, IO Size, TP, PP, page size |
+| Command reference: ct-perftest | Custom traffic pattern benchmarking | MEDIUM | Config YAML structure, matrix file format, CUDA_VISIBLE_DEVICES note |
+| Command reference: sequential-ct-perftest | Multi-pattern sequential testing | MEDIUM | YAML config with traffic_patterns list, sleep_after_launch_sec, JSON output |
+| Model configuration guide | YAML schema is not self-evident; devs need to know what each field means | MEDIUM | strategy/runtime/system sections with field descriptions; block vs layer access patterns; already exists in kvbench/docs/creating-a-model-config.md |
+| CLI override arguments | --pp, --tp, --isl, --osl etc let devs test without editing YAML files | LOW | Table of all override args with defaults |
+| LLM architecture examples | Concrete examples for DeepSeek R1 and Llama 3.1 with sample outputs | MEDIUM | Block vs layer access, TP/PP combinations, sample nixlbench command output |
+| CTP examples | Matrix file format and YAML config are both non-obvious | MEDIUM | Matrix file syntax, YAML config, matgen script usage, srun/Slurm usage |
+
+### Differentiators (Worth Having)
+
+| Page / Section | Value Proposition | Complexity | Notes |
+|----------------|-------------------|------------|-------|
+| Extending KVBench: new model architecture | Developer audience will want to add their own models | MEDIUM | Already exists in kvbench/docs/adding-a-new-model-architecture.md; inherit BaseModelArch, implement 3 methods, register in factory |
+| KVBench to NIXLBench relationship explainer | It is not obvious that profile calls nixlbench or that plan just generates commands | LOW | Short diagram or callout: "KVBench is a command generator / runner wrapper around NIXLBench" |
+| GDS profiling tutorial | End-to-end tutorial for a realistic use case (GPU-to-storage KV offload) | LOW | Already exists in kvbench/docs/tutorial-gds.md; write→read sequence with sample output |
+| Traffic matrix generation | inference_workload_matgen.py is a separate helper not covered in README beyond a snippet | LOW | Brief callout with the matgen CLI flags table |
+
+### Anti-Features (Explicitly Avoid)
+
+| Page / Section | Why Requested | Why Problematic | Alternative |
+|----------------|---------------|-----------------|-------------|
+| Full NIXLBench CLI reference duplicated | Devs want all flags in one place | KVBench passes flags through to nixlbench; duplicating creates drift | Link to the NIXLBench CLI reference page with a note about which args are passed through |
+| Separate "Installation" page | Mirrors Building | Same duplication risk as NIXLBench | Single "Building KVBench" page covering Docker and Python venv |
+| Model architecture deep-dive (transformer math) | Might seem helpful for understanding KV cache sizes | Out of scope; KVBench abstracts the math | One-line explanation of KV cache sizing, link to model YAML format |
+
+---
+
+## Feature Dependencies
+
+```
+NIXLBench Overview
+ └──must precede──> NIXLBench CLI Reference
+ └──must precede──> NIXLBench Backend Examples
+ └──must precede──> NIXLBench Troubleshooting
+
+KVBench Overview
+ └──must precede──> KVBench Command Reference (plan, profile, kvcache)
+ └──must precede──> KVBench CTP Command Reference (ct-perftest, sequential-ct-perftest)
+
+KVBench Model Config Guide
+ └──must precede──> KVBench LLM Architecture Examples
+ └──must precede──> KVBench GDS Tutorial
+
+NIXLBench page (built)
+ └──required by──> KVBench profile command docs
+ (profile invokes nixlbench; must link to nixlbench CLI ref)
+
+Existing: Developer Guide > Building NIXL from Source
+ └──referenced by──> NIXLBench Native Build section (links, does not duplicate)
+
+Existing: User Guide > Metadata Exchange with ETCD
+ └──referenced by──> NIXLBench ETCD Coordination section (links, does not duplicate)
+
+Existing: User Guide > NIXL Backends (UCX, GDS, etc.)
+ └──referenced by──> Both NIXLBench and KVBench backend examples (links, does not duplicate)
+```
+
+### Dependency Notes
+
+- **KVBench profile requires NIXLBench to exist:** The profile command calls nixlbench as a subprocess. KVBench docs must be sequenced after NIXLBench docs or written with a cross-link placeholder.
+- **Model config guide required by LLM examples:** Examples use YAML files; readers must understand the schema before examples make sense.
+- **ETCD coordination is shared state:** Both tools rely on ETCD. NIXLBench docs explain ETCD setup. KVBench docs must link to NIXLBench's ETCD section and to the existing ETCD User Guide page rather than re-explaining it.
+
+---
+
+## MVP Definition
+
+This is a documentation milestone, so "MVP" = minimum docs to make the tools usable without reading source code.
+
+### Launch With (v1.1 — this milestone)
+
+NIXLBench:
+- [x] Overview page
+- [x] Building page (Docker + native, combined)
+- [x] Quick start section (Docker only)
+- [x] CLI reference (core + storage + config file)
+- [x] ETCD coordination section
+- [x] Backend-specific examples
+- [x] Troubleshooting page
+
+KVBench:
+- [x] Overview page (with relationship to NIXLBench)
+- [x] Building page (Docker + Python venv)
+- [x] Command reference (plan, profile, kvcache)
+- [x] Command reference (ct-perftest, sequential-ct-perftest)
+- [x] Model configuration guide (from kvbench/docs/creating-a-model-config.md)
+- [x] LLM architecture examples (DeepSeek R1, Llama 3.1)
+- [x] CTP examples (matrix format, YAML config, matgen script)
+
+### Add After Validation (v1.x)
+
+- [ ] Extending KVBench: adding a new model architecture — add when developer contributors increase
+- [ ] KVBench GDS profiling tutorial — add when GDS is a high-traffic backend
+- [ ] NIXLBench performance tuning standalone guide — add when perf troubleshooting is top support topic
+- [ ] Traffic matrix generation deep-dive — add if matgen becomes widely used
+
+### Future Consideration (v2+)
+
+- [ ] Interactive benchmark configuration builder — nice UX but requires significant tooling investment
+- [ ] Benchmark results comparison tables across backends — requires standardized output format
+
+---
+
+## Feature Prioritization Matrix
+
+| Documentation Feature | User Value | Authoring Cost | Priority |
+|-----------------------|------------|----------------|----------|
+| NIXLBench CLI reference | HIGH | MEDIUM | P1 |
+| NIXLBench backend-specific examples | HIGH | MEDIUM | P1 |
+| NIXLBench building (Docker) | HIGH | LOW | P1 |
+| NIXLBench ETCD coordination section | HIGH | LOW | P1 |
+| NIXLBench troubleshooting | HIGH | LOW | P1 |
+| KVBench command reference (plan/profile/kvcache) | HIGH | MEDIUM | P1 |
+| KVBench model configuration guide | HIGH | LOW | P1 |
+| KVBench LLM architecture examples | HIGH | MEDIUM | P1 |
+| KVBench CTP command reference | MEDIUM | MEDIUM | P1 |
+| KVBench CTP examples | MEDIUM | MEDIUM | P1 |
+| NIXLBench building (native) | MEDIUM | HIGH | P2 |
+| KVBench GDS profiling tutorial | MEDIUM | LOW | P2 |
+| KVBench: adding new model architecture | MEDIUM | LOW | P2 |
+| NIXLBench performance tuning section | MEDIUM | LOW | P2 |
+| Traffic matrix generation guide | LOW | LOW | P3 |
+
+**Priority key:**
+- P1: Must have for this milestone
+- P2: Include if content is straightforward (most are); defer only if timeline is tight
+- P3: Nice to have, future consideration
+
+---
+
+## Sources
+
+- `/home/omrik/Projects/nixl.gitlab/benchmark/nixlbench/README.md` — direct source for NIXLBench features, CLI flags, examples, troubleshooting
+- `/home/omrik/Projects/nixl.gitlab/benchmark/kvbench/README.md` — direct source for KVBench commands, arguments, examples
+- `/home/omrik/Projects/nixl.gitlab/benchmark/kvbench/docs/tutorial-gds.md` — KVBench GDS tutorial content
+- `/home/omrik/Projects/nixl.gitlab/benchmark/kvbench/docs/creating-a-model-config.md` — KVBench model config guide content
+- `/home/omrik/Projects/nixl.gitlab/benchmark/kvbench/docs/adding-a-new-model-architecture.md` — KVBench extension guide content
+- `/home/omrik/Projects/nixl.gitlab/benchmark/kvbench/docs/ct-perftest.md` — CTPerftest implementation and usage details
+- `/home/omrik/Projects/nixl.gitlab/docs/index.yml` — existing site navigation (to identify cross-link targets and avoid duplication)
+- `/home/omrik/Projects/nixl.gitlab/.planning/PROJECT.md` — milestone scope and constraints
+
+---
+*Feature research for: NIXLBench and KVBench documentation pages (NIXL v1.1 milestone)*
+*Researched: 2026-04-07*
diff --git a/.planning/research/PITFALLS.md b/.planning/research/PITFALLS.md
new file mode 100644
index 0000000000..8bcaef2cf9
--- /dev/null
+++ b/.planning/research/PITFALLS.md
@@ -0,0 +1,347 @@
+# Pitfalls Research
+
+**Domain:** Adding benchmark/CLI tool documentation (NIXLBench + KVBench) to an existing Fern docs site
+**Researched:** 2026-04-07
+**Confidence:** HIGH — based on direct inspection of source READMEs, existing docs structure, `docs/index.yml`, and five diagnosed debug files from prior phases
+
+---
+
+## Critical Pitfalls
+
+### Pitfall 1: Duplicating Build Instructions Already in "Building NIXL from Source"
+
+**What goes wrong:**
+The NIXLBench Docker build requires cloning the NIXL repo and running `./build.sh` from `benchmark/nixlbench/contrib`. The existing docs already have a full "Building NIXL from Source" section (Docker, C++/Meson, Python Bindings, Rust Bindings) at `user-guide/building-nixl/`. Writers copy the CUDA Toolkit install steps, apt dependency lists, UCX build-from-source block, and Docker installation commands verbatim into the new NIXLBench page instead of linking.
+
+**Why it happens:**
+The README contains a complete, self-contained build guide spanning ~350 lines. A first draft that converts it to docs without auditing what already exists in the published site will copy everything. The overlap is not obvious unless you read both documents side-by-side.
+
+**How to avoid:**
+Before writing, do an explicit cross-reference audit: for each build step in the README, check whether an equivalent page already exists. The overlap map is:
+
+| NIXLBench README section | Existing doc page |
+|---|---|
+| Docker Installation (Ubuntu/RHEL) | Link to `user-guide/building-nixl/docker` |
+| CUDA Toolkit Installation | Link to `user-guide/building-nixl/nixl-cpp` or Quick Start |
+| UCX Build from Source | Link to backend page `user-guide/backends/ucx` |
+| Building NIXL (meson setup build) | Link to `user-guide/building-nixl/nixl-cpp` |
+
+The NIXLBench page should document only what is unique: the `etcd-cpp-api` installation, the NIXLBench-specific `meson setup build -Dnixl_path=...` invocation, and the `build.sh` container build options table. Everything else is a link.
+
+**Warning signs:**
+- NIXLBench build page exceeds ~80 lines of bash for prerequisites
+- The page contains `sudo apt-get install -y build-essential cmake ninja-build`
+- The page contains `wget https://developer.download.nvidia.com/compute/cuda/`
+
+**Phase to address:**
+NIXLBench page authoring phase — before writing, not during review.
+
+---
+
+### Pitfall 2: Duplicating ETCD Coordination Setup
+
+**What goes wrong:**
+NIXLBench uses ETCD for worker coordination, and the README contains a full ETCD setup section including the `docker run quay.io/coreos/etcd` command and a `sudo apt install etcd-server` block. The existing site already has a detailed "Metadata Exchange with etcd" page at `user-guide/etcd-metadata-exchange.md` covering prerequisites, environment variables, and deployment patterns. New NIXLBench docs reproduce ETCD setup inline rather than linking to the existing page.
+
+**Why it happens:**
+NIXLBench uses ETCD differently (for worker coordination/barrier synchronization) than the metadata exchange page describes (for agent metadata publication). Writers see a difference in purpose and assume duplication is justified. In practice the setup steps (running an etcd server, pointing clients at port 2379) are identical and readers benefit from a single authoritative source.
+
+**How to avoid:**
+The NIXLBench doc should include a short note explaining why ETCD is required (worker coordination/barrier), then link to the existing ETCD page for server setup. The only NIXLBench-specific ETCD content is the `--etcd_endpoints` flag and the 60-second barrier timeout behavior — those belong on the NIXLBench page. Everything else is a cross-reference.
+
+**Warning signs:**
+- NIXLBench page contains `docker run ... quay.io/coreos/etcd` command
+- NIXLBench page contains `sudo apt install etcd-server`
+- The phrase "Start ETCD server" appears as a heading on the NIXLBench page with a full fenced-code block
+
+**Phase to address:**
+NIXLBench page authoring phase — define a "link, don't duplicate" policy for ETCD setup content before writing.
+
+---
+
+### Pitfall 3: Terminology Drift from Established Standards
+
+**What goes wrong:**
+The existing docs use specific terms established across phases 21–27. Both READMEs and the resulting docs introduce inconsistent variants. The most common drifts from actual source material:
+
+| Established term (existing docs) | README / wrong variant |
+|---|---|
+| `plug-in` (hyphenated) | `plugin` (no hyphen) in README prose |
+| `etcd` (lowercase) | `ETCD` (all caps) in README headings and prose |
+| `GPUDirect Storage` | `GDS` used as a standalone noun in README |
+| "backend" (noun) | "Backend" (capitalized mid-sentence) |
+| `DRAM` / `VRAM` | Same — these are consistent already |
+
+When new pages use the README's capitalization and hyphenation without normalizing to the established style, readers see inconsistency and editors accumulate a growing gap to fix.
+
+**Why it happens:**
+The README was written by engineers for engineers; it was never copy-edited for the docs style. Converting it to docs content is a translation task, not just a formatting task. Writers who translate quickly miss style-level decisions.
+
+**How to avoid:**
+Run a terminology normalization pass as a dedicated step after first-draft writing, not during it. The rules to apply:
+- Replace `plugin` with `plug-in` everywhere in prose (code examples keep `--worker_type nixl` as-is — flag names are not prose)
+- Replace `ETCD` (all-caps) with `etcd` in prose and headings; keep `ETCD` only when referencing the `--runtime_type ETCD` flag value
+- Replace standalone `GDS` as noun with `GPUDirect Storage (GDS)` on first use per page, then `GDS` thereafter
+- Do not capitalize "backend" mid-sentence unless it is part of a proper name like "UCX backend"
+
+**Warning signs:**
+- Search new files for `\bplugin\b` (no hyphen) in prose sentences
+- Search for `\bETCD\b` outside of code blocks or flag value context
+- A heading like "ETCD Coordination Setup" rather than "etcd Coordination Setup"
+
+**Phase to address:**
+Terminology normalization pass — a dedicated review step at the end of each page's drafting, before the phase is marked complete.
+
+---
+
+### Pitfall 4: Overly Complex Page Structures That Bury the CLI Reference
+
+**What goes wrong:**
+Both NIXLBench and KVBench have large CLI flag surfaces. NIXLBench has 8 flag groups (core, memory, performance, device/network, storage, backend-specific for 7 backends). KVBench has 6 flag groups (common, CLI override, plan-specific, shared benchmark, CTP-specific). When this is all placed on a single "Usage" page or squeezed into a "Getting Started" page, the result is a 2,000+ line page that is exhausting to scan. Readers who need to find `--gds_batch_pool_size` cannot.
+
+**Why it happens:**
+The README is linear (top-to-bottom scroll) which works for a GitHub README. Fern docs are navigation-tree based; long single-page docs feel like reading a manual. Writers convert README sections 1:1 to a single page without considering how readers approach docs (goal-directed, not top-to-bottom).
+
+**How to avoid:**
+Plan the page hierarchy before writing. Recommended split:
+- NIXLBench: Overview, Build, Usage Guide (ETCD setup + basic patterns), CLI Reference (full flags table), Examples (one page, or backend-specific sub-pages linked from the nav)
+- KVBench: Overview, Build/Install, Command Reference (plan/profile/kvcache/ct-perftest as sections), Examples
+
+The CLI Reference pages should be pure reference (tables, no prose paragraphs). The Usage Guide page should have short prose + examples. Keep them separate so readers can bookmark the reference.
+
+**Warning signs:**
+- A single NIXLBench page exceeds 300 lines in the `.md` file
+- The words "Overview", "Build", "Usage", and "CLI Reference" all appear as `##` headings on the same page
+- The page's `## Command Line Options` section has 7+ sub-sections with their own `###` headings
+
+**Phase to address:**
+Page structure planning phase — define the page hierarchy and nav entries in `docs/index.yml` before writing content.
+
+---
+
+### Pitfall 5: Navigation Conflicts in `docs/index.yml`
+
+**What goes wrong:**
+New pages are added to `docs/index.yml` with incorrect nesting, duplicate slugs, or placement outside the intended "Developer Guide" section. Common specific mistakes:
+1. Adding NIXLBench/KVBench as top-level sections instead of entries under "Developer Guide"
+2. Using a `section:` node for NIXLBench without providing a `path:` (index page) — Fern renders the section title as non-clickable but still expects an index page
+3. Placing `collapsed: open-by-default` on NIXLBench sub-pages when they should be on the parent section node
+4. Using a file path that does not match the actual file location (e.g., `path: developer-guide/nixlbench.md` when the file is at `docs/developer-guide/nixlbench/overview.md`)
+
+**Why it happens:**
+`docs/index.yml` uses a specific Fern schema where `section:` with `contents:` requires either a `path:` (index page) or no path (section-title-only). The existing entries show the correct pattern (e.g., `Building NIXL from Source` uses `path: user-guide/building-nixl/index.md` with `collapsed: open-by-default`), but writers unfamiliar with this pattern write new entries inconsistently.
+
+**How to avoid:**
+Copy the exact structure of the existing "Building NIXL from Source" section as the template for NIXLBench and KVBench sections. The pattern is:
+```yaml
+- section: NIXLBench
+ collapsed: open-by-default
+ path: developer-guide/nixlbench/index.md
+ contents:
+ - page: Overview
+ path: developer-guide/nixlbench/overview.md
+ - page: Build
+ path: developer-guide/nixlbench/build.md
+```
+Verify each `path:` entry actually exists on disk before marking the phase complete.
+
+**Warning signs:**
+- `fern docs dev` throws a "page not found" or "could not resolve path" error on startup
+- A section heading in the sidebar is not clickable (missing `path:` on a `section:` node)
+- The "Developer Guide" section in the sidebar suddenly has no children (path conflict collapsed the tree)
+
+**Phase to address:**
+Navigation setup phase — add `docs/index.yml` entries as the first task of each page-set phase, before writing content, so `fern docs dev` can validate the structure.
+
+---
+
+### Pitfall 6: Stale CLI Reference Tables After Source Changes
+
+**What goes wrong:**
+The NIXLBench README documents 70+ flags across 8 groups. KVBench documents 30+ arguments across 6 groups. CLI reference tables in the docs become stale as the tools evolve — new flags are added (e.g., `--sequential-ct-perftest` was added to KVBench after `ct-perftest`), defaults change, or flags are renamed. The docs reflect the README at the time of writing, not the current binary.
+
+**Why it happens:**
+There is no automated mechanism linking the docs CLI tables to the actual binary. The source of truth is the `--help` output, but documentation is hand-authored from the README. When the README is updated, the docs are not automatically updated.
+
+**How to avoid:**
+Two strategies:
+1. During authoring, validate the CLI reference by running `nixlbench --help` and `python main.py --help` (and each subcommand) and comparing against the README. Mark any divergence.
+2. In the docs, add a note like "Run `nixlbench --help` for the canonical flag list — this page reflects version X.Y." This sets reader expectations and reduces the cost of minor staleness.
+
+Do not copy-paste the CLI tables directly from the README without running `--help` to validate. The README itself has known minor discrepancies (e.g., the KVBench README lists `--etcd-endpoints` while NIXLBench README uses `--etcd_endpoints` — underscore vs hyphen difference between Python CLI and C++ binary).
+
+**Warning signs:**
+- CLI reference table was copy-pasted from the README without a `--help` validation step in the task checklist
+- The KVBench `--etcd-endpoints` flag (hyphen, Python style) appears in the NIXLBench CLI reference (which uses underscore)
+- A flag appears in the table that does not appear in `--help` output
+
+**Phase to address:**
+CLI Reference authoring phase — include a `--help` validation step as an explicit checklist item before marking the phase complete.
+
+---
+
+### Pitfall 7: Missed Cross-References to Existing Backend Pages
+
+**What goes wrong:**
+NIXLBench and KVBench both support all NIXL backends (UCX, GDS, GDS_MT, POSIX, HF3FS, OBJ, GPUNETIO, Mooncake, Libfabric, GUSLI, AZURE_BLOB). The backend-specific examples in the NIXLBench docs explain how to run the benchmark against each backend, but do not link to the corresponding backend page in "User Guide: NIXL Backends". Readers who hit a GDS configuration error have no obvious path to the GDS backend page where prerequisites and configuration are documented.
+
+**Why it happens:**
+The README was written without hyperlinks to the broader NIXL docs ecosystem (it predates the docs site). Converting it to docs without adding cross-references misses the primary advantage of a multi-page docs site over a flat README.
+
+**How to avoid:**
+For every backend mentioned in the NIXLBench or KVBench examples, add a cross-reference on first use. The map is:
+
+| Backend in benchmark docs | Link target |
+|---|---|
+| UCX | `user-guide/backends/ucx` |
+| GDS / GPUDirect Storage | `user-guide/backends/gds` |
+| GDS_MT | `user-guide/backends/gds-mt` |
+| POSIX | `user-guide/backends/posix` |
+| HF3FS | `user-guide/backends/hf3fs` |
+| OBJ (S3) | `user-guide/backends/obj` |
+| GPUNETIO / DOCA GPUNetIO | `user-guide/backends/gpunetio` |
+| Mooncake | `user-guide/backends/mooncake` |
+| Libfabric | `user-guide/backends/libfabric` |
+| GUSLI | `user-guide/backends/gusli` |
+| Azure Blob | `user-guide/backends/azure-blob` |
+
+Also link ETCD coordination back to `user-guide/etcd-metadata-exchange` on first mention.
+
+**Warning signs:**
+- Backend names appear in the benchmark docs without a hyperlink on first use
+- The phrase "see the backend documentation" appears without a link target
+- GDS or POSIX examples appear without a prerequisite note pointing to the backend page
+
+**Phase to address:**
+Cross-reference audit phase — a dedicated pass after first-draft writing, specifically checking each backend name for a link.
+
+---
+
+### Pitfall 8: KVBench Described as a Benchmark When It Generates Commands for NIXLBench
+
+**What goes wrong:**
+KVBench's `profile` command does run NIXLBench internally, but the primary purpose of KVBench is to generate `nixlbench` commands configured for specific LLM architectures. Writing the KVBench docs as if it is a standalone benchmark tool obscures the relationship: KVBench is a command generator and configuration helper that wraps NIXLBench. Readers who only read the KVBench docs may not know they need NIXLBench installed.
+
+**Why it happens:**
+The KVBench README title says "A comprehensive utility for generating NIXL Bench commands" — this is accurate. But the `profile` command description says "actually runs the benchmark with nixlbench, collecting performance data" which sounds standalone. Writers see both descriptions and pick the more exciting framing.
+
+**How to avoid:**
+The KVBench overview page must explicitly state: KVBench is a Python utility that generates and optionally executes `nixlbench` commands. The `profile` command invokes `nixlbench` as a subprocess — NIXLBench must be installed and on the `PATH`. Link to the NIXLBench build page as a prerequisite.
+
+**Warning signs:**
+- The KVBench overview describes KVBench as a "performance testing tool" without mentioning its dependency on NIXLBench
+- The KVBench build page does not list NIXLBench as a prerequisite for `profile` command usage
+- No link from KVBench pages to the NIXLBench docs
+
+**Phase to address:**
+KVBench overview page authoring — the NIXLBench dependency relationship must be established in the first paragraph of the overview, not discovered by readers in a footnote.
+
+---
+
+## Technical Debt Patterns
+
+| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable |
+|---|---|---|---|
+| Copy README directly as doc page | Fast first draft | Terminology drift, duplication, no cross-references — requires a full rewrite later | Never — always normalize in the first pass |
+| Put all CLI flags on one page | Simpler nav tree | 2000+ line pages readers cannot scan; CLI reference and usage guide mixed together | Never for this volume of flags |
+| Skip `--help` validation, trust README | Saves ~30 minutes | Stale tables, flag name mismatches between tools (underscore vs hyphen) | Never — this is a one-time investment per phase |
+| Defer cross-references to "a later cleanup phase" | Faster writing | Cross-references are forgotten; orphaned pages become the norm | Never — add while writing, not after |
+| Use same page for overview and build instructions | One fewer page to maintain | Overview pages become 50% build docs; readers who already built NIXL cannot skip to usage | Acceptable only if the build is trivially short (< 10 lines) |
+
+---
+
+## Integration Gotchas
+
+| Integration | Common Mistake | Correct Approach |
+|---|---|---|
+| Fern `docs/index.yml` nav | Adding a `section:` with `contents:` but no `path:` causes a non-clickable section title | Always provide `path:` pointing to an index page when using `section:` with children; copy the "Building NIXL from Source" pattern |
+| Fern local dev (`fern docs dev`) | Testing edit-this-page, search, and Ask AI features in local dev — all are disabled | Test server-dependent features via `fern generate --docs --preview` (preview links) only; never report a bug from local dev observations alone (confirmed by debug files) |
+| KVBench `profile` invoking NIXLBench | Documenting `profile` without noting that NIXLBench binary must be on `PATH` | State the subprocess dependency explicitly; link to NIXLBench install page from KVBench prerequisites |
+| ETCD coordination (NIXLBench) | Writing a full ETCD setup guide in the NIXLBench page, duplicating `user-guide/etcd-metadata-exchange` | Link to the existing ETCD page; only document NIXLBench-specific behavior (60s barrier timeout, `--etcd_endpoints` flag) |
+| Backend flag values in CLI tables | Mixing Python CLI style (`--etcd-endpoints`, hyphen) with C++ binary style (`--etcd_endpoints`, underscore) in the same reference table | Use the exact flag string from `--help` output; NIXLBench uses underscores, KVBench uses hyphens — keep them separate |
+
+---
+
+## Performance Traps
+
+Not applicable to documentation authoring — this domain has no runtime performance concerns.
+
+---
+
+## Security Mistakes
+
+| Mistake | Risk | Prevention |
+|---|---|---|
+| Including real AWS credentials or S3 secrets in NIXLBench examples | Credentials committed to the GitLab repo | Use ``, `` placeholders in all examples; the README already uses environment variable form (`AWS_ACCESS_KEY_ID`) — follow that pattern |
+| Documenting `--obj_secret_key` with an example value | Same as above | Placeholder only; add a note to use environment variables instead of CLI flags for secrets |
+
+---
+
+## UX Pitfalls
+
+| Pitfall | User Impact | Better Approach |
+|---|---|---|
+| All backend-specific CLI options on one monolithic page | Readers trying to configure the GUSLI backend must scroll past GDS, POSIX, HF3FS, OBJ, and Azure Blob sections | Group backend-specific options with a clear anchor or separate page per backend group; use Fern `` component for backend variants if on one page |
+| KVBench examples only show DeepSeek R1 and LLaMA 3.1 | Readers with other architectures cannot extrapolate | Explicitly document that `--model` accepts any YAML following the model config schema; link to the "Creating a Model Configuration" developer guide |
+| NIXLBench GUSLI `--device_list` format undocumented in prose | The `id:type:path` format (e.g., `11:F:./store0.bin`) is cryptic without a schema explanation | Add a Device Types table (`F` = file, `K` = kernel block device, `N` = networked server with `t`/`u` prefix) as shown in the README's GUSLI Device Types section |
+| CTP YAML config format not cross-referenced from CLI docs | Readers know the `ct-perftest` command exists but cannot find the YAML config schema | The CLI reference for `ct-perftest` must link to or inline the YAML configuration schema (matrix file format, traffic pattern fields) |
+
+---
+
+## "Looks Done But Isn't" Checklist
+
+- [ ] **NIXLBench build page:** Confirm it does NOT reproduce Docker installation steps, CUDA toolkit install, or UCX build-from-source — these must be links to existing pages.
+- [ ] **ETCD setup content:** Confirm `docker run quay.io/coreos/etcd` command does NOT appear on the NIXLBench page — must be a link to `user-guide/etcd-metadata-exchange`.
+- [ ] **KVBench overview:** Confirm the first paragraph explicitly states KVBench generates/invokes `nixlbench` commands and links to the NIXLBench pages.
+- [ ] **CLI reference validation:** Confirm `--help` output was run for both binaries and compared against the tables before the phase was marked complete.
+- [ ] **Terminology pass:** Run `grep -rn '\bplugin\b' docs/developer-guide/` and `grep -rn '\bETCD\b' docs/developer-guide/` — both should return zero matches in prose (code blocks exempt).
+- [ ] **Cross-references:** Every backend name (UCX, GDS, GDS_MT, POSIX, HF3FS, OBJ, GPUNETIO, Mooncake, Libfabric, GUSLI, Azure Blob) has a link on first use.
+- [ ] **Nav validation:** `fern docs dev` starts without errors after each `docs/index.yml` edit.
+- [ ] **Page length check:** No single docs page exceeds ~250 lines (overview, build, usage, and CLI reference are separate files).
+- [ ] **Secrets:** No real credentials appear in any example — all use `` or environment variable form.
+
+---
+
+## Recovery Strategies
+
+| Pitfall | Recovery Cost | Recovery Steps |
+|---|---|---|
+| Duplication discovered post-publish | MEDIUM | Audit overlapping content, convert duplicated sections to links, verify no information is lost in the source page before removing it |
+| Terminology drift found in review | LOW | Sed-pass on new files only; do not modify existing pages unless a separate terminology phase is scoped |
+| Navigation conflict (Fern build failure) | LOW | Check `docs/index.yml` path values against filesystem; Fern error messages include the unresolved path |
+| Stale CLI table discovered by user | LOW–MEDIUM | Run `--help`, diff against table, update affected rows; add a version note to the page header |
+| KVBench/NIXLBench relationship unclear to readers | MEDIUM | Rewrite KVBench overview page; add a "How it works" diagram or callout box explaining the subprocess relationship |
+
+---
+
+## Pitfall-to-Phase Mapping
+
+| Pitfall | Prevention Phase | Verification |
+|---|---|---|
+| Duplicating build instructions | NIXLBench Build page authoring | Grep new page for `apt-get install build-essential` — must be zero matches |
+| Duplicating ETCD setup | NIXLBench Usage page authoring | Grep new page for `quay.io/coreos/etcd` — must be zero matches |
+| Terminology drift | Terminology normalization pass (end of each page phase) | `grep -rn '\bplugin\b\|\bETCD\b' docs/developer-guide/` in prose context |
+| Overly complex page structure | Navigation planning phase (before writing) | Page count for NIXLBench >= 4 separate files (overview, build, usage, CLI ref) |
+| Navigation conflicts | Nav setup as first task of each page phase | `fern docs dev` starts cleanly after each `docs/index.yml` change |
+| Stale CLI reference | CLI reference phase — `--help` validation step | Checklist item: "ran `nixlbench --help` and `python main.py [cmd] --help` and compared all flags" |
+| Missed backend cross-references | Cross-reference audit (dedicated pass after first draft) | Every backend name is a hyperlink on first use |
+| KVBench/NIXLBench relationship | KVBench overview authoring | First paragraph of overview explicitly names NIXLBench as a dependency |
+
+---
+
+## Sources
+
+- Direct inspection: `benchmark/nixlbench/README.md` (v1.1, 2026-04-07)
+- Direct inspection: `benchmark/kvbench/README.md` (v1.1, 2026-04-07)
+- Direct inspection: `docs/index.yml` (current nav structure)
+- Direct inspection: `docs/user-guide/building-nixl/index.md` (existing build docs)
+- Direct inspection: `docs/user-guide/etcd-metadata-exchange.md` (existing ETCD docs)
+- Direct inspection: `.planning/debug/fern-local-dev-limitations.md` (confirmed: search, edit-this-page, ask-ai all disabled in `fern docs dev`)
+- Direct inspection: `.planning/debug/edit-this-page-not-visible.md` and `edit-this-page-round2.md`
+- Direct inspection: `.planning/debug/search-bar-no-entry-symbol.md`
+- Direct inspection: `.planning/debug/page-actions-toolbar-layout.md`
+- Direct inspection: `.planning/PROJECT.md` (milestone context, established conventions)
+
+---
+*Pitfalls research for: Adding NIXLBench and KVBench documentation to existing NIXL Fern docs site*
+*Researched: 2026-04-07*
diff --git a/.planning/research/STACK.md b/.planning/research/STACK.md
new file mode 100644
index 0000000000..36e26cb49d
--- /dev/null
+++ b/.planning/research/STACK.md
@@ -0,0 +1,100 @@
+# Stack Research
+
+**Domain:** CLI tool documentation on Fern — benchmark tools with complex option tables, multi-step build instructions, and code-heavy examples
+**Researched:** 2026-04-07
+**Confidence:** HIGH (all findings verified against existing codebase; Fern component inventory drawn from live docs in `docs/`)
+
+## Recommended Stack
+
+### Core Technologies
+
+| Technology | Version | Purpose | Why Recommended |
+|------------|---------|---------|-----------------|
+| Fern MDX | Current (site already live) | Page authoring format | Already validated platform; all tooling and CI in place |
+| Standard Markdown tables | N/A | CLI option reference tables | Fern renders GFM tables natively; no component wrapper needed for flat option lists |
+| Fern `` + `` | Built-in | Build method selection (Docker vs native) | Both tools have two install paths; tabs present alternatives without vertical scrolling or duplicated prose |
+| Fern `` | Built-in | Multi-language or multi-variant code blocks | Used on Quick Start for Python/C++/Rust; appropriate when one concept has multiple equivalent invocations |
+| Fern callouts (``, ``, ``) | Built-in | Inline contextual guidance | Already used consistently across all v1.0 pages; `` for ETCD required/optional distinctions, `` for workflow shortcuts |
+| Fern `` | Built-in | Image captions | Used on GDS example for sequence diagrams; applicable if architecture diagrams are added later |
+| MDX snippets (`` per option | Only if individual options need long prose descriptions; NIXLBench/KVBench options are terse enough for table rows |
+| `` for Docker vs native build | Separate pages per build method | Separate pages only if build paths diverge significantly (>10 steps each); current paths are short enough for tabs |
+| `` / `` callouts | Inline bold text | Callouts for must-not-miss operational requirements (ETCD coordination, CUDA_VISIBLE_DEVICES); inline bold for softer hints |
+| MDX snippets for shared option groups | Duplicated table content | Snippets only if the same option table appears verbatim on 2+ pages; NIXLBench and KVBench share `--backend` values but the tables differ enough to stay separate |
+| `title=` attribute on code blocks | Inline comment headers | `title=` is visually distinct and matches existing site convention; don't mix approaches |
+
+## What NOT to Use
+
+| Avoid | Why | Use Instead |
+|-------|-----|-------------|
+| `` for single-language examples | Adds tab UI overhead when there is only one language; NIXLBench is C++ CLI only, KVBench is Python CLI only | Plain fenced code block with `title=` attribute |
+| Custom React components (new ones) | `fern/components/` only has `CustomFooter.tsx`; introducing new TSX increases maintenance surface with no DX gain | Fern built-in components cover all benchmark doc needs |
+| `` as a mandatory wrapper | GDS example uses it for SVG sequence diagrams; benchmark pages have no diagrams in source material | Use `` only if architecture diagrams are added; don't wrap code output blocks |
+| Option descriptions as prose paragraphs | Benchmark tools have 30+ options each; prose becomes unscannable | Markdown tables grouped by option category |
+| Numbered steps written as plain paragraphs | Readers skip steps in prose; build instructions must be sequential and checkable | Use numbered lists (`1.`, `2.`, `3.`) or H3-level step headings for multi-command build sequences |
+| A single "all options" mega-table | NIXLBench has 8 backend-specific option groups; one flat table is 50+ rows and unusable | Separate H3-level tables per group (Core, Memory/Transfer, Performance, Storage, per-backend) |
+
+## Stack Patterns by Variant
+
+**NIXLBench (C++ binary, complex CLI):**
+- `` with titles "Docker (Recommended)" and "Native Build" for the build section
+- H3 tables per option group (Core Configuration, Memory and Transfer, Performance and Threading, Storage Backends, Backend-Specific) — mirrors README structure that readers already know
+- `` for the ETCD requirement: network backends require ETCD; storage backends do not — this is a common stumbling block
+- `` for the TOML config file equivalence (command-line args map 1:1 to config file keys)
+- Backend-specific examples as H2 sections with `bash` code blocks using `title=` (e.g., `title="UCX: GPU-to-GPU"`)
+
+**KVBench (Python CLI with subcommands):**
+- `` with titles "Docker" and "Python (venv)" for the build section — KVBench's Docker path re-uses NIXLBench's container, worth a `` cross-link
+- Subcommand reference as H2 sections (`plan`, `profile`, `kvcache`, `ct-perftest`, `sequential-ct-perftest`) rather than one flat list
+- Common arguments table at top-level, then per-subcommand tables for subcommand-specific args — matches how Click CLI help is structured
+- `` for `CUDA_VISIBLE_DEVICES` requirement on CTP tests (already flagged as "Important note" in README)
+- `yaml` language tag for all CTP YAML config file examples; `text` for tabular terminal output (kvcache table, ct-perftest results)
+
+## Version Compatibility
+
+| Component | Constraint | Notes |
+|-----------|------------|-------|
+| Fern MDX components | Current site version | All components (``, ``, ``, ``, ``, ``, ``, ``) verified as live on site |
+| `experimental.mdx-components` | Set to `./components` in docs.yml | Custom components path is configured; no changes to docs.yml needed for standard built-in components |
+| Front-matter `title` + `description` | Required per Fern convention | Every existing page has both; benchmark pages must match |
+
+## Sources
+
+- Existing live docs (`docs/getting-started/quick-start.md`) — verified ``, ``, ``, ``, ``, `` syntax in production
+- Existing live docs (`docs/user-guide/backends/ucx.md`) — verified Markdown table pattern for option reference
+- Existing live docs (`docs/resources/environment-variables.md`) — verified large multi-group table structure
+- Existing live docs (`docs/examples/gds-direct-storage.md`) — verified ``, ``, `` callout usage
+- Existing live docs (`docs/user-guide/building-nixl/docker.md`) — verified `` and `` for build guidance
+- `fern/docs.yml` — verified `experimental.mdx-components`, layout, and component path
+- `fern/snippets/env-vars-ucx.mdx` — verified ``, ``, ``, ``, fenced code blocks with `title=`) are already live on the production site
+- Do NOT use `` for single-language examples (NIXLBench is C++ CLI only, KVBench is Python CLI only)
+- Use `text` language tag for terminal output blocks (kvcache table, latency output) to prevent false syntax highlighting
+- `` for Docker vs. native build (both tools have exactly two install paths)
+- `` for ETCD 60s barrier timeout and KVBench `CUDA_VISIBLE_DEVICES` requirement
+
+---
+
+## Feature Table Stakes (P1 — this milestone)
+
+**NIXLBench:**
+- Overview page
+- Building page (Docker + native combined, with ``)
+- CLI reference (core flags + storage flags + config file flags — grouped, not one mega-table)
+- ETCD coordination callout
+- Backend-specific examples (8 backends)
+- Troubleshooting
+
+**KVBench:**
+- Overview page (NIXLBench subprocess dependency in paragraph one — mandatory)
+- Building page (Docker + Python venv)
+- Command reference (all 5 commands: plan, profile, kvcache, ct-perftest, sequential-ct-perftest)
+- Model configuration guide
+- LLM architecture examples (DeepSeek R1, Llama 3.1)
+- CTP examples
+
+**P2 (include if scope allows):** KVBench GDS tutorial (source: `benchmark/kvbench/docs/tutorial-gds.md`), KVBench extension guide (source: `benchmark/kvbench/docs/adding-a-new-model-architecture.md`)
+
+---
+
+## Architecture
+
+- **9 new files** under `docs/development/benchmarks/nixlbench/` (5 pages) and `docs/development/benchmarks/kvbench/` (4 pages)
+- **1 modified file:** `docs/index.yml` — two new `section:` blocks under Developer Guide after "Building a Backend Plugin"
+- No changes to `fern/docs.yml`, no existing page modifications, no new components
+- Authoring order: `docs/index.yml` first → NIXLBench pages → KVBench pages
+
+---
+
+## Watch Out For
+
+1. **Content duplication** — READMEs contain full Docker/CUDA/UCX/NIXL/ETCD walkthroughs already in existing docs. Audit before writing; substitute cross-links.
+2. **KVBench misdescribed as standalone** — `profile` invokes `nixlbench` as a subprocess. State this in the first paragraph of KVBench overview.
+3. **Monolithic pages** — NIXLBench has 70+ flags, KVBench has 30+ args. Enforce page split by creating `docs/index.yml` entries before writing content.
+4. **Navigation YAML** — Copy "Building NIXL from Source" section pattern exactly (`section:` + `path:` + `contents:`).
+5. **Stale CLI tables** — Run `nixlbench --help` and `python main.py [cmd] --help` before marking CLI reference phases complete. Known discrepancy: NIXLBench uses `--etcd_endpoints` (underscores); KVBench uses `--etcd-endpoints` (hyphens).
+6. **Terminology drift** — `plugin` → `plug-in`, `ETCD` → `etcd` in prose; batch normalization pass after all drafts.
+7. **Backend cross-references** — 11 backends, each should link to its User Guide page on first mention.
+
+---
+
+## Suggested Phase Structure (6 phases, starting at Phase 32)
+
+| Phase | Name | Goal |
+|-------|------|------|
+| 32 | Navigation Setup and Directory Structure | Create `docs/development/benchmarks/` tree and `docs/index.yml` entries before any content |
+| 33 | NIXLBench Core Pages | Overview, Build (Docker + native), Usage guide |
+| 34 | NIXLBench CLI Reference and Backend Examples | All flags validated against `--help`, backend cross-refs |
+| 35 | KVBench Core Pages | Overview (NIXLBench dependency stated), Build, Command reference |
+| 36 | KVBench Model Config and LLM Examples | Model config guide, DeepSeek R1 + Llama examples, CTP examples |
+| 37 | Terminology Normalization and Cross-Reference Audit | Batch grep pass, all backend names hyperlinked, `fern check` clean |
+
+---
+
+## Confidence: HIGH
+
+All source material is in the codebase. All Fern platform patterns are verified live on the production site. This is an authoring and content organization task throughout — no phases require additional research.
diff --git a/docs/api-reference/cpp-api.md b/docs/api-reference/cpp-api.md
new file mode 100644
index 0000000000..329d7a51f2
--- /dev/null
+++ b/docs/api-reference/cpp-api.md
@@ -0,0 +1,982 @@
+---
+title: C++ API Reference
+description: Complete C++ API reference for NIXL data transfers.
+---
+
+This is the C++ API reference for the NVIDIA Inference Xfer Library (NIXL). C++ is the native API -- all other language bindings are built on top of it. For Python bindings, see the [Python API Reference](./python-api). For Rust bindings, see the [Rust API Reference](./rust-api).
+
+Behavior shared by all bindings is documented in [Northbound API Semantics](./northbound-api).
+
+To use the NIXL C++ API, include a single header:
+
+```cpp
+#include "nixl.h"
+```
+
+All public types, enumerations, and the `nixlAgent` class are available through this header.
+
+## Types, Enums and Defines
+
+This section documents the enumerations, type aliases, configuration structs, descriptor classes, constants, and handle types used throughout the NIXL C++ API. These are defined in `nixl_types.h`, `nixl_params.h`, and `nixl_descriptors.h`.
+
+### Enumerations
+
+### nixl_mem_t
+
+Memory segment types supported by NIXL. Each type represents a different class of memory or storage that NIXL can register and transfer.
+
+| Value | Description |
+|-------|-------------|
+| `DRAM_SEG` | Standard host memory (CPU DRAM) |
+| `VRAM_SEG` | GPU high-bandwidth memory (HBM/VRAM) |
+| `BLK_SEG` | Block-level storage devices |
+| `OBJ_SEG` | Distributed object stores (S3, Azure Blob) |
+| `FILE_SEG` | Local and remote file systems |
+
+```cpp
+nixl_reg_dlist_t descs(VRAM_SEG); // Create descriptor list for GPU memory
+```
+
+### nixl_xfer_op_t
+
+Transfer operation direction.
+
+| Value | Description |
+|-------|-------------|
+| `NIXL_READ` | Read data from the remote side into local buffers |
+| `NIXL_WRITE` | Write data from local buffers to the remote side |
+
+```cpp
+agent.createXferReq(NIXL_WRITE, local_descs, remote_descs, "target", req);
+```
+
+### nixl_status_t
+
+Status codes and error values returned by NIXL API methods.
+
+| Value | Code | Description |
+|-------|------|-------------|
+| `NIXL_IN_PROG` | 1 | Transfer is in progress |
+| `NIXL_SUCCESS` | 0 | Operation completed successfully |
+| `NIXL_ERR_NOT_POSTED` | -1 | Transfer was not posted |
+| `NIXL_ERR_INVALID_PARAM` | -2 | Invalid parameter provided |
+| `NIXL_ERR_BACKEND` | -3 | Backend-level error |
+| `NIXL_ERR_NOT_FOUND` | -4 | Requested resource not found |
+| `NIXL_ERR_MISMATCH` | -5 | Mismatched parameters or types |
+| `NIXL_ERR_NOT_ALLOWED` | -6 | Operation not allowed in current state |
+| `NIXL_ERR_REPOST_ACTIVE` | -7 | Attempting to repost an active transfer |
+| `NIXL_ERR_UNKNOWN` | -8 | Unknown error |
+| `NIXL_ERR_NOT_SUPPORTED` | -9 | Operation not supported by backend |
+| `NIXL_ERR_REMOTE_DISCONNECT` | -10 | Remote agent disconnected |
+| `NIXL_ERR_CANCELED` | -11 | Transfer was canceled |
+| `NIXL_ERR_NO_TELEMETRY` | -12 | Telemetry data not available |
+
+```cpp
+nixl_status_t status = agent.getXferStatus(req);
+if (status == NIXL_SUCCESS) { /* transfer complete */ }
+```
+
+### nixl_thread_sync_t
+
+Thread synchronization modes for multi-threaded agent usage.
+
+| Value | Description |
+|-------|-------------|
+| `NIXL_THREAD_SYNC_NONE` | No synchronization (default). Single-threaded usage only. |
+| `NIXL_THREAD_SYNC_STRICT` | Full mutual exclusion on all operations. |
+| `NIXL_THREAD_SYNC_RW` | Reader-writer lock: concurrent reads, exclusive writes. |
+
+
+`NIXL_THREAD_SYNC_DEFAULT` is an alias for `NIXL_THREAD_SYNC_NONE`. This is an `enum class`, so values must be qualified: `nixl_thread_sync_t::NIXL_THREAD_SYNC_STRICT`.
+
+
+```cpp
+nixlAgentConfig cfg;
+cfg.syncMode = nixl_thread_sync_t::NIXL_THREAD_SYNC_RW;
+```
+
+### nixl_cost_t
+
+Cost estimation method identifiers.
+
+| Value | Description |
+|-------|-------------|
+| `ANALYTICAL_BACKEND` | Analytical backend cost estimate (value 0) |
+
+
+This is an `enum class`, so the value must be qualified: `nixl_cost_t::ANALYTICAL_BACKEND`.
+
+
+### Type Aliases
+
+| Alias | Underlying Type | Purpose |
+|-------|----------------|---------|
+| `nixl_backend_t` | `std::string` | Backend identifier string |
+| `nixl_blob_t` | `std::string` | Binary data blob (supports embedded `\0`) |
+| `nixl_mem_list_t` | `std::vector` | List of memory types |
+| `nixl_b_params_t` | `std::unordered_map` | Backend key-value parameters |
+| `nixl_notifs_t` | `std::unordered_map>` | Notification map (agent name to messages) |
+| `nixl_opt_args_t` | `nixlAgentOptionalArgs` | Optional method arguments struct |
+| `nixl_xfer_dlist_t` | `nixlDescList` | Transfer descriptor list |
+| `nixl_reg_dlist_t` | `nixlDescList` | Registration descriptor list |
+| `nixl_remote_dlist_t` | `nixlDescList` | Remote descriptor list |
+| `nixl_local_dlist_t` | `nixlDescList` | Local descriptor list |
+| `nixl_query_resp_t` | `std::optional` | Query response (empty if no data) |
+| `nixl_xfer_telem_t` | `nixlXferTelemetry` | Transfer telemetry data |
+| `nixlMemViewH` | `void*` | Memory view handle |
+
+
+`nixl_blob_t` is a typedef for `std::string` but is intended to hold binary data (including embedded null bytes). Do not call `.c_str()` on it -- use `.data()` and `.size()` instead.
+
+
+### Configuration Structs
+
+### nixlAgentConfig
+
+Per-agent configuration struct. All fields have defaults, so a default-constructed `nixlAgentConfig` is valid.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `useProgThread` | `bool` | `false` | Enable progress thread for asynchronous transfer advancement |
+| `useListenThread` | `bool` | `false` | Enable listener thread for incoming peer connections |
+| `listenPort` | `int` | `0` | Port for the listener thread (0 = system-assigned) |
+| `syncMode` | `nixl_thread_sync_t` | `NIXL_THREAD_SYNC_NONE` | Thread synchronization mode for multi-threaded usage |
+| `captureTelemetry` | `bool` | `false` | Enable telemetry capture regardless of environment variables |
+| `pthrDelay` | `uint64_t` | `0` | Progress thread delay between iterations (microseconds) |
+| `lthrDelay` | `uint64_t` | `100000` | Listener thread sleep duration (microseconds) |
+| `etcdWatchTimeout` | `std::chrono::microseconds` | `5000000` (5 seconds) | Timeout for etcd watch operations |
+
+```cpp
+nixlAgentConfig cfg;
+cfg.useProgThread = true;
+cfg.useListenThread = true;
+cfg.listenPort = 5555;
+nixlAgent agent("my_agent", cfg);
+```
+
+
+`nixlAgentConfig` also provides a parameterized constructor that accepts all fields as arguments. Using the default constructor and setting fields individually is recommended for clarity.
+
+
+### nixlAgentOptionalArgs
+
+Optional arguments struct passed to many agent methods via `const nixl_opt_args_t*`. Pass `nullptr` to use defaults.
+
+**Current fields:**
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `backends` | `std::vector` | empty | Limit operation to specified backends |
+| `notif` | `std::optional` | empty | Notification message for transfer operations. If set, enables notification even for empty strings. |
+| `ipAddr` | `std::string` | empty | IP address for peer-to-peer metadata operations |
+| `port` | `int` | `8888` | Port for metadata operations (requires `ipAddr` to be set) |
+| `metadataLabel` | `std::string` | empty | Label for etcd metadata scoping |
+| `customParam` | `nixl_blob_t` | empty | Custom backend parameter string |
+
+
+The following fields are kept for backward compatibility but should not be used in new code:
+
+- `notifMsg` (`nixl_blob_t`) -- Use `notif` instead. The `notif` optional provides cleaner notification semantics.
+- `hasNotif` (`bool`, default `false`) -- Use `notif` instead. Setting `notif` to any value (including empty string) enables notifications.
+- `skipDescMerge` (`bool`, default `false`) -- Deprecated. Descriptor merging behavior is handled internally.
+- `includeConnInfo` (`bool`, default `false`) -- Deprecated. Connection info inclusion is handled via `backends` and descriptor list contents.
+
+
+```cpp
+nixl_opt_args_t args;
+args.notif = "transfer_complete"; // Enable notification with message
+args.backends.push_back(backend); // Limit to specific backend
+agent.postXferReq(req, &args);
+```
+
+### nixlXferTelemetry
+
+Transfer telemetry data returned by `getXferTelemetry()`.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `startTime` | `chrono_point_t` | Time point when the transfer was posted |
+| `postDuration` | `chrono_period_us_t` | Duration of the post operation (microseconds) |
+| `xferDuration` | `chrono_period_us_t` | Duration of the entire transfer (microseconds) |
+| `totalBytes` | `size_t` | Total bytes transferred in the request |
+| `descCount` | `size_t` | Number of descriptors in the transfer (reflects merging if any) |
+
+
+`chrono_point_t` is `std::chrono::steady_clock::time_point` and `chrono_period_us_t` is `std::chrono::microseconds`. If `getXferStatus()` is called late after completion, `xferDuration` may overestimate the actual transfer time.
+
+
+### Constants
+
+| Constant | Type | Value | Description |
+|----------|------|-------|-------------|
+| `NIXL_INIT_AGENT` | `#define` | `""` (empty string) | Indicates local agent in `prepXferDlist` |
+| `default_comm_port` | `constexpr int` | `8888` | Default port for peer-to-peer communication |
+| `default_metadata_label` | `const std::string` | -- | Default etcd key label for full metadata |
+| `default_partial_metadata_label` | `const std::string` | -- | Default etcd key label for partial metadata |
+
+### Handle Types
+
+The following are opaque handle types returned by NIXL methods. They should not be constructed directly.
+
+| Handle | Description |
+|--------|-------------|
+| `nixlBackendH*` | Backend handle returned by `createBackend()` |
+| `nixlDlistH*` | Prepared descriptor list handle returned by `prepXferDlist()` |
+| `nixlXferReqH*` | Transfer request handle returned by `makeXferReq()` / `createXferReq()` |
+| `nixlMemViewH` | Memory view handle returned by `prepMemView()` (typedef for `void*`) |
+
+## NIXL Descriptors
+
+NIXL uses a hierarchy of descriptor classes to represent memory regions for registration and transfer.
+
+(buf), 256, 0);
+```
+
+### nixlBlobDesc
+
+Extends `nixlBasicDesc` with metadata blob. Used for memory registration.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| *(inherits `addr`, `len`, `devId`)* | | |
+| `metaInfo` | `nixl_blob_t` | Metadata blob (e.g., file path for FILE_SEG) |
+
+```cpp
+nixlBlobDesc desc(reinterpret_cast(buf), 256, 0, "metadata");
+```
+
+### nixlRemoteDesc
+
+Extends `nixlBasicDesc` with remote agent name. Used for memory view operations.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| *(inherits `addr`, `len`, `devId`)* | | |
+| `remoteAgent` | `std::string` | Name of the remote agent owning this buffer |
+
+```cpp
+nixlRemoteDesc desc(addr, 256, 0, "remote_agent");
+```
+
+### nixlDescList\
+
+Template container for descriptor lists. Parameterized by descriptor type.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `nixlDescList(nixl_mem_t type, int init_size = 0)` | -- | Constructor with memory type and optional initial capacity |
+| `addDesc(const T& desc)` | `void` | Add a descriptor to the list |
+| `descCount()` | `int` | Number of descriptors in the list |
+| `isEmpty()` | `bool` | Check if the list is empty |
+| `getType()` | `nixl_mem_t` | Get the memory type of this list |
+| `operator[](size_t index)` | `T&` / `const T&` | Access descriptor by index |
+| `begin()` / `end()` | iterator | Standard iteration support |
+| `remDesc(int index)` | `void` | Remove descriptor at index (may throw `std::out_of_range`) |
+| `trim()` | `nixlDescList` | Convert to basic descriptor list (strips metadata) |
+| `getIndex(const nixlBasicDesc& query)` | `int` | Find index of matching descriptor (negative on error) |
+| `clear()` | `void` | Remove all descriptors |
+| `resize(size_t count)` | `void` | Resize the descriptor list |
+
+```cpp
+nixl_reg_dlist_t descs(DRAM_SEG);
+descs.addDesc(nixlBlobDesc(reinterpret_cast(buf), 256, 0, ""));
+// descs.descCount() == 1
+```
+
+
+## Initialization and Configuration
+
+
+For a complete workflow example, see [Quick Start -- Agent Initialization](../getting-started/quick-start#agent-initialization).
+
+
+### nixlAgent
+
+Constructor for the Transfer Agent. Each agent represents one endpoint in a data transfer and manages backends, memory registrations, metadata, and transfer operations.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `name` | `const std::string&` | Unique name for this agent. Used to identify the agent in metadata exchange and transfers. |
+| `cfg` | `const nixlAgentConfig&` | Agent configuration (threading, telemetry, etc.) |
+| **Returns** | `nixlAgent` | Constructed agent object |
+
+
+`nixlAgent` is non-movable and non-copyable. It must be created in place or managed via a pointer. The destructor handles all resource cleanup.
+
+
+```cpp
+nixlAgentConfig cfg;
+cfg.useProgThread = true;
+nixlAgent agent("my_agent", cfg);
+```
+
+### ~nixlAgent
+
+Destructor. Cleans up all backends, registered memory, metadata, and internal resources.
+
+```cpp
+{
+ nixlAgent agent("my_agent", cfg);
+ // ... use agent ...
+} // destructor called here
+```
+
+### getAvailPlugins
+
+Discover the available backend plug-ins found in the plug-in search paths.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `plugins` | `std::vector&` | [out] Populated with the names of available backend plug-ins |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+std::vector plugins;
+agent.getAvailPlugins(plugins);
+// plugins might contain: {"UCX", "GDS", "POSIX", ...}
+```
+
+## Backend Management
+
+
+For a complete workflow example, see [Quick Start -- Backend Creation](../getting-started/quick-start#backend-creation).
+
+
+### getPluginParams
+
+Get the supported memory types and initialization parameters (with defaults) for a backend plug-in, before creating an instance.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `type` | `const nixl_backend_t&` | Plugin backend type name (e.g., `"UCX"`) |
+| `mems` | `nixl_mem_list_t&` | [out] Memory types supported by this plug-in |
+| `params` | `nixl_b_params_t&` | [out] Init parameters and their default values |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_mem_list_t mems;
+nixl_b_params_t params;
+agent.getPluginParams("UCX", mems, params);
+```
+
+### getBackendParams
+
+Get the parameters of an already-instantiated backend. This returns a comprehensive list including defaults that were not explicitly specified during creation.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `backend` | `const nixlBackendH*` | Backend handle obtained from `createBackend()` |
+| `mems` | `nixl_mem_list_t&` | [out] Memory types supported by this backend |
+| `params` | `nixl_b_params_t&` | [out] Parameters and their current values |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_mem_list_t mems;
+nixl_b_params_t params;
+agent.getBackendParams(backend, mems, params);
+```
+
+### createBackend
+
+Instantiate a backend engine with the given parameters.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `type` | `const nixl_backend_t&` | Backend type name (e.g., `"UCX"`, `"GDS"`) |
+| `params` | `const nixl_b_params_t&` | Backend-specific initialization parameters |
+| `backend` | `nixlBackendH*&` | [out] Backend handle for subsequent operations |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+Multiple backends can be created on the same agent. NIXL automatically selects the best backend for each transfer based on the source and destination memory types and the backends available on both agents.
+
+
+```cpp
+nixlBackendH* backend;
+nixl_b_params_t params;
+agent.createBackend("UCX", params, backend);
+```
+
+## Memory Registration
+
+
+For a complete workflow example, see [Quick Start -- Memory Registration](../getting-started/quick-start#memory-registration).
+
+
+### registerMem
+
+Register memory or storage segments with NIXL. Registration creates internal data structures that backends use to track memory regions and generate metadata for remote access.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_reg_dlist_t&` | Descriptor list of buffers to register |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `backends` is set, registration is limited to those backends. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+If no backend hints are provided, NIXL auto-selects all compatible backends for the given memory type. Memory must be registered before metadata exchange -- metadata exchanged before registration will not include the segment information needed for transfers.
+
+
+```cpp
+nixl_reg_dlist_t descs(DRAM_SEG);
+descs.addDesc(nixlBlobDesc(reinterpret_cast(buf), 256, 0, ""));
+agent.registerMem(descs);
+```
+
+### deregisterMem
+
+Deregister previously registered memory or storage segments from NIXL.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_reg_dlist_t&` | Descriptor list of buffers to deregister (must match registration descriptors) |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `backends` is set, deregistration is limited to those backends. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+agent.deregisterMem(descs);
+```
+
+### queryMem
+
+Query information about registered memory or storage segments from a specific backend.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_reg_dlist_t&` | Descriptor list of buffers to query |
+| `resp` | `std::vector&` | [out] Query response for each descriptor. Use `.has_value()` to check validity and `.value()` to access the key-value dictionary. |
+| `extra_params` | `const nixl_opt_args_t*` | Required. The target backend should be specified via `extra_params->backends`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+std::vector resp;
+nixl_opt_args_t args;
+args.backends.push_back(backend);
+agent.queryMem(descs, resp, &args);
+```
+
+### makeConnection
+
+Proactively establish a connection to a remote agent, instead of deferring it to the first transfer.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the remote agent to connect to |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `backends` is set, connection is limited to those backends. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+Connections are normally established lazily on the first transfer. Use `makeConnection()` to pre-establish connections and avoid first-transfer latency.
+
+
+```cpp
+agent.makeConnection("remote_agent");
+```
+
+## Transfer Preparation
+
+
+For a complete workflow example, see [Quick Start -- Creating and Executing Transfers](../getting-started/quick-start#creating-and-executing-transfers).
+
+
+### prepXferDlist (4-parameter)
+
+Prepare a descriptor list for use in transfer requests. Elements from the prepared list can later be selected by index in `makeXferReq()`. This overload accepts an explicit agent name for the descriptor side.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `agent_name` | `const std::string&` | Agent name for the prepared list. Use `NIXL_INIT_AGENT` (empty string) for local descriptors, the remote agent's name for remote descriptors, or the local agent's own name for loopback transfers. |
+| `descs` | `const nixl_xfer_dlist_t&` | Descriptor list to prepare |
+| `dlist_hndl` | `nixlDlistH*&` | [out] Prepared descriptor list handle |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `backends` is set, preparation is limited to those backends. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+Preparation succeeds if at least one backend can handle all elements in the descriptor list. Use this method when you need fine-grained control over which descriptors participate in each transfer via index selection.
+
+
+```cpp
+nixlDlistH* local_hndl;
+agent.prepXferDlist(NIXL_INIT_AGENT, local_descs, local_hndl);
+
+nixlDlistH* remote_hndl;
+agent.prepXferDlist("remote_agent", remote_descs, remote_hndl);
+```
+
+### prepXferDlist (3-parameter)
+
+Convenience overload that prepares a local descriptor list. Equivalent to calling the 4-parameter overload with `NIXL_INIT_AGENT` as the agent name.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_xfer_dlist_t&` | Descriptor list to prepare |
+| `dlist_hndl` | `nixlDlistH*&` | [out] Prepared descriptor list handle |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixlDlistH* local_hndl;
+agent.prepXferDlist(local_descs, local_hndl);
+```
+
+### makeXferReq
+
+Create a transfer request by selecting indices from already-prepared descriptor list handles. NIXL automatically determines the backend for the transfer.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `const nixl_xfer_op_t&` | Transfer direction (`NIXL_READ` or `NIXL_WRITE`) |
+| `local_side` | `const nixlDlistH*` | Local prepared descriptor list handle |
+| `local_indices` | `const std::vector&` | Indices into the local descriptor list |
+| `remote_side` | `const nixlDlistH*` | Remote (or loopback) prepared descriptor list handle |
+| `remote_indices` | `const std::vector&` | Indices into the remote descriptor list |
+| `req_hndl` | `nixlXferReqH*&` | [out] Transfer request handle |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Use `backends` to limit backend selection; use `notif` to attach a notification message. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixlXferReqH* req;
+std::vector indices = {0, 1, 2};
+agent.makeXferReq(NIXL_WRITE, local_hndl, indices,
+ remote_hndl, indices, req);
+```
+
+### createXferReq
+
+Combined API that creates a transfer request directly from two descriptor lists. Internally prepares both sides and creates the transfer handle. Equivalent to calling `prepXferDlist()` for each side followed by `makeXferReq()` with all indices.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `const nixl_xfer_op_t&` | Transfer direction (`NIXL_READ` or `NIXL_WRITE`) |
+| `local_descs` | `const nixl_xfer_dlist_t&` | Local descriptor list |
+| `remote_descs` | `const nixl_xfer_dlist_t&` | Remote (or loopback) descriptor list |
+| `remote_agent` | `const std::string&` | Remote (or self) agent name |
+| `req_hndl` | `nixlXferReqH*&` | [out] Transfer request handle |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Use `backends` to limit backend selection; use `notif` to attach a notification message. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+If the same descriptors are reused across multiple transfers, prefer `prepXferDlist()` + `makeXferReq()` to avoid repeated preparation overhead. `createXferReq()` is simpler for one-off transfers.
+
+
+```cpp
+nixlXferReqH* req;
+nixl_opt_args_t args;
+args.notif = "done";
+agent.createXferReq(NIXL_WRITE, local_descs, remote_descs,
+ "remote_agent", req, &args);
+```
+
+## Transfer Operations
+
+
+For a complete workflow example, see [Quick Start -- Creating and Executing Transfers](../getting-started/quick-start#creating-and-executing-transfers).
+
+
+### estimateXferCost
+
+Estimate the cost (duration) of executing a transfer request before posting it.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_hndl` | `const nixlXferReqH*` | Transfer request handle |
+| `duration` | `std::chrono::microseconds&` | [out] Estimated transfer duration |
+| `err_margin` | `std::chrono::microseconds&` | [out] Estimated error margin |
+| `method` | `nixl_cost_t&` | [out] Method used to compute the estimate |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+std::chrono::microseconds duration, margin;
+nixl_cost_t method;
+agent.estimateXferCost(req, duration, margin, method);
+```
+
+### postXferReq
+
+Submit a transfer request, initiating the data transfer. The operation is non-blocking -- after posting, use `getXferStatus()` to poll for completion.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_hndl` | `nixlXferReqH*` | Transfer request handle from `makeXferReq()` or `createXferReq()` |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Notification message can be provided or updated per re-post. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` if transfer completed inline, `NIXL_IN_PROG` if transfer is asynchronous, or error code |
+
+
+Small transfers may complete within the `postXferReq()` call itself, returning `NIXL_SUCCESS` immediately. For larger transfers, expect `NIXL_IN_PROG` and poll with `getXferStatus()`.
+
+
+```cpp
+nixl_status_t status = agent.postXferReq(req);
+if (status == NIXL_IN_PROG) {
+ // Poll for completion
+}
+```
+
+### getXferStatus
+
+Check the status of a posted transfer request.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_hndl` | `nixlXferReqH*` | Transfer request handle after `postXferReq()` |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` when complete, `NIXL_IN_PROG` while running, or negative error code |
+
+```cpp
+nixl_status_t status;
+do {
+ status = agent.getXferStatus(req);
+} while (status == NIXL_IN_PROG);
+```
+
+### getXferTelemetry
+
+Retrieve telemetry data for a transfer request. Telemetry capture must be enabled via `nixlAgentConfig::captureTelemetry`.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_hndl` | `const nixlXferReqH*` | Transfer request handle |
+| `telemetry` | `nixl_xfer_telem_t&` | [out] Populated telemetry data |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_NO_TELEMETRY` if capture is not enabled |
+
+```cpp
+nixl_xfer_telem_t telem;
+if (agent.getXferTelemetry(req, telem) == NIXL_SUCCESS) {
+ auto us = telem.xferDuration.count();
+}
+```
+
+### queryXferBackend
+
+Query the backend chosen for a transfer request. Useful when you need to target the same backend for a notification via `genNotif()`.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_hndl` | `const nixlXferReqH*` | Transfer request handle |
+| `backend` | `nixlBackendH*&` | [out] Backend handle chosen for this transfer |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixlBackendH* xfer_backend;
+agent.queryXferBackend(req, xfer_backend);
+```
+
+### releaseXferReq
+
+Release a transfer request handle. If the transfer is still active, it will be canceled (or an error returned if the transfer cannot be aborted).
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_hndl` | `nixlXferReqH*` | Transfer request handle to release |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+agent.releaseXferReq(req);
+```
+
+### releasedDlistH
+
+Release a prepared descriptor list handle.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `dlist_hndl` | `nixlDlistH*` | Prepared descriptor list handle to release |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+The method name is `releasedDlistH` (with a trailing 'd') -- this is the canonical name from the source header.
+
+
+```cpp
+agent.releasedDlistH(local_hndl);
+agent.releasedDlistH(remote_hndl);
+```
+
+## Memory View
+
+### prepMemView (remote)
+
+Prepare a memory view handle for remote buffers. The handle can later be used for memory transfer operations.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `dlist` | `const nixl_remote_dlist_t&` | Descriptor list for the remote buffers |
+| `mvh` | `nixlMemViewH&` | [out] Memory view handle |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `backends` is set, limits backend selection. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_remote_dlist_t remote_descs(VRAM_SEG);
+remote_descs.addDesc(nixlRemoteDesc(addr, size, 0, "remote_agent"));
+nixlMemViewH mvh;
+agent.prepMemView(remote_descs, mvh);
+```
+
+### prepMemView (local)
+
+Prepare a memory view handle for local buffers.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `dlist` | `const nixl_local_dlist_t&` | Descriptor list for the local buffers |
+| `mvh` | `nixlMemViewH&` | [out] Memory view handle |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_local_dlist_t local_descs(DRAM_SEG);
+local_descs.addDesc(nixlBasicDesc(reinterpret_cast(buf), 256, 0));
+nixlMemViewH mvh;
+agent.prepMemView(local_descs, mvh);
+```
+
+### releaseMemView
+
+Release a memory view handle.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `mvh` | `nixlMemViewH` | Memory view handle to release |
+| **Returns** | `void` | |
+
+```cpp
+agent.releaseMemView(mvh);
+```
+
+## Notification Handling
+
+### getNotifs
+
+Retrieve pending notifications from remote agents. Entries are added to the input map (which may already contain entries) and released from the agent's internal queue.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `notif_map` | `nixl_notifs_t&` | Map from agent name to list of notification blobs. New entries are appended. |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `backends` is set, only retrieves notifications from those backends. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+Notifications are consumed by this call -- they are removed from the agent's internal queue after being added to `notif_map`.
+
+
+```cpp
+nixl_notifs_t notifs;
+agent.getNotifs(notifs);
+for (auto& [agent_name, msgs] : notifs) {
+ for (auto& msg : msgs) {
+ // Process notification from agent_name
+ }
+}
+```
+
+### genNotif
+
+Generate and send a standalone notification to a remote agent. The remote agent's metadata must be available before calling this method.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the remote agent |
+| `msg` | `const nixl_blob_t&` | Notification message to send |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Use `backends` to specify which backend to use for sending. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+Standalone notifications sent via `genNotif()` are received by the remote agent through `getNotifs()`, alongside transfer-bound notifications. Use `queryXferBackend()` if you want to send the notification over the same backend as a specific transfer.
+
+
+```cpp
+agent.genNotif("remote_agent", "control_message");
+```
+
+## Metadata -- Side Channel
+
+
+For a complete workflow example, see [Quick Start -- Metadata Exchange](../getting-started/quick-start#metadata-exchange).
+
+
+Side-channel metadata methods use a serialized blob that the application transports between agents using its own mechanism (shared memory, message queue, custom RPC, etc.).
+
+### getLocalMD
+
+Get this agent's full metadata as a serialized blob. The blob can be transported to other agents and loaded with `loadRemoteMD()`.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `str` | `nixl_blob_t&` | [out] Serialized metadata blob |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_blob_t metadata;
+agent.getLocalMD(metadata);
+// Transport metadata to remote agent via your own mechanism
+```
+
+### getLocalPartialMD
+
+Get a partial metadata blob containing only the specified descriptors and optionally backend connection information.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_reg_dlist_t&` | Descriptor list to include. If empty, only backend connection info is included. |
+| `str` | `nixl_blob_t&` | [out] Serialized partial metadata blob |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. Use `includeConnInfo` to add connection info for backends supporting the memory type. Use `backends` to filter which backends are included. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+When `descs` is empty, only backends' connection info is included in the metadata, regardless of the value of `extra_params->includeConnInfo`.
+
+
+```cpp
+nixl_blob_t partial_md;
+agent.getLocalPartialMD(descs, partial_md);
+```
+
+### loadRemoteMD
+
+Load and unpack a remote agent's metadata blob. After loading, the local agent can initiate transfers toward the remote agent.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_metadata` | `const nixl_blob_t&` | Serialized metadata blob received from the remote agent |
+| `agent_name` | `std::string&` | [out] Agent name extracted from the loaded metadata |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+std::string remote_name;
+agent.loadRemoteMD(received_metadata, remote_name);
+// Now can transfer to/from remote_name
+```
+
+### invalidateRemoteMD
+
+Invalidate the cached metadata for a remote agent. This disconnects from the remote agent and prevents further transfers to it.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the remote agent to invalidate |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+agent.invalidateRemoteMD("remote_agent");
+```
+
+## Metadata -- Direct Channel
+
+
+For a complete workflow example, see [Quick Start -- Metadata Exchange](../getting-started/quick-start#metadata-exchange).
+
+
+Direct-channel methods send and fetch metadata over peer-to-peer TCP sockets or an etcd metadata server, without the application needing to transport blobs manually.
+
+### sendLocalMD
+
+Send this agent's full metadata to a remote peer or the etcd metadata server.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `ipAddr` is set, sends to the specified peer (peer-to-peer mode). If `ipAddr` is not set, sends to the etcd metadata server. `port` defaults to `default_comm_port` (8888). Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+// Send to ETCD (no IP specified)
+agent.sendLocalMD();
+
+// Send to a specific peer
+nixl_opt_args_t args;
+args.ipAddr = "10.0.0.2";
+args.port = 5555;
+agent.sendLocalMD(&args);
+```
+
+### sendLocalPartialMD
+
+Send partial metadata to a remote peer or the etcd metadata server.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_reg_dlist_t&` | Descriptor list to include. If empty, only backend connection info is sent. |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `ipAddr` is set, sends to a single peer. If `ipAddr` is not set, sends to the etcd metadata server. `metadataLabel` is required when sending to etcd and ignored for peer-to-peer. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+
+When sending partial metadata to etcd, `extra_params->metadataLabel` is required to scope the metadata under a specific key label. When sending to a peer, the label is ignored.
+
+
+```cpp
+nixl_opt_args_t args;
+args.metadataLabel = "gpu_buffers";
+agent.sendLocalPartialMD(descs, &args);
+```
+
+### fetchRemoteMD
+
+Fetch a remote agent's metadata from a peer or the etcd metadata server, then unpack it internally.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_name` | `const std::string` | Name of the remote agent to fetch metadata for |
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `ipAddr` is set, fetches from the specified peer (only full metadata). If `ipAddr` is not set, fetches from etcd. `metadataLabel` can specify partial metadata to fetch from etcd. `port` defaults to `default_comm_port`. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+// Fetch from ETCD
+agent.fetchRemoteMD("remote_agent");
+
+// Fetch from a specific peer
+nixl_opt_args_t args;
+args.ipAddr = "10.0.0.2";
+agent.fetchRemoteMD("remote_agent", &args);
+```
+
+### invalidateLocalMD
+
+Invalidate this agent's metadata on a remote peer or the etcd metadata server.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `extra_params` | `const nixl_opt_args_t*` | Optional. If `ipAddr` is set, invalidates metadata on the specified peer. If `ipAddr` is not set, invalidates all of this agent's labels from the etcd server. Default: `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+// Invalidate from ETCD (removes all labels)
+agent.invalidateLocalMD();
+```
+
+### checkRemoteMD
+
+Check if metadata is available for a remote agent. Optionally verify that specific descriptors are included.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_name` | `const std::string` | Name of the remote agent to check |
+| `descs` | `const nixl_xfer_dlist_t&` | Descriptor list to check for. Pass an empty list to check for any metadata from the agent. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` if metadata is available, `NIXL_ERR_NOT_FOUND` if not found |
+
+```cpp
+nixl_xfer_dlist_t empty(DRAM_SEG);
+if (agent.checkRemoteMD("remote_agent", empty) == NIXL_SUCCESS) {
+ // Remote metadata is available
+}
+```
diff --git a/docs/api-reference/device-api.md b/docs/api-reference/device-api.md
new file mode 100644
index 0000000000..b9171963b4
--- /dev/null
+++ b/docs/api-reference/device-api.md
@@ -0,0 +1,199 @@
+---
+title: Device API Reference
+description: CUDA device-side API for GPU-initiated data transfers in NIXL.
+---
+
+This is the CUDA device-side API reference for NIXL. The Device API enables GPU kernels to initiate and monitor data transfers directly from device code, without returning to the host. For the host-side C++ API, see the [C++ API Reference](./cpp-api).
+
+To use the NIXL Device API, include the device header:
+
+```cpp
+#include "nixl_device.cuh"
+```
+
+## Types, Enums and Defines
+
+### nixl_gpu_level_t
+
+An enumeration of GPU execution levels for transfer requests.
+
+| Value | Description |
+|-------|-------------|
+| `THREAD` | Thread-level transfer request (default) |
+| `WARP` | Warp-level transfer request |
+| `BLOCK` | Block-level transfer request |
+| `GRID` | Grid-level transfer request |
+
+The level template parameter controls the granularity of the transfer operation. Higher levels (WARP, BLOCK, GRID) can batch operations for better throughput but require all threads in the group to participate.
+
+```cpp
+// Thread-level (default)
+nixlPut(src, dst, size, channel_id);
+
+// Warp-level
+nixlPut(src, dst, size, channel_id);
+```
+
+### nixlGpuXferStatusH
+
+Transfer status handle for tracking asynchronous operations.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `device_request` | `ucp_device_request_t` | Internal UCX device request handle |
+
+Pass a pointer to a `nixlGpuXferStatusH` to transfer functions, then query completion with `nixlGpuGetXferStatus`.
+
+### nixlMemViewElem
+
+Memory view element referencing a buffer in a prepared memory view.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `mvh` | `nixlMemViewH` | Memory view handle (from host-side `nixlAgent::prepMemView`) |
+| `index` | `size_t` | Index in the memory view |
+| `offset` | `size_t` | Byte offset within the buffer |
+
+Memory views must be prepared on the host using `nixlAgent::prepMemView` before device code can access them. See the [C++ API Reference](./cpp-api) for details.
+
+## Flags
+
+### nixl_gpu_flags
+
+Namespace containing transfer flag constants and utilities.
+
+| Constant | Type | Value | Description |
+|----------|------|-------|-------------|
+| `defer` | `uint64_t` | `1` | Defer the transfer (do not send immediately). When set, the transfer is batched until the next non-deferred operation or explicit flush. |
+
+#### to_ucp_flags
+
+Device-side helper function that converts NIXL flags to internal UCP flags.
+
+```cpp
+__device__ inline uint64_t to_ucp_flags(uint64_t nixl_flags) noexcept;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `nixl_flags` | `uint64_t` | NIXL transfer flags (combination of `nixl_gpu_flags` constants) |
+
+Returns `uint64_t` -- Converted UCP flags for internal use.
+
+
+This is an internal helper. Application code typically passes flags directly to `nixlPut` or `nixlAtomicAdd`.
+
+
+## Functions
+
+### nixlPut
+
+Post a single-region memory transfer from local to remote GPU.
+
+```cpp
+template
+__device__ nixl_status_t
+nixlPut(const nixlMemViewElem &src,
+ const nixlMemViewElem &dst,
+ size_t size,
+ unsigned channel_id = 0,
+ uint64_t flags = 0,
+ nixlGpuXferStatusH *xfer_status = nullptr);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `src` | `const nixlMemViewElem &` | Source memory view element (local) |
+| `dst` | `const nixlMemViewElem &` | Destination memory view element (remote) |
+| `size` | `size_t` | Number of bytes to transfer |
+| `channel_id` | `unsigned` | Channel ID for the transfer (default: `0`) |
+| `flags` | `uint64_t` | Transfer flags from `nixl_gpu_flags` (default: `0`) |
+| `xfer_status` | `nixlGpuXferStatusH *` | Optional status handle for tracking completion (default: `nullptr`) |
+
+**Template parameter:** `level` -- GPU execution level (default: `nixl_gpu_level_t::THREAD`).
+
+**Returns:**
+
+| Value | Description |
+|-------|-------------|
+| `NIXL_IN_PROG` | Transfer posted successfully |
+| `NIXL_ERR_BACKEND` | An error occurred in the UCX backend |
+
+### nixlAtomicAdd
+
+Atomic add to remote GPU memory.
+
+```cpp
+template
+__device__ nixl_status_t
+nixlAtomicAdd(uint64_t value,
+ const nixlMemViewElem &counter,
+ unsigned channel_id = 0,
+ uint64_t flags = 0,
+ nixlGpuXferStatusH *xfer_status = nullptr);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `value` | `uint64_t` | Value to add to the remote counter |
+| `counter` | `const nixlMemViewElem &` | Counter memory view element (remote) |
+| `channel_id` | `unsigned` | Channel ID for the transfer (default: `0`) |
+| `flags` | `uint64_t` | Transfer flags from `nixl_gpu_flags` (default: `0`) |
+| `xfer_status` | `nixlGpuXferStatusH *` | Optional status handle for tracking completion (default: `nullptr`) |
+
+**Template parameter:** `level` -- GPU execution level (default: `nixl_gpu_level_t::THREAD`).
+
+**Returns:**
+
+| Value | Description |
+|-------|-------------|
+| `NIXL_IN_PROG` | Atomic add posted successfully |
+| `NIXL_ERR_BACKEND` | An error occurred in the UCX backend |
+
+
+The atomic increment is visible only after previous writes complete.
+
+
+### nixlGetPtr
+
+Get a local pointer to remote memory.
+
+```cpp
+__device__ inline void *
+nixlGetPtr(nixlMemViewH mvh, size_t index);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `mvh` | `nixlMemViewH` | Memory view handle (remote buffers) |
+| `index` | `size_t` | Index in the memory view |
+
+Returns `void *` -- Pointer to the mapped memory, or `nullptr` if not available.
+
+
+The memory view must be prepared on the host using `nixlAgent::prepMemView`.
+
+
+### nixlGpuGetXferStatus
+
+Get the status of a transfer request.
+
+```cpp
+template
+__device__ nixl_status_t
+nixlGpuGetXferStatus(nixlGpuXferStatusH &xfer_status);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `xfer_status` | `nixlGpuXferStatusH &` | Status handle passed to a previous transfer function |
+
+**Template parameter:** `level` -- GPU execution level (default: `nixl_gpu_level_t::THREAD`).
+
+**Returns:**
+
+| Value | Description |
+|-------|-------------|
+| `NIXL_SUCCESS` | The request has completed, no more operations are in progress |
+| `NIXL_IN_PROG` | One or more operations in the request have not completed |
+| `NIXL_ERR_BACKEND` | An error occurred in the UCX backend |
diff --git a/docs/api-reference/northbound-api.md b/docs/api-reference/northbound-api.md
new file mode 100644
index 0000000000..09f1f70497
--- /dev/null
+++ b/docs/api-reference/northbound-api.md
@@ -0,0 +1,145 @@
+---
+title: Northbound API Semantics
+description: Binding-independent rules for descriptor matching, registration, transfers, and cleanup.
+---
+
+This page covers behavior shared by the C++, Python, and Rust bindings. For the
+overall data-transfer lifecycle, see [Architecture](../getting-started/architecture).
+For a working end-to-end sequence, see [Quick Start](../getting-started/quick-start).
+Exact type and method names are documented in the
+[C++](./cpp-api), [Python](./python-api), and [Rust](./rust-api) API references.
+
+## Descriptor matching
+
+NIXL identifies a registration by this tuple:
+
+```text
+(memory type, device ID, address or offset, length)
+```
+
+The metadata attached to a registration is not part of its identity. Registration
+descriptors carry metadata because a backend may need configuration such as a file
+path or object key. Transfer descriptors contain only the address or offset, length,
+and device ID.
+
+A descriptor list has one memory type. The client must set that type correctly for
+every descriptor in the list.
+
+The client may select a subrange of a registered region in a transfer descriptor.
+The client must use the same device ID as the registration, and the requested
+address range must be fully covered by it.
+
+For file, object, and block segments, the client may use a zero-length registration
+to describe an unbounded range starting at the supplied offset. For DRAM and VRAM,
+the client must provide the actual region length.
+
+## Registration behavior
+
+Registrations belong to one agent. The client must register each local region with
+the agent that owns it. An agent cannot use another local agent's registration as
+its own.
+
+A descriptor list is registered transactionally within each backend. If one
+descriptor fails, that backend rolls back the descriptors from the same list that it
+already registered.
+
+Registration across multiple backends is best-effort:
+
+- The call succeeds when at least one selected or compatible backend registers the
+ complete descriptor list.
+- A successful call does not mean that every compatible backend accepted the list.
+- If all backends fail, the call returns an error and the per-backend attempts are
+ rolled back.
+
+The client must not assume that registration validates every external resource or
+reports every error immediately. For example, an object-store registration can
+succeed even when its object key does not exist; the missing key is reported only
+when a transfer tries to access it. The client must assume the same deferred-error
+behavior for file path mode, even when a backend currently performs some checks
+during registration, and must handle these failures when the transfer is executed.
+
+Duplicate-registration behavior is backend-specific. The client must not assume
+that the same descriptor can be registered more than once. If a backend accepts a
+duplicate, the client must balance every successful registration with a separate
+deregistration.
+
+## Device IDs
+
+A device ID is a memory-type-specific lookup key, not always a hardware device
+number.
+
+| Memory type | Client requirement |
+|---|---|
+| DRAM | The client normally uses `0`; compatible backends may ignore the value. |
+| VRAM | The client must use the CUDA ordinal of the GPU that owns the allocation. |
+| Block | The client must use the resource identifier expected by the backend. |
+| File, fd mode | The client must provide an open file descriptor as the device-ID value. |
+| File, path mode | The client must assign a file identity that is unique among active path-mode registrations. |
+| Object | The client must assign a lookup identity that is unique among active object registrations. |
+
+The client must not reuse a device ID while the resource it identifies remains
+registered. File path-mode backends reject a duplicate active ID. Current object
+backends keep one object-key mapping per ID, so reuse can replace the existing
+mapping.
+
+### File path mode
+
+File path mode lets the backend open and close a file on behalf of the caller. A
+registration is recognized as path mode when its metadata matches:
+
+```text
+[,...]:
+```
+
+Supported access values are `ro` and `rw`. Supported flags are `direct`,
+`sync`, `noatime`, and `create`.
+
+Examples:
+
+```text
+ro:/var/cache/model.bin
+rw,direct:/var/cache/model.bin
+rw,create:/tmp/output.bin
+```
+
+When the client requests path mode, the metadata must match this grammar exactly.
+A non-matching string is handled as fd mode, in which case the client must provide
+an already-open file descriptor as the device ID.
+
+The client must assign a unique device ID to every active path-mode registration.
+Path-mode backends reject an ID that is already in use. The client may reuse the ID
+after the corresponding registration has been deregistered.
+
+## Transfer behavior
+
+The client must ensure that every local and remote transfer descriptor resolves to
+a registered region with the same memory type and device ID. The requested address
+range must fit inside that registration.
+
+The client may post a transfer request again only after it reaches a terminal state.
+The client must not post the same request while it is active; the binding reports
+this through its normal error mechanism.
+
+After a transfer error, the client must not assume that the destination is unchanged
+or contains a valid partial result. NIXL does not roll back writes that already
+completed. The client must treat the destination as invalid and repeat the complete
+logical transfer from a known state.
+
+The client must not release or reuse transfer buffers while the request is still in
+progress. It must wait until the request reports success or an error.
+
+## Deregistration behavior
+
+Deregistration matches the same identity tuple used for registration. The client
+must use the same memory type, device ID, address or offset, and length that it used
+for registration. Registration metadata is ignored during this lookup, so the
+client does not need to repeat it.
+
+Within one backend, NIXL checks that the complete descriptor list is present before
+removing any entry from that list. Across multiple backends, cleanup is best-effort;
+an error from one backend can leave other backends already deregistered.
+
+Each successful registration consumes one registration lifetime. If a backend
+accepted the same identity more than once, the client must deregister it the same
+number of times. After all matching registrations are removed, the client may reuse
+caller-selected device IDs.
diff --git a/docs/api-reference/python-api.md b/docs/api-reference/python-api.md
new file mode 100644
index 0000000000..85529f24b8
--- /dev/null
+++ b/docs/api-reference/python-api.md
@@ -0,0 +1,967 @@
+---
+title: Python API Reference
+description: Complete Python API reference for NIXL data transfers.
+---
+
+This is the Python API reference for the NVIDIA Inference Xfer Library (NIXL). Python provides a high-level, Pythonic interface to NIXL via the `nixl_agent` class. For the C++ API, see the [C++ API Reference](./cpp-api). For Rust bindings, see the [Rust API Reference](./rust-api).
+
+Behavior shared by all bindings is documented in [Northbound API Semantics](./northbound-api).
+
+Key Python-specific features:
+
+- **Auto-backend initialization**: Backends specified in the config are created automatically during agent construction
+- **String status returns**: Transfer methods return `"DONE"`, `"PROC"`, or `"ERR"` instead of enum values
+- **Tensor/numpy auto-conversion**: Methods accept PyTorch tensors, numpy arrays, or raw tuples for memory descriptors
+- **Pythonic method names**: `register_memory()` instead of `registerMem()`, `transfer()` instead of `postXferReq()`
+
+To use the NIXL Python API:
+
+```python
+from nixl._api import nixl_agent
+```
+
+## Types, Enums and Defines
+
+This section documents the configuration class, handle types, and Python-specific conventions used throughout the NIXL Python API. Subsections are organized to mirror the C++ API structure.
+
+### Configuration
+
+### nixl_agent_config
+
+Configuration dataclass for creating a Transfer Agent. Pass an instance to the `nixl_agent` constructor to customize agent behavior.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `enable_prog_thread` | `bool` | `True` | Enable the progress thread for asynchronous transfer completion |
+| `enable_listen_thread` | `bool` | `False` | Enable the listener thread for direct metadata communication |
+| `listen_port` | `int` | `0` | Port for the listener thread (0 for auto-assign) |
+| `capture_telemetry` | `bool` | `False` | Enable telemetry capture for transfer performance metrics |
+| `num_threads` | `int` | `0` | Number of threads for supported multi-threaded backends |
+| `backends` | `list[str]` | `["UCX"]` | Backends to auto-create at agent initialization |
+
+```python
+from nixl._api import nixl_agent, nixl_agent_config
+
+# Default config -- UCX backend auto-created
+config = nixl_agent_config()
+
+# Custom config with multiple backends and telemetry
+config = nixl_agent_config(
+ backends=["UCX", "GDS"],
+ capture_telemetry=True,
+ num_threads=4
+)
+```
+
+
+The `backends` field triggers auto-creation of the specified backends during agent construction. This is a Python-only convenience -- C++ and Rust require explicit `createBackend()` calls. You can still create additional backends later with `create_backend()`.
+
+
+### nixl_agent
+
+The main agent class. All NIXL operations are performed through methods on this class. See the functional group sections below for the complete method reference.
+
+### Handle Types
+
+### nixl_prepped_dlist_handle
+
+Opaque handle returned by `prep_xfer_dlist()`. Represents a prepared transfer descriptor list that can be reused across multiple transfers.
+
+| Method | Description |
+|--------|-------------|
+| `release()` | Explicitly free resources associated with this handle |
+
+The handle performs best-effort cleanup through `__del__` if `release()` is not called. If finalization fails, an error is logged.
+
+```python
+handle = agent.prep_xfer_dlist("remote_agent", descs)
+# ... use handle ...
+handle.release() # Explicit cleanup
+```
+
+### nixl_xfer_handle
+
+Opaque handle returned by `make_prepped_xfer()` and `initialize_xfer()`. Represents an active or prepared transfer operation.
+
+| Method | Description |
+|--------|-------------|
+| `release()` | Explicitly free resources. If the transfer is active, NIXL will attempt to cancel it |
+
+If `release()` is not called, `__del__` attempts cleanup. Failed finalization queues the handle in an internal leaked-handles list for re-release during agent destruction.
+
+```python
+xfer_handle = agent.make_prepped_xfer("WRITE", local_h, [0], remote_h, [0])
+agent.transfer(xfer_handle)
+# ... wait for completion ...
+xfer_handle.release()
+```
+
+
+Always call `release()` on transfer handles when done. Relying on garbage collection may delay cleanup and leak resources until agent destruction.
+
+
+### nixl_backend_handle
+
+Type alias for `int`. Returned by `create_backend()` and used internally. You typically interact with backends by their string names in the Python API.
+
+### Memory Type and Operation Strings
+
+Python uses string names instead of C++ enumerations for memory types and transfer operations.
+
+**Memory types:**
+
+| String | C++ Equivalent | Description |
+|--------|---------------|-------------|
+| `"DRAM"` | `DRAM_SEG` | Standard host memory (CPU DRAM) |
+| `"VRAM"` | `VRAM_SEG` | GPU high-bandwidth memory |
+| `"BLOCK"` | `BLK_SEG` | Block-level storage devices |
+| `"OBJ"` | `OBJ_SEG` | Distributed object stores |
+| `"FILE"` | `FILE_SEG` | Local and remote file systems |
+
+The Python API also accepts device-style aliases `"cpu"` (maps to `DRAM`) and `"cuda"` (maps to `VRAM`) for convenience.
+
+**Transfer operations:**
+
+| String | C++ Equivalent | Description |
+|--------|---------------|-------------|
+| `"READ"` | `NIXL_READ` | Read data from the remote side into local buffers |
+| `"WRITE"` | `NIXL_WRITE` | Write data from local buffers to the remote side |
+
+**Transfer status strings:**
+
+| String | C++ Equivalent | Description |
+|--------|---------------|-------------|
+| `"DONE"` | `NIXL_SUCCESS` | Transfer completed successfully |
+| `"PROC"` | `NIXL_IN_PROG` | Transfer is in progress |
+| `"ERR"` | `NIXL_ERR_*` | Transfer encountered an error |
+
+## Initialization and Configuration
+
+
+For a complete walkthrough of agent setup and the overall transfer workflow, see [Quick Start -- Agent Initialization](../getting-started/quick-start#agent-initialization).
+
+
+### \_\_init\_\_
+
+Create a new Transfer Agent. The agent manages backends, memory registrations, metadata, and transfer operations for a single process.
+
+**C++ equivalent:** [`nixlAgent`](./cpp-api#nixlagent)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `agent_name` | `str` | *(required)* | Unique name for this agent |
+| `nixl_conf` | `nixl_agent_config` | `None` | Configuration object. If `None`, uses default config |
+| `instantiate_all` | `bool` | `False` | If `True`, auto-create all available plug-in backends |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixl_agent` | New agent instance |
+
+```python
+from nixl._api import nixl_agent, nixl_agent_config
+
+# Default agent -- auto-creates UCX backend
+agent = nixl_agent("my_agent")
+
+# Agent with custom configuration
+config = nixl_agent_config(backends=["UCX", "GDS"], capture_telemetry=True)
+agent = nixl_agent("my_agent", config)
+```
+
+
+Unlike C++, Python auto-creates backends listed in the config during construction. The constructor queries all available plug-ins, caches their parameters and memory types, and then initializes the specified backends. If a requested backend plug-in is not available, a warning is logged and that backend is skipped.
+
+
+### get_plugin_list
+
+Get the list of all available backend plug-ins discovered at agent initialization.
+
+**C++ equivalent:** [`getAvailPlugins`](./cpp-api#getavailplugins)
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `list[str]` | List of plug-in names (e.g., `["UCX", "GDS", "POSIX"]`) |
+
+```python
+plugins = agent.get_plugin_list()
+# ['UCX', 'GDS', 'POSIX', ...]
+```
+
+
+The plug-in list is cached at agent initialization. This call returns the cached list without re-querying the system.
+
+
+### get_plugin_mem_types
+
+Get the memory types supported by a specific plug-in.
+
+**C++ equivalent:** [`getPluginParams`](./cpp-api#getpluginparams) (memory types portion)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `backend` | `str` | Name of the plug-in |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `list[str]` | Supported memory type strings (e.g., `["DRAM", "VRAM"]`) |
+
+```python
+mem_types = agent.get_plugin_mem_types("UCX")
+# ['DRAM', 'VRAM']
+```
+
+### get_plugin_params
+
+Get the initialization parameters of a plug-in. Returns a dictionary where keys are parameter names and values are their default values.
+
+**C++ equivalent:** [`getPluginParams`](./cpp-api#getpluginparams) (parameters portion)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `backend` | `str` | Name of the plug-in |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `dict[str, str]` | Parameter name to default value mapping |
+
+```python
+params = agent.get_plugin_params("UCX")
+# {'num_threads': '0', ...}
+```
+
+## Backend Management
+
+
+For a walkthrough of backend creation and selection, see [Quick Start -- Backend Creation](../getting-started/quick-start#backend-creation).
+
+
+### get_backend_mem_types
+
+Get the memory types supported by an initialized backend. After initialization, supported memory types may differ from the plug-in defaults.
+
+**C++ equivalent:** [`getBackendParams`](./cpp-api#getbackendparams) (memory types portion)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `backend` | `str` | Name of the backend |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `list[str]` | Supported memory type strings |
+
+```python
+mem_types = agent.get_backend_mem_types("UCX")
+```
+
+### get_backend_params
+
+Get the parameters of an initialized backend. Available parameters may differ from the plug-in defaults after initialization.
+
+**C++ equivalent:** [`getBackendParams`](./cpp-api#getbackendparams) (parameters portion)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `backend` | `str` | Name of the backend |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `dict[str, str]` | Parameter name to value mapping |
+
+```python
+params = agent.get_backend_params("UCX")
+```
+
+### create_backend
+
+Initialize a backend with the specified parameters. This is only needed for backends not listed in the `backends` config field at agent creation time.
+
+**C++ equivalent:** [`createBackend`](./cpp-api#createbackend)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `backend` | `str` | *(required)* | Name of the backend plug-in to initialize |
+| `initParams` | `dict[str, str]` | `{}` | Initialization parameters |
+
+```python
+# Create a GDS backend with custom threads
+agent.create_backend("GDS", {"num_threads": "4"})
+```
+
+
+The Python API automatically caches the backend's parameters and supported memory types after creation. You can query them later with `get_backend_params()` and `get_backend_mem_types()`.
+
+
+## Memory Registration
+
+
+For a complete walkthrough of memory registration in the transfer workflow, see [Quick Start -- Memory Registration](../getting-started/quick-start#memory-registration).
+
+
+### register_memory
+
+Register memory regions with one or more backends. Accepts multiple input formats including PyTorch tensors for convenience.
+
+**C++ equivalent:** [`registerMem`](./cpp-api#registermem)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `reg_list` | *see below* | *(required)* | Memory regions to register |
+| `mem_type` | `str` | `None` | Memory type string (required for tuples and numpy) |
+| `backends` | `list[str]` | `[]` | Backend names to register with; empty means all compatible backends |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixlRegDList` | Registration descriptor list for use with `deregister_memory()` |
+
+**Accepted input types for `reg_list`:**
+
+| Input Type | `mem_type` Required? | Description |
+|------------|---------------------|-------------|
+| `torch.Tensor` | No | Single contiguous tensor; memory type auto-detected from device |
+| `list[torch.Tensor]` | No | List of contiguous tensors on the same device |
+| `numpy.ndarray` (Nx3) | Yes | Each row is `[address, size, device_id]` |
+| `list[tuple]` | Yes | List of 4-tuples `(address, size, device_id, meta_info)` |
+| `nixlRegDList` | No | Passed through directly |
+
+```python
+import torch
+
+# Register a GPU tensor (simplest form)
+tensor = torch.zeros(1024, device="cuda:0")
+reg_descs = agent.register_memory(tensor)
+
+# Register multiple tensors
+tensors = [torch.zeros(512, device="cuda:0") for _ in range(4)]
+reg_descs = agent.register_memory(tensors)
+
+# Register with raw tuples and specific backend
+descs = [(addr, size, 0, "")]
+reg_descs = agent.register_memory(descs, mem_type="DRAM", backends=["UCX"])
+```
+
+
+You can pass tensors directly -- there is no need to manually extract `data_ptr()` and create tuple lists. The Python API handles tensor-to-descriptor conversion internally via `get_reg_descs()`.
+
+
+### deregister_memory
+
+Deregister memory regions from backends.
+
+**C++ equivalent:** [`deregisterMem`](./cpp-api#deregistermem)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `dereg_list` | `nixlRegDList` | *(required)* | Descriptor list from `register_memory()` or `get_reg_descs()` |
+| `backends` | `list[str]` | `[]` | Backend names to deregister from; empty means all backends that have these regions |
+
+```python
+agent.deregister_memory(reg_descs)
+```
+
+### query_memory
+
+Query information about registered memory or storage for a specific backend.
+
+**C++ equivalent:** [`queryMem`](./cpp-api#querymem)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `reg_list` | *various* | *(required)* | Memory regions, tensors, or `nixlRegDList` to query |
+| `backend` | `str` | *(required)* | Backend name to query |
+| `mem_type` | `str` | `None` | Memory type (required for tuples/numpy) |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `list[Optional[dict[str, str]]]` | Query results; `None` for entries not found, otherwise a dict with backend-specific info |
+
+```python
+results = agent.query_memory(reg_descs, "UCX")
+```
+
+### make_connection
+
+Proactively establish a connection with a remote agent to reduce first-transfer latency. This is optional -- NIXL establishes connections on demand if not called.
+
+**C++ equivalent:** [`makeConnection`](./cpp-api#makeconnection)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `remote_agent` | `str` | *(required)* | Name of the remote agent |
+| `backends` | `list[str]` | `[]` | Limit connections to specific backends; empty means all applicable backends |
+
+```python
+agent.make_connection("remote_agent")
+```
+
+### Convenience Methods: Descriptor Conversion
+
+These Python-only helper methods convert various input formats into NIXL descriptor lists. They are called internally by `register_memory()`, `prep_xfer_dlist()`, and `initialize_xfer()`, but can also be used directly for advanced workflows.
+
+#### get_reg_descs
+
+Convert various input types into a registration descriptor list.
+
+**C++ equivalent:** None (Python-only convenience)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `descs` | *various* | *(required)* | Memory descriptors in any supported format |
+| `mem_type` | `str` | `None` | Memory type (required for tuples and numpy inputs) |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixlRegDList` | Registration descriptor list |
+
+**Accepted input types:**
+
+| Input Type | `mem_type` Required? | Description |
+|------------|---------------------|-------------|
+| `torch.Tensor` | No | Single contiguous tensor |
+| `list[torch.Tensor]` | No | List of contiguous tensors on the same device |
+| `numpy.ndarray` (Nx3) | Yes | Each row is `[address, size, device_id]`; empty meta info |
+| `list[tuple]` | Yes | List of 4-tuples `(address, size, device_id, meta_info)` |
+| `nixlRegDList` | No | Passed through directly |
+
+```python
+import torch
+import numpy as np
+
+# From a single tensor
+reg_descs = agent.get_reg_descs(torch.zeros(1024, device="cuda:0"))
+
+# From a tensor list
+reg_descs = agent.get_reg_descs([t1, t2], "VRAM")
+
+# From a numpy array
+reg_descs = agent.get_reg_descs(np.array([[addr, size, dev]]), "DRAM")
+
+# From raw tuples
+reg_descs = agent.get_reg_descs([(addr, size, dev, "meta")], "DRAM")
+```
+
+#### get_xfer_descs
+
+Convert various input types into a transfer descriptor list.
+
+**C++ equivalent:** None (Python-only convenience)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `descs` | *various* | *(required)* | Transfer descriptors in any supported format |
+| `mem_type` | `str` | `None` | Memory type (required for tuples and numpy inputs) |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixlXferDList` | Transfer descriptor list |
+
+**Accepted input types:**
+
+| Input Type | `mem_type` Required? | Description |
+|------------|---------------------|-------------|
+| `torch.Tensor` | No | Single contiguous tensor |
+| `list[torch.Tensor]` | No | List of contiguous tensors on the same device |
+| `numpy.ndarray` (Nx3) | Yes | Each row is `[address, size, device_id]` |
+| `list[tuple]` | Yes | List of 3-tuples `(address, size, device_id)` |
+| `nixlXferDList` | No | Passed through directly |
+
+```python
+# From a tensor
+xfer_descs = agent.get_xfer_descs(torch.zeros(1024, device="cuda:0"))
+
+# From raw tuples
+xfer_descs = agent.get_xfer_descs([(addr, size, 0)], "VRAM")
+```
+
+
+Transfer descriptors use 3-tuples `(address, size, device_id)` while registration descriptors use 4-tuples `(address, size, device_id, meta_info)`. The extra `meta_info` field in registration descriptors carries opaque metadata (e.g., file paths for FILE_SEG).
+
+
+## Transfer Preparation
+
+
+For a walkthrough of the full transfer lifecycle including preparation, posting, and status checking, see [Quick Start -- Creating and Executing Transfers](../getting-started/quick-start#creating-and-executing-transfers).
+
+
+### prep_xfer_dlist
+
+Prepare a transfer descriptor list for efficient reuse across multiple transfers. Both the local and remote sides of a transfer must be prepared before creating a transfer request.
+
+**C++ equivalent:** [`prepXferDlist`](./cpp-api#prepxferdlist-4-parameter)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `agent_name` | `str` | *(required)* | `"NIXL_INIT_AGENT"` or `""` for local descriptors; remote agent name for remote descriptors; local agent name for loopback |
+| `xfer_list` | *various* | *(required)* | Transfer descriptors (tensors, tuples, numpy, or `nixlXferDList`) |
+| `mem_type` | `str` | `None` | Memory type (required for tuples/numpy) |
+| `backends` | `list[str]` | `[]` | Limit which backends are used during preparation |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixl_prepped_dlist_handle` | Opaque handle to the prepared descriptor list |
+
+```python
+# Prepare local descriptors
+local_handle = agent.prep_xfer_dlist("NIXL_INIT_AGENT", local_tensors)
+
+# Prepare remote descriptors
+remote_handle = agent.prep_xfer_dlist("remote_agent", remote_descs, "VRAM")
+```
+
+
+Preparation succeeds if at least one backend can handle all elements in the descriptor list. The Python API internally detects whether the call is local (when `agent_name` is `"NIXL_INIT_AGENT"` or `""`) and dispatches to the appropriate C++ overload.
+
+
+### estimate_xfer_cost
+
+Estimate the cost of a transfer operation.
+
+**C++ equivalent:** [`estimateXferCost`](./cpp-api#estimatexfercost)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req_handle` | `nixl_xfer_handle` | Handle to the transfer operation |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `tuple[int, int, str]` | `(duration, error_margin, method)` -- times are in microseconds, `method` is `"ANALYTICAL_BACKEND"` or `"UNKNOWN"` |
+
+```python
+duration, err_margin, method = agent.estimate_xfer_cost(xfer_handle)
+```
+
+### make_prepped_xfer
+
+Create a transfer request from prepared descriptor list handles. This is the recommended approach when the same descriptors are used in multiple transfers.
+
+**C++ equivalent:** [`makeXferReq`](./cpp-api#makexferreq)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `operation` | `str` | *(required)* | `"WRITE"` or `"READ"` |
+| `local_xfer_side` | `nixl_prepped_dlist_handle` | *(required)* | Local descriptor handle from `prep_xfer_dlist()` |
+| `local_indices` | `list[int]` or `numpy.ndarray` | *(required)* | Indices selecting local descriptors |
+| `remote_xfer_side` | `nixl_prepped_dlist_handle` | *(required)* | Remote descriptor handle from `prep_xfer_dlist()` |
+| `remote_indices` | `list[int]` or `numpy.ndarray` | *(required)* | Indices selecting remote descriptors |
+| `notif_msg` | `bytes` | `b""` | Notification message sent after transfer completion |
+| `backends` | `list[str]` | `[]` | Limit which backends NIXL can use |
+| `skip_desc_merge` | `bool` | `False` | *Deprecated.* Whether to skip descriptor merging optimization |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixl_xfer_handle` | Opaque handle for posting and checking the transfer |
+
+```python
+xfer_handle = agent.make_prepped_xfer(
+ "WRITE", local_handle, [0, 1], remote_handle, [0, 1],
+ notif_msg=b"transfer_complete"
+)
+```
+
+### initialize_xfer
+
+Create a transfer request directly from descriptor lists without prior preparation. This is a combined API that prepares the descriptor lists and creates the transfer in one call.
+
+**C++ equivalent:** [`createXferReq`](./cpp-api#createxferreq)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `operation` | `str` | *(required)* | `"WRITE"` or `"READ"` |
+| `local_descs` | `nixlXferDList` | *(required)* | Local transfer descriptors (from `get_xfer_descs()`) |
+| `remote_descs` | `nixlXferDList` | *(required)* | Remote transfer descriptors (from `get_xfer_descs()`) |
+| `remote_agent` | `str` | *(required)* | Name of the remote agent |
+| `notif_msg` | `bytes` | `b""` | Notification message sent after transfer completion |
+| `backends` | `list[str]` | `[]` | Limit which backends NIXL can use |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixl_xfer_handle` | Opaque handle for posting and checking the transfer |
+
+```python
+local_descs = agent.get_xfer_descs(local_tensor)
+remote_descs = agent.get_xfer_descs(remote_tensor)
+xfer_handle = agent.initialize_xfer("WRITE", local_descs, remote_descs, "remote_agent")
+```
+
+
+If you share common descriptors across different transfer requests, prefer `prep_xfer_dlist()` with `make_prepped_xfer()` to avoid repeated preparation overhead. Use `initialize_xfer()` for one-off transfers where simplicity matters more than reuse.
+
+
+## Transfer Operations
+
+
+For a walkthrough of transfer posting and status checking, see [Quick Start -- Creating and Executing Transfers](../getting-started/quick-start#creating-and-executing-transfers).
+
+
+### transfer
+
+Initiate a data transfer. After calling this, poll `check_xfer_state()` until completion.
+
+**C++ equivalent:** [`postXferReq`](./cpp-api#postxferreq)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `handle` | `nixl_xfer_handle` | *(required)* | Transfer handle from `make_prepped_xfer()` or `initialize_xfer()` |
+| `notif_msg` | `bytes` | `b""` | Notification message (can override or set per transfer call) |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `str` | `"DONE"` if completed immediately, `"PROC"` if in progress, `"ERR"` on error |
+
+```python
+status = agent.transfer(xfer_handle)
+if status == "PROC":
+ while agent.check_xfer_state(xfer_handle) == "PROC":
+ pass # Poll until complete
+```
+
+### check_xfer_state
+
+Check the current state of a transfer operation.
+
+**C++ equivalent:** [`getXferStatus`](./cpp-api#getxferstatus)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixl_xfer_handle` | Transfer handle |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `str` | `"DONE"` if complete, `"PROC"` if in progress, `"ERR"` on error |
+
+```python
+status = agent.check_xfer_state(xfer_handle)
+```
+
+### get_xfer_telemetry
+
+Get telemetry data for a transfer request. Requires `capture_telemetry=True` in the agent config.
+
+**C++ equivalent:** [`getXferTelemetry`](./cpp-api#getxfertelemetry)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixl_xfer_handle` | Transfer handle |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `nixlXferTelemetry` | Telemetry object with `startTime`, `postDuration`, `xferDuration` (microseconds), `totalBytes`, and `descCount` fields |
+
+```python
+telem = agent.get_xfer_telemetry(xfer_handle)
+print(f"Transfer took {telem.xferDuration} us for {telem.totalBytes} bytes")
+```
+
+### query_xfer_backend
+
+Query which backend was selected for a transfer operation.
+
+**C++ equivalent:** [`queryXferBackend`](./cpp-api#queryxferbackend)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixl_xfer_handle` | Transfer handle |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `str` | Name of the backend chosen for this transfer (e.g., `"UCX"`) |
+
+```python
+backend = agent.query_xfer_backend(xfer_handle)
+# "UCX"
+```
+
+### release_xfer_handle
+
+Release a transfer handle, freeing associated resources. If the transfer is active, NIXL will attempt to cancel it.
+
+**C++ equivalent:** [`releaseXferReq`](./cpp-api#releasexferreq)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixl_xfer_handle` | Transfer handle to release |
+
+```python
+agent.release_xfer_handle(xfer_handle)
+```
+
+
+This delegates to `handle.release()`. You can also call `release()` directly on the handle object.
+
+
+### release_dlist_handle
+
+Release a prepared descriptor list handle, freeing associated resources.
+
+**C++ equivalent:** [`releasedDlistH`](./cpp-api#releaseddlisth)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixl_prepped_dlist_handle` | Descriptor list handle to release |
+
+```python
+agent.release_dlist_handle(prep_handle)
+```
+
+## Memory View
+
+
+Memory View APIs (`prepMemView`, `releaseMemView`) are not currently exposed in the Python bindings. For memory view operations, refer to the [C++ API Reference -- Memory View](./cpp-api#memory-view).
+
+
+## Notification Handling
+
+### get_new_notifs
+
+Get new notifications that have arrived at the agent since the last call. Returns a fresh dictionary each time (does not accumulate).
+
+**C++ equivalent:** [`getNotifs`](./cpp-api#getnotifs)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `backends` | `list[str]` | `[]` | Limit which backends are checked for notifications; empty means all |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `dict[str, list[bytes]]` | Map of remote agent names to lists of notification messages |
+
+```python
+notifs = agent.get_new_notifs()
+for agent_name, messages in notifs.items():
+ print(f"From {agent_name}: {messages}")
+```
+
+### update_notifs
+
+Get new notifications and accumulate them in the agent's internal notification map. Unlike `get_new_notifs()`, this builds up all unhandled notifications over time.
+
+**C++ equivalent:** [`getNotifs`](./cpp-api#getnotifs) (with accumulation)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `backends` | `list[str]` | `[]` | Limit which backends are checked; empty means all |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `dict[str, list[bytes]]` | Accumulated notification map (same reference as `agent.notifs`) |
+
+```python
+all_notifs = agent.update_notifs()
+```
+
+### check_remote_xfer_done
+
+Check whether a specific notification has been received from a remote agent. This is a Python-only convenience method that combines `update_notifs()` with a tag-based lookup.
+
+**C++ equivalent:** None (Python-only convenience)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `remote_agent_name` | `str` | *(required)* | Name of the remote agent |
+| `lookup_tag` | `bytes` | *(required)* | Tag to match against notification messages |
+| `backends` | `list[str]` | `[]` | Limit which backends are checked |
+| `tag_is_prefix` | `bool` | `True` | If `True`, match `lookup_tag` as a prefix; if `False`, match as substring |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `bool` | `True` if a matching notification was found and removed; `False` otherwise |
+
+```python
+# Poll until remote agent signals completion
+while not agent.check_remote_xfer_done("remote_agent", b"xfer_done"):
+ pass
+```
+
+
+When a matching notification is found, it is removed from the internal notification map. Subsequent calls will not find the same notification.
+
+
+### send_notif
+
+Send a standalone notification to a remote agent, not bound to any transfer.
+
+**C++ equivalent:** [`genNotif`](./cpp-api#gennotif)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `remote_agent_name` | `str` | *(required)* | Name of the remote agent |
+| `notif_msg` | `bytes` | *(required)* | Message to send (received as `bytes` by the target) |
+| `backend` | `str` | `None` | Specific backend to send through; `None` uses the default |
+
+```python
+agent.send_notif("remote_agent", b"step_complete")
+```
+
+## Metadata -- Side Channel
+
+Side-channel metadata exchange uses an out-of-band mechanism to serialize and transfer agent metadata as opaque byte blobs. This is useful when agents communicate through a shared store or message queue.
+
+
+For a walkthrough of metadata exchange patterns, see [Quick Start -- Metadata Exchange](../getting-started/quick-start#metadata-exchange).
+
+
+### get_agent_metadata
+
+Get the full serialized metadata of the local agent, including all registered memory regions and backend connection information.
+
+**C++ equivalent:** [`getLocalMD`](./cpp-api#getlocalmd)
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `bytes` | Serialized agent metadata |
+
+```python
+metadata = agent.get_agent_metadata()
+# Send metadata to remote agent via your preferred transport
+```
+
+### get_partial_agent_metadata
+
+Get partial metadata containing only specified descriptors and optionally connection information. Useful for incremental metadata updates.
+
+**C++ equivalent:** [`getLocalPartialMD`](./cpp-api#getlocalpartialmd)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `descs` | `nixlRegDList` | *(required)* | Descriptors to include; can be empty if only sending connection info |
+| `inc_conn_info` | `bool` | `False` | Whether to include backend connection information |
+| `backends` | `list[str]` | `[]` | Backends to consider when constructing metadata |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `bytes` | Serialized partial metadata |
+
+```python
+partial_md = agent.get_partial_agent_metadata(reg_descs, inc_conn_info=True)
+```
+
+### add_remote_agent
+
+Add a remote agent using its serialized metadata. After this call, the local agent can initiate transfers toward the remote agent.
+
+**C++ equivalent:** [`loadRemoteMD`](./cpp-api#loadremotemd)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `metadata` | `bytes` | Serialized metadata from the remote agent |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `str` | Name of the added remote agent |
+
+```python
+remote_name = agent.add_remote_agent(remote_metadata)
+```
+
+### remove_remote_agent
+
+Remove a remote agent. After this call, the local agent cannot initiate transfers toward that remote agent. This also disconnects the two agents.
+
+**C++ equivalent:** [`invalidateRemoteMD`](./cpp-api#invalidateremotemd)
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `agent` | `str` | Name of the remote agent to remove |
+
+```python
+agent.remove_remote_agent("old_remote_agent")
+```
+
+## Metadata -- Direct Channel
+
+Direct-channel metadata exchange uses the NIXL listen thread to send and receive metadata over the network, or communicates through a central metadata server (e.g., etcd). The listen thread must be enabled in the agent config.
+
+
+For a walkthrough of direct metadata exchange and etcd-based patterns, see [Quick Start -- Metadata Exchange](../getting-started/quick-start#metadata-exchange).
+
+
+### send_local_metadata
+
+Send all local metadata to a specific peer or to a central metadata server.
+
+**C++ equivalent:** [`sendLocalMD`](./cpp-api#sendlocalmd)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `ip_addr` | `str` | `""` | Peer IP address; empty sends to central metadata server |
+| `port` | `int` | `DEFAULT_COMM_PORT` | Port for the peer; ignored when sending to central server |
+
+```python
+# Send to a specific peer
+agent.send_local_metadata(ip_addr="192.168.1.10", port=12345)
+
+# Send to central metadata server
+agent.send_local_metadata()
+```
+
+### send_partial_agent_metadata
+
+Send partial metadata (specific descriptors and optional connection info) to a peer or central server.
+
+**C++ equivalent:** [`sendLocalPartialMD`](./cpp-api#sendlocalpartialmd)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `descs` | `nixlRegDList` | *(required)* | Descriptors to include; can be empty for connection info only |
+| `inc_conn_info` | `bool` | `False` | Whether to include connection information |
+| `backends` | `list[str]` | `[]` | Backends to consider |
+| `ip_addr` | `str` | `""` | Peer IP address; empty sends to central server |
+| `port` | `int` | `DEFAULT_COMM_PORT` | Port for the peer |
+| `label` | `str` | `""` | Label for central metadata server; ignored for peer sends |
+
+```python
+agent.send_partial_agent_metadata(
+ reg_descs, inc_conn_info=True, ip_addr="192.168.1.10", port=12345
+)
+```
+
+### fetch_remote_metadata
+
+Request metadata from a central metadata server or a specific peer.
+
+**C++ equivalent:** [`fetchRemoteMD`](./cpp-api#fetchremotemd)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `remote_agent` | `str` | *(required)* | Name of the remote agent to fetch metadata for |
+| `ip_addr` | `str` | `""` | Peer IP address; empty uses central server |
+| `port` | `int` | `DEFAULT_COMM_PORT` | Port for the peer |
+| `label` | `str` | `""` | Label for central metadata server lookup |
+
+```python
+agent.fetch_remote_metadata("remote_agent", ip_addr="192.168.1.10")
+```
+
+### invalidate_local_metadata
+
+Invalidate local metadata from a central metadata server or a specific peer.
+
+**C++ equivalent:** [`invalidateLocalMD`](./cpp-api#invalidatelocalmd)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `ip_addr` | `str` | `""` | Peer IP address; empty invalidates from central server |
+| `port` | `int` | `DEFAULT_COMM_PORT` | Port for the peer |
+
+```python
+agent.invalidate_local_metadata()
+```
+
+### check_remote_metadata
+
+Check if remote metadata for a specific agent is available. When partial metadata methods are used, you can specify which descriptors to check for.
+
+**C++ equivalent:** [`checkRemoteMD`](./cpp-api#checkremotemd)
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `agent` | `str` | *(required)* | Name of the remote agent |
+| `descs` | `nixlXferDList` | `None` | Specific descriptors to check for; `None` checks for any metadata |
+
+| **Returns** | Type | Description |
+|-------------|------|-------------|
+| | `bool` | `True` if metadata is available, `False` otherwise |
+
+```python
+if agent.check_remote_metadata("remote_agent"):
+ print("Remote metadata is available")
+```
diff --git a/docs/api-reference/rust-api.md b/docs/api-reference/rust-api.md
new file mode 100644
index 0000000000..6c71d5c4b0
--- /dev/null
+++ b/docs/api-reference/rust-api.md
@@ -0,0 +1,1231 @@
+---
+title: Rust API Reference
+description: Complete Rust API reference for NIXL data transfers.
+---
+
+This is the Rust API reference for the NVIDIA Inference Xfer Library (NIXL). The Rust bindings provide a safe, idiomatic interface through the `Agent` struct. For the C++ native API, see the [C++ API Reference](./cpp-api). For Python bindings, see the [Python API Reference](./python-api).
+
+Behavior shared by all bindings is documented in [Northbound API Semantics](./northbound-api).
+
+Key Rust-specific features of the NIXL bindings:
+
+- **Result-based error handling**: All fallible operations return `Result`
+- **RAII resource management**: `Agent`, `XferRequest`, `XferDlistHandle`, `RegistrationHandle`, `Backend`, `OptArgs`, `NotificationMap`, `RegDescList`, `XferDescList`, `QueryResponseList`, `Params`, and `MemList` all implement `Drop` for automatic cleanup
+- **Thread safety**: `Agent` wraps inner state in `Arc>`, making it `Clone`, `Send`, and `Sync`
+- **Trait-based extensibility**: `MemoryRegion`, `NixlDescriptor`, `NixlRegistration` traits for custom memory types
+
+
+`Agent::clone()` creates a handle to the **same** underlying agent via `Arc`, not an independent copy. All clones share the same backends, registrations, and metadata. This is intentional for multi-threaded usage -- clone the `Agent` and move it into threads or tasks.
+
+
+To use the NIXL Rust API, add `nixl_sys` as a dependency and import:
+
+```rust
+use nixl_sys::Agent;
+```
+
+## Types, Enums and Defines
+
+This section documents the structs, enums, traits, and handle types that make up the Rust NIXL API. Subsections are organized to mirror the C++ API structure.
+
+### Structs and Configuration
+
+### Agent
+
+The main Transfer Agent struct. Each agent represents one endpoint in a data transfer and manages backends, memory registrations, metadata, and transfer operations.
+
+`Agent` implements `Clone` (shared state via `Arc>`), `Send`, `Sync`, `Debug`, and `Drop`. The `Drop` implementation invalidates all remote metadata, destroys all backends, and destroys the underlying C agent handle.
+
+```rust
+use nixl_sys::Agent;
+let agent = Agent::new("my_agent")?;
+```
+
+### AgentConfig
+
+Per-agent configuration struct with a `Default` implementation. All fields have defaults, so `AgentConfig::default()` is valid for most use cases.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `enable_prog_thread` | `bool` | `true` | Enable progress thread for asynchronous transfer advancement |
+| `enable_listen_thread` | `bool` | `false` | Enable listener thread for incoming peer connections |
+| `listen_port` | `i32` | `0` | Port for the listener thread (0 = system-assigned) |
+| `thread_sync` | `ThreadSync` | `ThreadSync::None` | Thread synchronization mode for multi-threaded usage |
+| `num_workers` | `u32` | `1` | Number of worker threads |
+| `pthr_delay_us` | `u64` | `0` | Progress thread delay between iterations (microseconds) |
+| `lthr_delay_us` | `u64` | `100_000` | Listener thread sleep duration (microseconds) |
+| `capture_telemetry` | `bool` | `false` | Enable telemetry capture for transfer performance metrics |
+
+```rust
+use nixl_sys::{Agent, AgentConfig, ThreadSync};
+
+let cfg = AgentConfig {
+ enable_prog_thread: true,
+ enable_listen_thread: true,
+ listen_port: 5555,
+ ..Default::default()
+};
+let agent = Agent::new_configured("my_agent", &cfg)?;
+```
+
+### Enumerations
+
+### ThreadSync
+
+Thread synchronization mode enum for multi-threaded agent usage.
+
+| Variant | Description |
+|---------|-------------|
+| `None` | No synchronization (default). Single-threaded usage only. |
+| `Strict` | Full mutual exclusion on all operations. |
+| `Rw` | Reader-writer lock: concurrent reads, exclusive writes. |
+| `Default` | Alias for `None`. |
+
+**C++ equivalent:** [`nixl_thread_sync_t`](./cpp-api#nixl_thread_sync_t)
+
+### XferStatus
+
+Transfer status enum returned by `get_xfer_status()`.
+
+| Variant | Description |
+|---------|-------------|
+| `Success` | Transfer completed successfully |
+| `InProgress` | Transfer is still running |
+
+The `is_success()` helper method returns `true` for `XferStatus::Success`.
+
+```rust
+let status = agent.get_xfer_status(&req)?;
+if status.is_success() {
+ println!("Transfer complete");
+}
+```
+
+### XferOp
+
+Transfer operation direction.
+
+| Variant | Description |
+|---------|-------------|
+| `Read` | Read data from the remote side into local buffers |
+| `Write` | Write data from local buffers to the remote side |
+
+**C++ equivalent:** [`nixl_xfer_op_t`](./cpp-api#nixl_xfer_op_t)
+
+### CostMethod
+
+Cost estimation method identifier returned by `estimate_xfer_cost()`.
+
+| Variant | Description |
+|---------|-------------|
+| `AnalyticalBackend` | Analytical backend cost estimate |
+| `Unknown` | Unknown estimation method |
+
+**C++ equivalent:** [`nixl_cost_t`](./cpp-api#nixl_cost_t)
+
+### NixlError
+
+Error enum for all NIXL operations. All fallible methods return `Result`.
+
+| Variant | Description |
+|---------|-------------|
+| `InvalidParam` | Invalid parameter provided to NIXL |
+| `BackendError` | Backend-level error occurred |
+| `StringConversionError(NulError)` | Failed to convert string (contains interior nul byte) |
+| `IndexOutOfBounds` | Index out of bounds |
+| `InvalidDataPointer` | Invalid data pointer returned from C API |
+| `FailedToCreateXferRequest` | Failed to create a transfer request |
+| `RegDescListCreationFailed` | Failed to create a registration descriptor list |
+| `RegDescAddFailed` | Failed to add a registration descriptor |
+| `FailedToCreateXferDlistHandle` | Failed to create a transfer descriptor list handle |
+| `FailedToCreateBackend` | Failed to create a backend |
+| `NoTelemetry` | Telemetry is not enabled or transfer is not complete |
+
+`NixlError` implements `std::error::Error` (via `thiserror`) and `Debug`, making it compatible with the standard Rust error handling ecosystem.
+
+```rust
+match agent.create_backend("UCX", ¶ms) {
+ Ok(backend) => { /* success */ }
+ Err(NixlError::InvalidParam) => { /* handle invalid parameter */ }
+ Err(NixlError::FailedToCreateBackend) => { /* handle backend creation failure */ }
+ Err(e) => { /* other error */ }
+}
+```
+
+### Backend
+
+Opaque handle to a NIXL backend engine. Implements `Send + Sync` (safe to share across threads) and `Debug`. The backend is destroyed when the owning `Agent` is dropped.
+
+**C++ equivalent:** [`nixlBackendH*`](./cpp-api#handle-types)
+
+### OptArgs
+
+Optional arguments struct passed to many agent methods. Uses setter methods to configure fields.
+
+| Method | Description |
+|--------|-------------|
+| `OptArgs::new()` | Create a new empty optional arguments struct |
+| `add_backend(&mut self, backend: &Backend)` | Add a backend to limit the operation scope |
+| `set_notification_message(&mut self, message: &[u8])` | Set the notification message as raw bytes |
+| `get_notification_message(&self)` | Get the notification message |
+| `set_has_notification(&mut self, has: bool)` | Enable or disable notification |
+| `has_notification(&self)` | Check if notification is enabled |
+| `set_skip_descriptor_merge(&mut self, skip: bool)` | Set whether to skip descriptor merging |
+| `skip_descriptor_merge(&self)` | Check if descriptor merging is skipped |
+| `set_ip_addr(&mut self, ip: &str)` | Set IP address for peer-to-peer metadata operations |
+| `set_port(&mut self, port: u16)` | Set port for metadata operations |
+
+All setter methods return `Result<(), NixlError>`. `OptArgs` implements `Drop` for automatic cleanup.
+
+**C++ equivalent:** [`nixlAgentOptionalArgs`](./cpp-api#nixlagentoptionalargs)
+
+```rust
+use nixl_sys::OptArgs;
+
+let mut opts = OptArgs::new()?;
+opts.set_notification_message(b"transfer_done")?;
+opts.add_backend(&backend)?;
+```
+
+### MemType
+
+Memory segment types supported by NIXL.
+
+| Variant | Description |
+|---------|-------------|
+| `Dram` | Standard host memory (CPU DRAM) |
+| `Vram` | GPU high-bandwidth memory (HBM/VRAM) |
+| `Block` | Block-level storage devices |
+| `Object` | Distributed object stores (S3, Azure Blob) |
+| `File` | Local and remote file systems |
+| `Unknown` | Unknown memory type |
+
+`MemType` derives `Debug`, `Clone`, `Copy`, `PartialEq`, `Eq`, `Serialize`, `Deserialize`, and implements `Display`.
+
+**C++ equivalent:** [`nixl_mem_t`](./cpp-api#nixl_mem_t)
+
+### Descriptor Types
+
+NIXL uses descriptor lists to represent sets of memory regions for registration and transfer.
+
+**`RegDescList`** -- Registration descriptor list for memory segments to register.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `RegDescList::new(mem_type: MemType)` | `Result` | Create a new list for the given memory type |
+| `add_desc(&mut self, addr, len, dev_id)` | `()` | Add a descriptor (address, length, device ID) |
+| `add_desc_with_meta(&mut self, addr, len, dev_id, metadata)` | `()` | Add a descriptor with metadata bytes |
+| `add_storage_desc(&mut self, desc: &dyn NixlDescriptor)` | `Result<(), NixlError>` | Add from a type implementing `NixlDescriptor` |
+| `len()` | `Result` | Number of descriptors |
+| `is_empty()` | `Result` | Check if list is empty |
+| `get(index)` | `Result<&RegDescriptor, NixlError>` | Access descriptor by index |
+| `rem_desc(index)` | `Result<(), NixlError>` | Remove descriptor at index |
+| `clear()` | `()` | Remove all descriptors |
+| `serialize()` / `deserialize(bytes)` | `Result, NixlError>` / `Result` | Bincode serialization |
+
+Also supports `Index`, `IndexMut`, `PartialEq`, `Debug`, and `Drop`.
+
+**C++ equivalent:** [`nixl_reg_dlist_t`](./cpp-api#nixl-descriptors)
+
+**`XferDescList`** -- Transfer descriptor list for memory regions in transfer operations.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `XferDescList::new(mem_type: MemType)` | `Result` | Create a new list for the given memory type |
+| `add_desc(&mut self, addr, len, dev_id)` | `()` | Add a descriptor (address, length, device ID) |
+| `add_storage_desc(&mut self, desc: &D)` | `Result<(), NixlError>` | Add from a type implementing `NixlDescriptor` |
+| `len()` | `Result` | Number of descriptors |
+| `is_empty()` | `Result` | Check if list is empty |
+| `get(index)` | `Result<&XferDescriptor, NixlError>` | Access descriptor by index |
+| `rem_desc(index)` | `Result<(), NixlError>` | Remove descriptor at index |
+| `clear()` | `()` | Remove all descriptors |
+| `serialize()` / `deserialize(bytes)` | `Result, NixlError>` / `Result` | Bincode serialization |
+
+Also supports `Index`, `IndexMut`, `PartialEq`, `Debug`, and `Drop`.
+
+**C++ equivalent:** [`nixl_xfer_dlist_t`](./cpp-api#nixl-descriptors)
+
+**`XferDlistHandle`** -- Opaque handle to a prepared descriptor list. Returned by `prepare_xfer_dlist()` and consumed by `make_xfer_req()`. Implements `Drop`, which releases the prepared list on the agent side.
+
+**C++ equivalent:** [`nixlDlistH*`](./cpp-api#handle-types)
+
+**`RegistrationHandle`** -- Handle to a registered memory region. Implements `Drop`, which automatically deregisters the memory via the agent. Call `deregister()` for explicit deregistration.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `agent_name()` | `Option` | Get the name of the agent this memory is registered with |
+| `deregister(&mut self)` | `Result<(), NixlError>` | Explicitly deregister the memory |
+
+
+When a `RegistrationHandle` is dropped, it automatically deregisters the memory from the agent. You do not need to call `deregister()` explicitly unless you need to handle errors.
+
+
+### Transfer and Notification Types
+
+### XferRequest
+
+Handle to an active transfer request. Implements `Send + Sync` (safe to share across threads) and `Drop` (auto-releases the request and underlying resources).
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `get_telemetry(&self)` | `Result` | Get telemetry data for this transfer |
+
+**C++ equivalent:** [`nixlXferReqH*`](./cpp-api#handle-types)
+
+### NotificationMap
+
+Container for notifications received from remote agents. Implements `Drop` for automatic cleanup.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `NotificationMap::new()` | `Result` | Create a new empty notification map |
+| `len()` | `Result` | Number of agents with notifications |
+| `is_empty()` | `Result` | Check if there are no notifications |
+| `agents()` | `NotificationMapAgentIterator` | Iterator over agent names |
+| `get_notifications_size(agent_name)` | `Result` | Number of notifications for an agent |
+| `get_notifications(agent_name)` | `Result` | Iterator over notifications (as raw bytes) |
+| `get_notification_bytes(agent_name, index)` | `Result, NixlError>` | Get a specific notification as bytes |
+| `take_notifs()` | `Result>, NixlError>` | Extract all notifications as strings and clear the map |
+
+```rust
+use nixl_sys::NotificationMap;
+
+let mut notifs = NotificationMap::new()?;
+agent.get_notifications(&mut notifs, None)?;
+for agent_name in notifs.agents() {
+ let name = agent_name?;
+ println!("Notifications from: {}", name);
+}
+```
+
+### XferTelemetry
+
+Transfer telemetry data containing timing and performance metrics.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `start_time_us` | `u64` | Start time in microseconds since epoch |
+| `post_duration_us` | `u64` | Post operation duration in microseconds |
+| `xfer_duration_us` | `u64` | Transfer duration in microseconds |
+| `total_bytes` | `u64` | Total bytes transferred |
+| `desc_count` | `u64` | Number of descriptors in the transfer |
+
+| Helper Method | Returns | Description |
+|---------------|---------|-------------|
+| `start_time()` | `Duration` | Start time as a `Duration` since Unix epoch |
+| `post_duration()` | `Duration` | Post operation duration |
+| `xfer_duration()` | `Duration` | Transfer duration |
+| `total_duration()` | `Duration` | Total duration (post + transfer) |
+| `transfer_rate_bps()` | `f64` | Transfer rate in bytes per second |
+
+**C++ equivalent:** [`nixlXferTelemetry`](./cpp-api#nixlxfertelemetry)
+
+### Collection Types
+
+### MemList
+
+List of memory types supported by a backend. Returned by `get_plugin_params()` and `get_backend_params()`. Implements `Drop` for automatic cleanup.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `is_empty()` | `Result` | Check if the list is empty |
+| `len()` | `Result` | Number of memory types |
+| `get(index)` | `Result` | Get memory type at index |
+| `iter()` | `MemListIterator` | Iterator over memory types |
+
+### Params
+
+Key-value parameter map used for backend configuration. Returned by `get_plugin_params()` and `get_backend_params()`, and accepted by `create_backend()`. Implements `Drop` for automatic cleanup.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `Params::from(iter)` | `Result` | Create from an iterator of `(key, value)` pairs |
+| `set(key, value)` | `Result<(), NixlError>` | Set a key-value pair |
+| `is_empty()` | `Result` | Check if empty |
+| `iter()` | `Result` | Iterator over key-value pairs |
+| `clone()` | `Result` | Deep copy the parameters |
+
+Supports `IntoIterator` (yields `(&str, &str)` pairs) and conversion to `HashMap`.
+
+**C++ equivalent:** [`nixl_b_params_t`](./cpp-api#type-aliases)
+
+```rust
+use nixl_sys::Params;
+use std::collections::HashMap;
+
+let params = Params::from([("key1", "value1"), ("key2", "value2")])?;
+let map: HashMap = HashMap::from(params.iter()?);
+```
+
+### QueryResponseList
+
+List of query responses from `query_mem()`. Each response may optionally contain a `Params` object. Implements `Drop`.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `len()` | `Result` | Number of responses |
+| `is_empty()` | `Result` | Check if empty |
+| `get(index)` | `Result` | Get response at index |
+| `iter()` | `Result` | Iterator over responses |
+
+Each `QueryResponse` has `has_value()` and `get_params()` methods.
+
+### SystemStorage
+
+Built-in DRAM (`MemType::Dram`) implementation of `NixlRegistration`. Provides a simple heap-allocated buffer that can be registered with an agent.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `SystemStorage::new(size: usize)` | `Result` | Allocate a new zero-initialized buffer |
+| `memset(value: u8)` | `()` | Fill the buffer with a byte value |
+| `as_slice()` | `&[u8]` | Get a read-only slice of the data |
+
+`SystemStorage` implements `MemoryRegion`, `NixlDescriptor` (with `mem_type() -> Dram`, `device_id() -> 0`), and `NixlRegistration`.
+
+```rust
+use nixl_sys::{Agent, SystemStorage, NixlRegistration};
+
+let agent = Agent::new("my_agent")?;
+let mut storage = SystemStorage::new(1024)?;
+storage.register(&agent, None)?;
+// storage is now registered with the agent
+// Drop automatically deregisters when storage goes out of scope
+```
+
+### Trait Hierarchy
+
+NIXL defines three traits for custom memory types, forming a hierarchy:
+
+```
+MemoryRegion
+ |
+ +-- NixlDescriptor
+ |
+ +-- NixlRegistration
+```
+
+**`MemoryRegion`** -- Base trait for types that represent a contiguous memory region. Requires `Debug + Send + Sync`.
+
+```rust
+pub trait MemoryRegion: std::fmt::Debug + Send + Sync {
+ /// Get a raw pointer to the storage
+ unsafe fn as_ptr(&self) -> *const u8;
+ /// Returns the total size in bytes
+ fn size(&self) -> usize;
+}
+```
+
+**`NixlDescriptor: MemoryRegion`** -- Extends `MemoryRegion` with NIXL-specific metadata.
+
+```rust
+pub trait NixlDescriptor: MemoryRegion {
+ /// Get the memory type
+ fn mem_type(&self) -> MemType;
+ /// Get the device ID
+ fn device_id(&self) -> u64;
+}
+```
+
+**`NixlRegistration: NixlDescriptor`** -- Extends `NixlDescriptor` with the ability to register memory with an agent.
+
+```rust
+pub trait NixlRegistration: NixlDescriptor {
+ fn register(
+ &mut self,
+ agent: &Agent,
+ opt_args: Option<&OptArgs>,
+ ) -> Result<(), NixlError>;
+}
+```
+
+Implement these traits on your own types to integrate custom memory (e.g., GPU memory, NVMe buffers) with NIXL. See `SystemStorage` for a reference implementation.
+
+## Initialization and Configuration
+
+
+For a complete workflow example, see [Quick Start -- Agent Initialization](../getting-started/quick-start#agent-initialization).
+
+
+### new
+
+Create a new Transfer Agent with default configuration.
+
+**C++ equivalent:** [`nixlAgent`](./cpp-api#nixlagent)
+
+```rust
+pub fn new(name: &str) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `name` | `&str` | Unique name for this agent |
+| **Returns** | `Result` | The constructed agent or error |
+
+```rust
+let agent = Agent::new("my_agent")?;
+```
+
+### new_configured
+
+Create a new Transfer Agent with custom configuration.
+
+**C++ equivalent:** [`nixlAgent`](./cpp-api#nixlagent) (with config parameter)
+
+```rust
+pub fn new_configured(name: &str, cfg: &AgentConfig) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `name` | `&str` | Unique name for this agent |
+| `cfg` | `&AgentConfig` | Agent configuration struct |
+| **Returns** | `Result` | The constructed agent or error |
+
+```rust
+let cfg = AgentConfig {
+ enable_prog_thread: true,
+ capture_telemetry: true,
+ ..Default::default()
+};
+let agent = Agent::new_configured("my_agent", &cfg)?;
+```
+
+### name
+
+Get the name of this agent.
+
+```rust
+pub fn name(&self) -> String
+```
+
+| **Returns** | `String` | The agent name |
+
+### get_available_plugins
+
+Discover the available backend plug-ins found in the plug-in search paths.
+
+**C++ equivalent:** [`getAvailPlugins`](./cpp-api#getavailplugins)
+
+```rust
+pub fn get_available_plugins(&self) -> Result
+```
+
+| **Returns** | `Result` | List of available plug-in names |
+
+```rust
+let plugins = agent.get_available_plugins()?;
+for i in 0..plugins.len()? {
+ println!("Plugin: {}", plugins.get(i)?);
+}
+```
+
+
+The `Agent` destructor (`Drop`) automatically invalidates all remote metadata, destroys all backends, and releases the underlying C handle. No explicit cleanup is needed.
+
+
+## Backend Management
+
+
+For a complete workflow example, see [Quick Start -- Backend Creation](../getting-started/quick-start#backend-creation).
+
+
+### get_plugin_params
+
+Get the supported memory types and initialization parameters for a backend plug-in, before creating an instance.
+
+**C++ equivalent:** [`getPluginParams`](./cpp-api#getpluginparams)
+
+```rust
+pub fn get_plugin_params(
+ &self,
+ plugin_name: &str,
+) -> Result<(MemList, Params), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `plugin_name` | `&str` | Plugin backend type name (e.g., `"UCX"`) |
+| **Returns** | `Result<(MemList, Params), NixlError>` | Supported memory types and default parameters |
+
+```rust
+let (mems, params) = agent.get_plugin_params("UCX")?;
+for mem in mems.iter() {
+ println!("Supports: {:?}", mem?);
+}
+```
+
+### get_backend_params
+
+Get the parameters and memory types of an already-instantiated backend.
+
+**C++ equivalent:** [`getBackendParams`](./cpp-api#getbackendparams)
+
+```rust
+pub fn get_backend_params(
+ &self,
+ backend: &Backend,
+) -> Result<(MemList, Params), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `backend` | `&Backend` | Backend handle from `create_backend()` |
+| **Returns** | `Result<(MemList, Params), NixlError>` | Memory types and current parameters |
+
+### create_backend
+
+Instantiate a backend engine with the given parameters.
+
+**C++ equivalent:** [`createBackend`](./cpp-api#createbackend)
+
+```rust
+pub fn create_backend(
+ &self,
+ plugin: &str,
+ params: &Params,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `plugin` | `&str` | Backend type name (e.g., `"UCX"`, `"GDS"`) |
+| `params` | `&Params` | Backend-specific initialization parameters |
+| **Returns** | `Result` | Backend handle for subsequent operations |
+
+
+Multiple backends can be created on the same agent. NIXL automatically selects the best backend for each transfer based on the source and destination memory types and the backends available on both agents.
+
+
+```rust
+let params = Params::from([("key", "value")])?;
+let backend = agent.create_backend("UCX", ¶ms)?;
+```
+
+### get_backend
+
+Get a backend by name.
+
+```rust
+pub fn get_backend(&self, name: &str) -> Option
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `name` | `&str` | Backend name (same as the plug-in name used in `create_backend()`) |
+| **Returns** | `Option` | Backend handle if found, `None` otherwise |
+
+## Memory Registration
+
+
+For a complete workflow example, see [Quick Start -- Memory Registration](../getting-started/quick-start#memory-registration).
+
+
+### register_memory
+
+Register a memory descriptor with the agent. Returns a `RegistrationHandle` that automatically deregisters the memory when dropped.
+
+**C++ equivalent:** [`registerMem`](./cpp-api#registermem)
+
+```rust
+pub fn register_memory(
+ &self,
+ descriptor: &impl NixlDescriptor,
+ opt_args: Option<&OptArgs>,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descriptor` | `&impl NixlDescriptor` | Memory descriptor to register (e.g., `&SystemStorage`) |
+| `opt_args` | `Option<&OptArgs>` | Optional. If `backends` is set via `add_backend()`, registration is limited to those backends. |
+| **Returns** | `Result` | Handle that auto-deregisters on drop |
+
+
+If no backend hints are provided, NIXL auto-selects all compatible backends for the given memory type. Memory must be registered before metadata exchange.
+
+
+You can also use the `NixlRegistration` trait pattern for a more ergonomic API:
+
+```rust
+use nixl_sys::{SystemStorage, NixlRegistration};
+
+let mut storage = SystemStorage::new(4096)?;
+storage.register(&agent, None)?;
+// storage now holds the RegistrationHandle internally
+```
+
+### query_mem
+
+Query information about registered memory or storage segments from a specific backend.
+
+**C++ equivalent:** [`queryMem`](./cpp-api#querymem)
+
+```rust
+pub fn query_mem(
+ &self,
+ descs: &RegDescList,
+ opt_args: Option<&OptArgs>,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `&RegDescList` | Registration descriptor list to query |
+| `opt_args` | `Option<&OptArgs>` | The target backend should be specified via `add_backend()` |
+| **Returns** | `Result` | Query responses for each descriptor |
+
+```rust
+let mut opts = OptArgs::new()?;
+opts.add_backend(&backend)?;
+let responses = agent.query_mem(&descs, Some(&opts))?;
+for resp in responses.iter()? {
+ if resp.has_value()? {
+ let params = resp.get_params()?;
+ // inspect params...
+ }
+}
+```
+
+### make_connection
+
+Proactively establish a connection to a remote agent, instead of deferring it to the first transfer.
+
+**C++ equivalent:** [`makeConnection`](./cpp-api#makeconnection)
+
+```rust
+pub fn make_connection(
+ &self,
+ remote_agent: &str,
+ opt_args: Option<&OptArgs>,
+) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `&str` | Name of the remote agent to connect to |
+| `opt_args` | `Option<&OptArgs>` | Optional. Limit connection to specific backends. |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+
+Connections are normally established lazily on the first transfer. Use `make_connection()` to pre-establish connections and avoid first-transfer latency.
+
+
+```rust
+agent.make_connection("remote_agent", None)?;
+```
+
+## Transfer Preparation
+
+
+For a complete workflow example, see [Quick Start -- Creating and Executing Transfers](../getting-started/quick-start#creating-and-executing-transfers).
+
+
+### prepare_xfer_dlist
+
+Prepare a descriptor list for use in transfer requests. Elements from the prepared list can later be selected by index in `make_xfer_req()`.
+
+**C++ equivalent:** [`prepXferDlist`](./cpp-api#prepxferdlist-4-parameter)
+
+```rust
+pub fn prepare_xfer_dlist(
+ &self,
+ agent_name: &str,
+ descs: &XferDescList,
+ opt_args: Option<&OptArgs>,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `agent_name` | `&str` | Agent name for the prepared list. Use `""` for local descriptors, the remote agent's name for remote descriptors. |
+| `descs` | `&XferDescList` | Descriptor list to prepare |
+| `opt_args` | `Option<&OptArgs>` | Optional. Limit preparation to specific backends. |
+| **Returns** | `Result` | Prepared descriptor list handle (auto-released on drop) |
+
+
+Use `""` (empty string) as `agent_name` for local descriptors, equivalent to `NIXL_INIT_AGENT` in C++. Use the remote agent's name for remote-side descriptors.
+
+
+```rust
+let local_hndl = agent.prepare_xfer_dlist("", &local_descs, None)?;
+let remote_hndl = agent.prepare_xfer_dlist("remote_agent", &remote_descs, None)?;
+```
+
+### make_xfer_req
+
+Create a transfer request by selecting indices from already-prepared descriptor list handles.
+
+**C++ equivalent:** [`makeXferReq`](./cpp-api#makexferreq)
+
+```rust
+pub fn make_xfer_req(
+ &self,
+ operation: XferOp,
+ local_descs: &XferDlistHandle,
+ local_indices: &[i32],
+ remote_descs: &XferDlistHandle,
+ remote_indices: &[i32],
+ opt_args: Option<&OptArgs>,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `XferOp` | Transfer direction (`Read` or `Write`) |
+| `local_descs` | `&XferDlistHandle` | Local prepared descriptor list handle |
+| `local_indices` | `&[i32]` | Indices into the local descriptor list |
+| `remote_descs` | `&XferDlistHandle` | Remote prepared descriptor list handle |
+| `remote_indices` | `&[i32]` | Indices into the remote descriptor list |
+| `opt_args` | `Option<&OptArgs>` | Optional. Limit backend selection or attach notification. |
+| **Returns** | `Result` | Transfer request handle (auto-released on drop) |
+
+```rust
+let indices = [0, 1, 2];
+let req = agent.make_xfer_req(
+ XferOp::Write,
+ &local_hndl, &indices,
+ &remote_hndl, &indices,
+ None,
+)?;
+```
+
+### create_xfer_req
+
+Combined API that creates a transfer request directly from two descriptor lists. Internally prepares both sides and creates the transfer handle. Equivalent to calling `prepare_xfer_dlist()` for each side followed by `make_xfer_req()` with all indices.
+
+**C++ equivalent:** [`createXferReq`](./cpp-api#createxferreq)
+
+```rust
+pub fn create_xfer_req(
+ &self,
+ operation: XferOp,
+ local_descs: &XferDescList,
+ remote_descs: &XferDescList,
+ remote_agent: &str,
+ opt_args: Option<&OptArgs>,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `XferOp` | Transfer direction (`Read` or `Write`) |
+| `local_descs` | `&XferDescList` | Local descriptor list |
+| `remote_descs` | `&XferDescList` | Remote descriptor list |
+| `remote_agent` | `&str` | Remote agent name |
+| `opt_args` | `Option<&OptArgs>` | Optional. Limit backend selection or attach notification. |
+| **Returns** | `Result` | Transfer request handle (auto-released on drop) |
+
+
+If the same descriptors are reused across multiple transfers, prefer `prepare_xfer_dlist()` + `make_xfer_req()` to avoid repeated preparation overhead. `create_xfer_req()` is simpler for one-off transfers.
+
+
+```rust
+let req = agent.create_xfer_req(
+ XferOp::Write,
+ &local_descs,
+ &remote_descs,
+ "remote_agent",
+ None,
+)?;
+```
+
+## Transfer Operations
+
+
+For a complete workflow example, see [Quick Start -- Creating and Executing Transfers](../getting-started/quick-start#creating-and-executing-transfers).
+
+
+### estimate_xfer_cost
+
+Estimate the cost (duration) of executing a transfer request before posting it.
+
+**C++ equivalent:** [`estimateXferCost`](./cpp-api#estimatexfercost)
+
+```rust
+pub fn estimate_xfer_cost(
+ &self,
+ req: &XferRequest,
+ opt_args: Option<&OptArgs>,
+) -> Result<(i64, i64, CostMethod), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req` | `&XferRequest` | Transfer request handle |
+| `opt_args` | `Option<&OptArgs>` | Optional arguments |
+| **Returns** | `Result<(i64, i64, CostMethod), NixlError>` | Tuple of (duration_us, error_margin_us, method) |
+
+```rust
+let (duration_us, margin_us, method) = agent.estimate_xfer_cost(&req, None)?;
+println!("Estimated: {}us +/- {}us ({:?})", duration_us, margin_us, method);
+```
+
+### post_xfer_req
+
+Submit a transfer request, initiating the data transfer. The operation is non-blocking.
+
+**C++ equivalent:** [`postXferReq`](./cpp-api#postxferreq)
+
+```rust
+pub fn post_xfer_req(
+ &self,
+ req: &XferRequest,
+ opt_args: Option<&OptArgs>,
+) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req` | `&XferRequest` | Transfer request handle |
+| `opt_args` | `Option<&OptArgs>` | Optional. Notification message can be provided. |
+| **Returns** | `Result` | `false` if transfer completed inline, `true` if in progress |
+
+```rust
+let in_progress = agent.post_xfer_req(&req, None)?;
+if in_progress {
+ // Poll with get_xfer_status()
+}
+```
+
+### get_xfer_status
+
+Check the status of a transfer request.
+
+**C++ equivalent:** [`getXferStatus`](./cpp-api#getxferstatus)
+
+```rust
+pub fn get_xfer_status(&self, req: &XferRequest) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req` | `&XferRequest` | Transfer request handle |
+| **Returns** | `Result` | `Success` or `InProgress` |
+
+```rust
+loop {
+ match agent.get_xfer_status(&req)? {
+ XferStatus::Success => {
+ println!("Transfer complete");
+ break;
+ }
+ XferStatus::InProgress => {
+ // continue polling
+ }
+ }
+}
+```
+
+### get_telemetry (on XferRequest)
+
+Get telemetry data for a completed transfer request. Called on the `XferRequest` directly.
+
+**C++ equivalent:** [`getXferTelemetry`](./cpp-api#getxfertelemetry)
+
+```rust
+pub fn get_telemetry(&self) -> Result
+```
+
+| **Returns** | `Result` | Timing and performance metrics |
+
+
+Telemetry is only available if `capture_telemetry` was set to `true` in `AgentConfig` and the transfer has completed. Returns `NixlError::NoTelemetry` otherwise.
+
+
+```rust
+let telemetry = req.get_telemetry()?;
+println!("Transfer rate: {:.2} MB/s",
+ telemetry.transfer_rate_bps() / 1_000_000.0);
+```
+
+### query_xfer_backend
+
+Query which backend was selected for a transfer request.
+
+**C++ equivalent:** [`queryXferBackend`](./cpp-api#queryxferbackend)
+
+```rust
+pub fn query_xfer_backend(&self, req: &XferRequest) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `req` | `&XferRequest` | Transfer request handle |
+| **Returns** | `Result` | Backend handle used for this transfer |
+
+
+`XferRequest` implements `Drop`, which automatically releases the transfer request and underlying resources. No explicit release call is needed -- simply let the request go out of scope.
+
+
+## Memory View
+
+
+Memory View is not currently exposed in the Rust bindings. For Memory View functionality, use the [C++ API](./cpp-api#memory-view) directly or request this feature in the NIXL repository.
+
+
+## Notification Handling
+
+### get_notifications
+
+Retrieve pending notifications from remote agents.
+
+**C++ equivalent:** [`getNotifs`](./cpp-api#getnotifs)
+
+```rust
+pub fn get_notifications(
+ &self,
+ notifs: &mut NotificationMap,
+ opt_args: Option<&OptArgs>,
+) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `notifs` | `&mut NotificationMap` | Notification map to populate |
+| `opt_args` | `Option<&OptArgs>` | Optional. Use `add_backend()` to filter by backend. |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+```rust
+let mut notifs = NotificationMap::new()?;
+agent.get_notifications(&mut notifs, None)?;
+
+// Iterate through notifications
+let all = notifs.take_notifs()?;
+for (agent_name, messages) in &all {
+ for msg in messages {
+ println!("{}: {}", agent_name, msg);
+ }
+}
+```
+
+### send_notification
+
+Send a notification to a remote agent.
+
+**C++ equivalent:** [`genNotif`](./cpp-api#gennotif)
+
+```rust
+pub fn send_notification(
+ &self,
+ remote_agent: &str,
+ message: &[u8],
+ backend: Option<&Backend>,
+) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `&str` | Name of the remote agent |
+| `message` | `&[u8]` | Notification message as raw bytes |
+| `backend` | `Option<&Backend>` | Optional backend to use for sending |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+```rust
+agent.send_notification("remote_agent", b"transfer_done", None)?;
+```
+
+## Metadata -- Side Channel
+
+
+For a complete workflow example, see [Quick Start -- Metadata Exchange](../getting-started/quick-start#metadata-exchange).
+
+
+The side channel uses serialized byte arrays to exchange metadata between agents without relying on etcd. One agent calls `get_local_md()` to serialize its metadata, sends the bytes over an application-level channel (TCP, gRPC, shared memory, etc.), and the other agent calls `load_remote_md()` to deserialize and load it.
+
+### get_local_md
+
+Get this agent's metadata serialized as a byte array, suitable for sending to a remote agent via an application-level channel.
+
+**C++ equivalent:** [`getLocalMD`](./cpp-api#getlocalmd)
+
+```rust
+pub fn get_local_md(&self) -> Result, NixlError>
+```
+
+| **Returns** | `Result, NixlError>` | Serialized metadata bytes |
+
+```rust
+let metadata = agent.get_local_md()?;
+// Send metadata bytes to remote agent via your transport
+```
+
+### get_local_partial_md
+
+Get partial metadata for specific registered memory regions.
+
+**C++ equivalent:** [`getLocalPartialMD`](./cpp-api#getlocalpartialmd)
+
+```rust
+pub fn get_local_partial_md(
+ &self,
+ descs: &RegDescList,
+ opt_args: Option<&OptArgs>,
+) -> Result, NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `&RegDescList` | Registration descriptor list to get metadata for |
+| `opt_args` | `Option<&OptArgs>` | Optional arguments |
+| **Returns** | `Result, NixlError>` | Serialized partial metadata bytes |
+
+### load_remote_md
+
+Load a remote agent's metadata from a byte slice received via an application-level channel.
+
+**C++ equivalent:** [`loadRemoteMD`](./cpp-api#loadremotemd)
+
+```rust
+pub fn load_remote_md(&self, metadata: &[u8]) -> Result
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `metadata` | `&[u8]` | Serialized metadata bytes from the remote agent |
+| **Returns** | `Result` | Name of the remote agent whose metadata was loaded |
+
+```rust
+// Receive metadata bytes from remote agent
+let remote_name = agent.load_remote_md(&remote_metadata_bytes)?;
+println!("Loaded metadata for: {}", remote_name);
+```
+
+### invalidate_remote_md
+
+Invalidate a specific remote agent's cached metadata.
+
+**C++ equivalent:** [`invalidateRemoteMD`](./cpp-api#invalidateremotemd)
+
+```rust
+pub fn invalidate_remote_md(&self, remote_agent: &str) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `&str` | Name of the remote agent to invalidate |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+### invalidate_all_remotes
+
+Invalidate all cached remote metadata.
+
+```rust
+pub fn invalidate_all_remotes(&self) -> Result<(), NixlError>
+```
+
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+## Metadata -- Direct Channel
+
+
+For a complete workflow example, see [Quick Start -- Metadata Exchange](../getting-started/quick-start#metadata-exchange).
+
+
+The direct channel uses etcd as a shared key-value store for automatic metadata discovery. Agents publish their metadata to etcd and fetch remote agent metadata by name, without needing an application-level transport.
+
+### send_local_md
+
+Publish this agent's metadata to etcd for discovery by other agents.
+
+**C++ equivalent:** [`sendLocalMD`](./cpp-api#sendlocalmd)
+
+```rust
+pub fn send_local_md(&self, opt_args: Option<&OptArgs>) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `opt_args` | `Option<&OptArgs>` | Optional. Use `set_ip_addr()` and `set_port()` for etcd connection. |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+```rust
+let mut opts = OptArgs::new()?;
+opts.set_ip_addr("127.0.0.1")?;
+opts.set_port(2379)?;
+agent.send_local_md(Some(&opts))?;
+```
+
+### send_local_partial_md
+
+Publish partial metadata for specific registered memory regions to etcd.
+
+**C++ equivalent:** [`sendLocalPartialMD`](./cpp-api#sendlocalpartialmd)
+
+```rust
+pub fn send_local_partial_md(
+ &self,
+ descs: &RegDescList,
+ opt_args: Option<&OptArgs>,
+) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `&RegDescList` | Registration descriptor list |
+| `opt_args` | `Option<&OptArgs>` | Optional. etcd connection settings. |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+### fetch_remote_md
+
+Fetch a remote agent's metadata from etcd. Once fetched, the metadata is loaded and cached locally, enabling communication with the remote agent.
+
+**C++ equivalent:** [`fetchRemoteMD`](./cpp-api#fetchremotemd)
+
+```rust
+pub fn fetch_remote_md(
+ &self,
+ remote_name: &str,
+ opt_args: Option<&OptArgs>,
+) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_name` | `&str` | Name of the remote agent to fetch metadata for |
+| `opt_args` | `Option<&OptArgs>` | Optional. etcd connection settings. |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+```rust
+agent.fetch_remote_md("remote_agent", Some(&opts))?;
+```
+
+### invalidate_local_md
+
+Remove this agent's metadata from etcd, signaling to other agents that it is no longer available.
+
+**C++ equivalent:** [`invalidateLocalMD`](./cpp-api#invalidatelocalmd)
+
+```rust
+pub fn invalidate_local_md(&self, opt_args: Option<&OptArgs>) -> Result<(), NixlError>
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `opt_args` | `Option<&OptArgs>` | Optional. etcd connection settings. |
+| **Returns** | `Result<(), NixlError>` | Success or error |
+
+### check_remote_metadata
+
+Check if a remote agent's metadata is available and optionally if specific descriptors can be found.
+
+**C++ equivalent:** [`checkRemoteMD`](./cpp-api#checkremotemd)
+
+```rust
+pub fn check_remote_metadata(
+ &self,
+ remote_agent: &str,
+ descs: Option<&XferDescList>,
+) -> bool
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `&str` | Name of the remote agent to check |
+| `descs` | `Option<&XferDescList>` | Optional descriptor list to validate against remote metadata |
+| **Returns** | `bool` | `true` if metadata is available (and descriptors found if provided) |
+
+```rust
+if agent.check_remote_metadata("remote_agent", None) {
+ println!("Remote agent is available");
+}
+```
diff --git a/docs/assets/NVIDIA_dark.svg b/docs/assets/NVIDIA_dark.svg
new file mode 100644
index 0000000000..f11ba3f01a
--- /dev/null
+++ b/docs/assets/NVIDIA_dark.svg
@@ -0,0 +1,35 @@
+
\ No newline at end of file
diff --git a/docs/assets/NVIDIA_light.svg b/docs/assets/NVIDIA_light.svg
new file mode 100644
index 0000000000..e1de898023
--- /dev/null
+++ b/docs/assets/NVIDIA_light.svg
@@ -0,0 +1,34 @@
+
\ No newline at end of file
diff --git a/docs/assets/NVIDIA_symbol.svg b/docs/assets/NVIDIA_symbol.svg
new file mode 100644
index 0000000000..130a7bcaa5
--- /dev/null
+++ b/docs/assets/NVIDIA_symbol.svg
@@ -0,0 +1,22 @@
+
\ No newline at end of file
diff --git a/docs/assets/fonts/NVIDIASans.woff2 b/docs/assets/fonts/NVIDIASans.woff2
new file mode 100644
index 0000000000..ce9b46080b
Binary files /dev/null and b/docs/assets/fonts/NVIDIASans.woff2 differ
diff --git a/docs/assets/fonts/NVIDIASans_Italic.woff2 b/docs/assets/fonts/NVIDIASans_Italic.woff2
new file mode 100644
index 0000000000..0b71c1f1cd
Binary files /dev/null and b/docs/assets/fonts/NVIDIASans_Italic.woff2 differ
diff --git a/docs/assets/fonts/RobotoMono.woff2 b/docs/assets/fonts/RobotoMono.woff2
new file mode 100644
index 0000000000..568f26a0ee
Binary files /dev/null and b/docs/assets/fonts/RobotoMono.woff2 differ
diff --git a/docs/data/stack-data.json b/docs/data/stack-data.json
new file mode 100644
index 0000000000..2ed6752684
--- /dev/null
+++ b/docs/data/stack-data.json
@@ -0,0 +1,124 @@
+[
+ {
+ "name": "AZURE_BLOB",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "OBJ"
+ ],
+ "sourceFile": "src/plugins/azure_blob/azure_blob_plugin.cpp",
+ "category": "object-storage"
+ },
+ {
+ "name": "GDS",
+ "version": "0.1.1",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM",
+ "FILE"
+ ],
+ "sourceFile": "src/plugins/cuda_gds/gds_plugin.cpp",
+ "category": "storage"
+ },
+ {
+ "name": "GDS_MT",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM",
+ "FILE"
+ ],
+ "sourceFile": "src/plugins/gds_mt/gds_mt_plugin.cpp",
+ "category": "storage"
+ },
+ {
+ "name": "DOCA GPUNetIO",
+ "version": "0.2.0",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM"
+ ],
+ "sourceFile": "src/plugins/gpunetio/gpunetio_plugin.cpp",
+ "category": "gpu-direct"
+ },
+ {
+ "name": "GUSLI",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "BLK",
+ "DRAM"
+ ],
+ "sourceFile": "src/plugins/gusli/gusli_plugin.cpp",
+ "category": "block-storage"
+ },
+ {
+ "name": "HF3FS",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "FILE",
+ "DRAM"
+ ],
+ "sourceFile": "src/plugins/hf3fs/hf3fs_plugin.cpp",
+ "category": "storage"
+ },
+ {
+ "name": "Libfabric",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM"
+ ],
+ "sourceFile": "src/plugins/libfabric/libfabric_plugin.cpp",
+ "category": "network"
+ },
+ {
+ "name": "MOONCAKE",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM"
+ ],
+ "sourceFile": "src/plugins/mooncake/mooncake_plugin.cpp",
+ "category": "network"
+ },
+ {
+ "name": "OBJ",
+ "version": "0.10.0",
+ "memoryTypes": [
+ "DRAM",
+ "OBJ"
+ ],
+ "sourceFile": "src/plugins/obj/obj_plugin.cpp",
+ "category": "object-storage"
+ },
+ {
+ "name": "POSIX",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "FILE"
+ ],
+ "sourceFile": "src/plugins/posix/posix_plugin.cpp",
+ "category": "storage"
+ },
+ {
+ "name": "UCCL",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM"
+ ],
+ "sourceFile": "src/plugins/uccl/uccl_plugin.cpp",
+ "category": "network"
+ },
+ {
+ "name": "UCX",
+ "version": "0.1.0",
+ "memoryTypes": [
+ "DRAM",
+ "VRAM"
+ ],
+ "sourceFile": "src/plugins/ucx/ucx_plugin.cpp",
+ "category": "network"
+ }
+]
diff --git a/docs/data/stack-data.schema.json b/docs/data/stack-data.schema.json
new file mode 100644
index 0000000000..4c099eb9df
--- /dev/null
+++ b/docs/data/stack-data.schema.json
@@ -0,0 +1,41 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "stack-data.schema.json",
+ "title": "NIXL Plugin Stack Data",
+ "description": "Auto-generated metadata for NIXL backend plugins",
+ "type": "array",
+ "items": {
+ "type": "object",
+ "required": ["name", "version", "memoryTypes", "sourceFile", "category"],
+ "additionalProperties": false,
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Plugin display name as defined in C++ source"
+ },
+ "version": {
+ "type": "string",
+ "pattern": "^\\d+\\.\\d+\\.\\d+$",
+ "description": "Plugin version string (semver)"
+ },
+ "memoryTypes": {
+ "type": "array",
+ "items": {
+ "type": "string",
+ "enum": ["DRAM", "VRAM", "FILE", "OBJ", "BLK"]
+ },
+ "minItems": 1,
+ "description": "Supported memory segment types"
+ },
+ "sourceFile": {
+ "type": "string",
+ "description": "Relative path to plugin source file from repo root"
+ },
+ "category": {
+ "type": "string",
+ "enum": ["network", "storage", "gpu-direct", "object-storage", "block-storage"],
+ "description": "Functional category for diagram grouping"
+ }
+ }
+ }
+}
diff --git a/docs/development/building-a-backend-plugin.md b/docs/development/building-a-backend-plugin.md
new file mode 100644
index 0000000000..bc398fa699
--- /dev/null
+++ b/docs/development/building-a-backend-plugin.md
@@ -0,0 +1,457 @@
+---
+title: Building a Backend Plugin
+description: Step-by-step guide to implementing a custom NIXL backend plug-in using the POSIX plug-in as a reference.
+---
+
+Build a custom NIXL backend plug-in from scratch, using the POSIX plug-in as a teaching example. A backend plug-in implements the Southbound API (SB API) to add support for a new transfer mechanism -- whether that is a networking transport, a GPU-direct storage interface, or a local file I/O system.
+
+Plug-in development is C++ only. All backend plug-ins inherit from `nixlBackendEngine` (defined in `src/api/cpp/backend/backend_engine.h`) and override the required virtual methods. The Plugin Manager handles loading your plug-in at runtime, discovering it from a shared library that exports a standard entry point. For the complete method-by-method SB API reference, see the [SB API Reference](./sb-api-reference). For the user-facing API that triggers these backend calls, see the [C++ API Reference](../api-reference/cpp-api).
+
+First, understand how user-facing (NB) API calls map to the backend (SB) API methods that your plug-in implements.
+
+## NB-to-SB API Mapping
+
+When a user interacts with a `nixlAgent`, the Transfer Agent translates those high-level operations into SB API calls on the appropriate backend plug-in. The following table shows this mapping:
+
+| User Action (NB API) | SB API Call(s) |
+|---|---|
+| `agent.createBackend("POSIX", params)` | `nixl_plugin_init()` then `create_engine(init_params)` |
+| `agent.registerMem(descs, backend)` | `backend.registerMem(mem, nixl_mem, out)` per descriptor |
+| `agent.getLocalMD(blob)` | `backend.getPublicData(meta, str)` per registered memory |
+| `agent.loadRemoteMD(blob)` | `backend.loadRemoteConnInfo(...)` + `backend.loadRemoteMD(...)` |
+| `agent.postXferReq(req)` | `backend.prepXfer(...)` then `backend.postXfer(...)` |
+| `agent.getXferStatus(req)` | `backend.checkXfer(handle)` |
+| `agent.releaseXferReq(req)` | `backend.releaseReqH(handle)` |
+| `agent.deregisterMem(descs, backend)` | `backend.deregisterMem(meta)` per descriptor |
+| `agent.getNotifs(notifs)` | `backend.getNotifs(notif_list)` per backend |
+| `agent.connect(remote)` | `backend.connect(remote_agent)` per common backend |
+
+
+See the [C++ API Reference](../api-reference/cpp-api) for full NB API documentation and the [Quick Start](../getting-started/quick-start) guide for the user workflow.
+
+
+Users call the NB API, the Transfer Agent routes calls to the appropriate backend, and your plug-in handles the data transfer through SB API methods.
+
+## The POSIX Plugin
+
+The POSIX plug-in is the simplest backend in NIXL, making it an ideal learning example. It provides local file I/O using asynchronous POSIX interfaces (io_uring, Linux AIO, or POSIX AIO depending on system availability).
+
+**Capabilities:**
+
+| Property | Value |
+|----------|-------|
+| `supportsLocal` | `true` -- file I/O is local to the agent |
+| `supportsRemote` | `false` -- no inter-agent communication |
+| `supportsNotif` | `false` -- no notification support |
+| Supported memory types | `FILE_SEG`, `DRAM_SEG` |
+| Plugin name | `"POSIX"` |
+| Plugin version | `"0.1.0"` |
+
+The POSIX plug-in is ideal for learning because it is a storage backend that only needs local transfer methods. Methods like `connect()`, `disconnect()`, and `getPublicData()` simply return `NIXL_SUCCESS` or are not needed at all, letting you focus on the core transfer lifecycle: registering memory, preparing transfers, submitting I/O, checking completion, and cleaning up. The POSIX plug-in has only 6-7 substantive method implementations, compared to a full network plug-in like UCX which implements all methods including remote metadata, notifications, and inter-agent connections.
+
+## Key Method Implementations
+
+This section walks through the 7 key methods of the POSIX plug-in. For each method, the actual source code is shown with annotations explaining the patterns and decisions. These are the methods you would customize when building your own backend.
+
+### Constructor
+
+The constructor initializes the backend engine with user-provided parameters. For the POSIX plug-in, this means selecting the async I/O queue type based on custom parameters.
+
+```cpp
+nixlPosixEngine::nixlPosixEngine(const nixlBackendInitParams *init_params)
+ : nixlBackendEngine(init_params),
+ io_queue_type_(getIoQueueType(init_params->customParams)),
+ io_queue_(nixlPosixIOQueue::instantiate(io_queue_type_,
+ getIOSPoolSize(init_params->customParams),
+ getKernelQueueSize(init_params->customParams))),
+ io_queue_lock_(init_params->syncMode) {
+ if (io_queue_type_.empty()) {
+ initErr = true;
+ NIXL_ERROR << "Failed to initialize POSIX backend - no supported io queue type found";
+ return;
+ }
+ NIXL_INFO << absl::StrFormat("POSIX backend initialized using io queue type: %s",
+ io_queue_type_);
+}
+```
+
+**Key patterns:**
+- Always call the base class constructor `nixlBackendEngine(init_params)` first -- it initializes `localAgent`, `backendType`, `customParams`, and `enableTelemetry_`.
+- Read custom parameters from `init_params->customParams` (a `std::map`) to configure backend-specific behavior. Here the POSIX plug-in reads `use_aio`, `use_uring`, and `use_posix_aio` parameters.
+- Set `initErr = true` if initialization fails. The Plugin Manager checks this via `getInitErr()` and will reject the backend if it reports an error.
+- Allocate internal resources (here, the I/O queue) during construction so they are ready for transfer operations.
+
+### registerMem
+
+Registers a memory region with the backend. The agent calls this once per descriptor when the user registers memory.
+
+```cpp
+nixl_status_t
+nixlPosixEngine::registerMem(const nixlBlobDesc &mem,
+ const nixl_mem_t &nixl_mem,
+ nixlBackendMD *&out) {
+ auto supported_mems = getSupportedMems();
+ if (std::find(supported_mems.begin(), supported_mems.end(), nixl_mem) != supported_mems.end())
+ return NIXL_SUCCESS;
+
+ return NIXL_ERR_NOT_SUPPORTED;
+}
+```
+
+**Key patterns:**
+- This is the simplest possible registration implementation -- it validates that the memory type is supported and does nothing else. The `out` parameter is left as `nullptr` since the POSIX plug-in does not need backend-specific metadata for registered memory.
+- More complex backends (like UCX) would create a metadata object, store it in `out`, and later use it during transfer operations for memory keys, handles, or registration tokens.
+- Return `NIXL_SUCCESS` to accept the memory or `NIXL_ERR_NOT_SUPPORTED` to reject it.
+
+### loadLocalMD
+
+Produces target-side metadata for local (within-agent) transfers. Some backends need different metadata for the initiator and target sides of a transfer.
+
+```cpp
+nixl_status_t
+loadLocalMD(nixlBackendMD *input, nixlBackendMD *&output) override {
+ output = input;
+ return NIXL_SUCCESS;
+}
+```
+
+**Key patterns:**
+- For local-only storage backends, this can be trivial: just return the input pointer as the output. This means the same metadata object is used for both initiator and target sides.
+- Network backends that support local transfers (like UCX) might create a separate metadata object here for target-side buffer access.
+- This method is only called when `supportsLocal()` returns `true`.
+
+### prepXfer
+
+Prepares a transfer by validating parameters and creating a backend-specific request handle.
+
+```cpp
+nixl_status_t
+nixlPosixEngine::prepXfer(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH *&handle,
+ const nixl_opt_b_args_t *opt_args) const {
+ if (!isValidPrepXferParams(operation, local, remote, remote_agent, localAgent)) {
+ return NIXL_ERR_INVALID_PARAM;
+ }
+
+ try {
+ auto posix_handle =
+ std::make_unique(operation, local, remote, opt_args, io_queue_);
+ NIXL_LOCK_GUARD(io_queue_lock_);
+ nixl_status_t status = posix_handle->prepXfer();
+ if (status != NIXL_SUCCESS) {
+ return status;
+ }
+
+ handle = posix_handle.release();
+ return NIXL_SUCCESS;
+ }
+ catch (const nixlPosixBackendReqH::exception &e) {
+ NIXL_ERROR << absl::StrFormat("Error: %s", e.what());
+ return e.code();
+ }
+ catch (const std::exception &e) {
+ NIXL_ERROR << absl::StrFormat("Unexpected error: %s", e.what());
+ return NIXL_ERR_BACKEND;
+ }
+}
+```
+
+**Key patterns:**
+- Validate parameters before doing any work. The POSIX plug-in checks that local descriptors are DRAM, remote descriptors are FILE, counts match, and the remote agent is actually the local agent (since POSIX is local-only).
+- Create a backend-specific request handle class (here `nixlPosixBackendReqH`) that inherits from `nixlBackendReqH`. This handle stores all state needed for the transfer.
+- Use `std::make_unique` for exception safety, then call `release()` to transfer ownership to the caller via the `handle` output parameter.
+- Wrap handle creation in try/catch to convert exceptions into `nixl_status_t` error codes.
+
+### postXfer
+
+Submits the transfer for execution. The agent calls this after `prepXfer` to actually initiate the I/O operations.
+
+```cpp
+nixl_status_t
+nixlPosixEngine::postXfer(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH *&handle,
+ const nixl_opt_b_args_t *opt_args) const {
+ try {
+ auto &posix_handle = castPosixHandle(handle);
+ NIXL_LOCK_GUARD(io_queue_lock_);
+ nixl_status_t status = posix_handle.postXfer();
+ if (status != NIXL_IN_PROG) {
+ NIXL_ERROR << "Error in submitting queue";
+ }
+ return status;
+ }
+ catch (const nixlPosixBackendReqH::exception &e) {
+ NIXL_ERROR << e.what();
+ return e.code();
+ }
+ return NIXL_ERR_BACKEND;
+}
+```
+
+**Key patterns:**
+- Cast the generic `nixlBackendReqH*` handle back to your backend-specific type using `dynamic_cast` (the `castPosixHandle` helper validates and casts).
+- Delegate the actual I/O submission to the request handle object. The POSIX handle iterates over descriptors and enqueues each I/O operation to the async I/O queue.
+- Return `NIXL_IN_PROG` on successful submission to indicate the transfer is now in progress and the caller should poll with `checkXfer`.
+- Thread safety is managed with `NIXL_LOCK_GUARD` since multiple transfers might share the same I/O queue.
+
+### checkXfer
+
+Polls for transfer completion. The agent calls this repeatedly until the transfer finishes.
+
+```cpp
+nixl_status_t
+nixlPosixEngine::checkXfer(nixlBackendReqH *handle) const {
+ try {
+ auto &posix_handle = castPosixHandle(handle);
+ NIXL_LOCK_GUARD(io_queue_lock_);
+ return posix_handle.checkXfer();
+ }
+ catch (const nixlPosixBackendReqH::exception &e) {
+ NIXL_ERROR << e.what();
+ return e.code();
+ }
+ return NIXL_ERR_BACKEND;
+}
+```
+
+**Key patterns:**
+- Return `NIXL_SUCCESS` when all I/O operations have completed, `NIXL_IN_PROG` when the transfer is still running, or a negative error code on failure.
+- The POSIX handle internally tracks the number of confirmed I/O completions against the total queue depth. Each call to `checkXfer` polls the I/O queue for new completions.
+- This method should be lightweight since the agent may call it in a tight polling loop.
+
+### releaseReqH
+
+Releases the backend-specific request handle and frees associated resources.
+
+```cpp
+nixl_status_t
+nixlPosixEngine::releaseReqH(nixlBackendReqH *handle) const {
+ try {
+ auto &posix_handle = castPosixHandle(handle);
+ posix_handle.~nixlPosixBackendReqH();
+ return NIXL_SUCCESS;
+ }
+ catch (const nixlPosixBackendReqH::exception &e) {
+ NIXL_ERROR << e.what();
+ return e.code();
+ }
+ return NIXL_ERR_BACKEND;
+}
+```
+
+**Key patterns:**
+- Cast the handle and explicitly call its destructor to clean up resources. The agent allocated the handle in `prepXfer`, and this is where it gets freed.
+- Always return `NIXL_SUCCESS` on successful cleanup, even if there is nothing to free. The agent relies on this to know it can safely discard the handle pointer.
+- This is called after the transfer completes (or is aborted), so the handle should not have any pending I/O at this point.
+
+
+Methods like `connect()`, `disconnect()`, and `deregisterMem()` simply return `NIXL_SUCCESS` in the POSIX plug-in since file I/O does not require remote connections or special deregistration. When building a network backend (like UCX), these methods would manage transport-level connections and memory deregistration with the networking stack.
+
+
+## Plugin Manager API
+
+Every backend plug-in must expose a set of entry points so the Plugin Manager can discover its name, version, supported memory types, and how to create and destroy engine instances. These are defined in the `nixlBackendPlugin` struct (from `src/api/cpp/backend/backend_plugin.h`).
+
+The `nixlBackendPluginCreator` template handles all of this automatically. You provide your engine class as the template parameter, and the template generates the required function pointers.
+
+### Entry Point Methods
+
+| Method | Description | How `nixlBackendPluginCreator` handles it |
+|--------|-------------|-------------------------------------------|
+| `get_plugin_name` | Returns the plug-in's display name (e.g., `"POSIX"`) | Stores the name string passed to `create()` and returns it via a lambda |
+| `get_plugin_version` | Returns the plug-in's version string (e.g., `"0.1.0"`) | Stores the version string passed to `create()` and returns it via a lambda |
+| `create_engine` | Creates a new engine instance from init params | Calls `new EngineType(init_params)` (or uses a factory for UCX) |
+| `destroy_engine` | Destroys an engine instance | Calls `delete engine` |
+| `get_backend_mems` | Returns the list of supported memory types | Stores the `nixl_mem_list_t` passed to `create()` and returns it via a lambda |
+| `get_backend_options` | Returns the custom parameter names the backend accepts | Stores the `nixl_b_params_t` passed to `create()` and returns it via a lambda |
+
+### API Version
+
+The `nixlBackendPlugin` struct includes an `api_version` field, currently set to `NIXL_PLUGIN_API_VERSION` (value: `1`). This version number ensures forward and backward compatibility: the Plugin Manager can check the API version before calling any function pointers, allowing NIXL to evolve the plug-in interface without breaking existing plug-ins.
+
+
+Always pass `NIXL_PLUGIN_API_VERSION` when creating your plug-in. Do not hardcode a numeric value -- the macro ensures your plug-in tracks the current API version at compile time.
+
+
+## Plugin Entry Points
+
+Your plug-in shared library must export two C-linkage functions that the Plugin Manager calls to initialize and clean up the plug-in. The POSIX plug-in's entry point file (`posix_plugin.cpp`) shows the complete pattern:
+
+```cpp
+#include
+#include "posix_backend.h"
+#include "backend/backend_plugin.h"
+
+// Plugin type alias for convenience
+using posix_plugin_t = nixlBackendPluginCreator;
+
+#ifdef STATIC_PLUGIN_POSIX
+nixlBackendPlugin *
+createStaticPOSIXPlugin() {
+ return posix_plugin_t::create(
+ NIXL_PLUGIN_API_VERSION, "POSIX", "0.1.0", {}, {DRAM_SEG, FILE_SEG});
+}
+#else
+extern "C" NIXL_PLUGIN_EXPORT nixlBackendPlugin *
+nixl_plugin_init() {
+ return posix_plugin_t::create(
+ NIXL_PLUGIN_API_VERSION, "POSIX", "0.1.0", {}, {DRAM_SEG, FILE_SEG});
+}
+
+extern "C" NIXL_PLUGIN_EXPORT void
+nixl_plugin_fini() {}
+#endif
+```
+
+**Key elements:**
+
+- **Type alias:** `using posix_plugin_t = nixlBackendPluginCreator` creates a convenience alias for the plug-in creator template specialized with your engine class.
+
+- **`nixl_plugin_init()`:** This is the main entry point called by the Plugin Manager when loading the plug-in. It must return a `nixlBackendPlugin*` created via the `nixlBackendPluginCreator::create()` method. Pass your plug-in name, version, backend options (empty `{}` if none), and supported memory types. Both `extern "C"` (to prevent C++ name mangling) and `NIXL_PLUGIN_EXPORT` (to set symbol visibility) are required.
+
+- **`nixl_plugin_fini()`:** Called when the plug-in is unloaded. For most plug-ins this is empty, but you can use it to clean up global resources if your plug-in allocates any during `nixl_plugin_init()`.
+
+- **Static plug-in variant:** The `#ifdef STATIC_PLUGIN_POSIX` block provides `createStaticPOSIXPlugin()` for compile-time plug-in inclusion. Static plug-ins are linked directly into the NIXL binary rather than loaded from disk, providing slightly better performance at the expense of a larger binary. The function signature differs (no `extern "C"` or `NIXL_PLUGIN_EXPORT` needed) but the creation call is identical.
+
+
+For your custom plug-in, replace `nixlPosixEngine` with your engine class, update the plug-in name and version strings, and list your supported memory types. The `nixlBackendPluginCreator` template handles all the boilerplate.
+
+
+## Build Integration
+
+NIXL uses Meson as its build system. Each plug-in is built as either a shared library (for dynamic loading) or a static library (for compile-time inclusion). The POSIX plug-in's `meson.build` shows the standard pattern:
+
+```meson
+plugin_deps = [nixl_infra, nixl_common_dep, file_utils_interface]
+
+# Define base source files
+posix_sources = [
+ 'posix_backend.cpp',
+ 'posix_backend.h',
+ 'posix_plugin.cpp',
+ 'io_queue.h',
+ 'io_queue.cpp'
+]
+
+compile_defs = []
+plugin_link_args = []
+
+# Conditional dependencies (plugin-specific)
+if has_linux_aio
+ compile_defs += ['-DHAVE_LINUXAIO']
+ posix_sources += ['linux_aio_io_queue.cpp']
+ plugin_deps += [ linux_aio_dep ]
+endif
+
+if has_io_uring
+ compile_defs += ['-DHAVE_LIBURING']
+ posix_sources += ['io_uring_io_queue.cpp']
+ plugin_deps += [ io_uring_dep ]
+endif
+
+if 'POSIX' in static_plugins
+ posix_backend_lib = static_library('POSIX',
+ posix_sources,
+ dependencies: plugin_deps,
+ link_args: plugin_link_args,
+ cpp_args: compile_defs + compile_flags,
+ include_directories: [nixl_inc_dirs, utils_inc_dirs],
+ install: false,
+ name_prefix: 'libplugin_')
+else
+ posix_backend_lib = shared_library('POSIX',
+ posix_sources,
+ dependencies: plugin_deps,
+ link_args: plugin_link_args,
+ cpp_args: compile_defs + ['-fPIC'],
+ include_directories: [nixl_inc_dirs, utils_inc_dirs],
+ install: true,
+ name_prefix: 'libplugin_',
+ install_dir: plugin_install_dir,
+ install_rpath: '$ORIGIN/..')
+endif
+
+posix_backend_interface = declare_dependency(link_with: posix_backend_lib)
+```
+
+**Key settings for your plug-in:**
+
+- **`shared_library('POSIX', ...)`** -- The first argument is the library's base name. Combined with `name_prefix: 'libplugin_'`, this produces `libplugin_POSIX.so` on Linux. Replace `'POSIX'` with your plug-in name.
+
+- **`static_library('POSIX', ...)`** -- Built when your plug-in name appears in the `static_plugins` build option. Static plug-ins are linked into the NIXL binary at compile time. Note `install: false` since they are embedded in the main library.
+
+- **`name_prefix: 'libplugin_'`** -- All NIXL plug-ins use this prefix convention. The Plugin Manager searches for libraries matching this pattern when discovering plug-ins.
+
+- **`install_dir: plugin_install_dir`** -- Resolves to `{libdir}/plugins` (typically `/usr/local/lib/plugins` or similar). This is where the Plugin Manager looks for dynamic plug-ins at runtime.
+
+- **`install_rpath: '$ORIGIN/..'`** -- Sets the runtime library search path relative to the plug-in's location, allowing the plug-in to find the main NIXL shared library without requiring `LD_LIBRARY_PATH`.
+
+- **Dependencies:** Every plug-in needs `nixl_infra` and `nixl_common_dep` at minimum. Add your plug-in-specific dependencies (e.g., `io_uring_dep` for the POSIX plug-in, `ucx_dep` for the UCX plug-in).
+
+
+When creating a new plug-in, add a `meson.build` file in your plug-in's directory under `src/plugins/your_plugin/`, and register it from the parent `src/plugins/meson.build` with a `subdir('your_plugin')` call.
+
+
+## Plugin Discovery
+
+The Plugin Manager finds and loads backend plug-ins at runtime through the following process:
+
+1. **Search directories:** The Plugin Manager scans configurable directories for shared libraries. The default search path is `{libdir}/plugins` (the `plugin_install_dir` from the build system).
+
+2. **Symbol verification:** For each shared library found, the Plugin Manager attempts to resolve the `nixl_plugin_init` symbol. Only libraries that export this symbol with the correct signature are considered valid plug-ins.
+
+3. **Plug-in initialization:** The Plugin Manager calls `nixl_plugin_init()` to get the `nixlBackendPlugin` struct, which provides function pointers for creating engines, querying capabilities, and getting the plug-in name/version.
+
+4. **Lifetime management:** Valid plug-ins remain loaded in memory for the application's lifetime. The `nixl_plugin_fini()` function is called during cleanup.
+
+5. **Static plug-ins:** Plug-ins compiled with the `STATIC_PLUGIN_*` preprocessor flag are registered at build time rather than discovered at runtime. They use a dedicated creator function (e.g., `createStaticPOSIXPlugin()`) instead of dynamic symbol resolution. Static plug-ins provide slightly better performance (no dynamic loading overhead) at the expense of a larger binary and less flexibility.
+
+
+Dynamic plug-ins must be compiled as position-independent code (`-fPIC`) and must export `nixl_plugin_init` and `nixl_plugin_fini` with `extern "C"` linkage and `NIXL_PLUGIN_EXPORT` visibility. Missing any of these will cause the Plugin Manager to silently skip your plug-in.
+
+
+## Comparing Backends
+
+Different backends implement different subsets of the SB API based on their capabilities. Understanding these patterns helps when deciding which methods your plug-in needs to implement substantively versus which can simply return `NIXL_SUCCESS`.
+
+### Network Backends
+
+Network plug-ins like UCX set all three capability flags (`supportsRemote`, `supportsLocal`, `supportsNotif`). They implement all SB API methods because they need to:
+- Manage inter-agent connections (`connect`, `disconnect`)
+- Exchange remote memory identifiers (`getPublicData`, `loadRemoteMD`)
+- Handle cross-agent notifications (`getNotifs`, `genNotif`)
+- Support both local and remote transfers
+
+### Storage Backends
+
+Storage plug-ins like GDS and POSIX only set `supportsLocal` to `true`. From the Transfer Agent's perspective, all storage access is local -- even for remote distributed storage, a local client on the agent handles the communication with the storage system. This means:
+
+- No remote agent communication is needed, so `connect()` and `disconnect()` return `NIXL_SUCCESS` immediately
+- No remote metadata exchange, so `getPublicData()`, `loadRemoteConnInfo()`, and `loadRemoteMD()` are not required
+- No notification support, so `getNotifs()` and `genNotif()` are not required
+- `loadLocalMD()` can return the input pointer unchanged (identity return)
+
+The only methods that need substantive implementation are the core transfer lifecycle: `registerMem`, `deregisterMem`, `prepXfer`, `postXfer`, `checkXfer`, and `releaseReqH`.
+
+
+If you are building a storage backend, the POSIX plug-in is your closest reference. If you are building a network backend, look at the UCX plug-in in `src/plugins/ucx/` for a complete example of remote metadata exchange, connection management, and notification handling.
+
+
+### Capability Summary
+
+| Capability | Network | Storage |
+|------------|:---:|:---:|
+| `supportsLocal` | Yes | Yes |
+| `supportsRemote` | Yes | No |
+| `supportsNotif` | Yes | No |
+| Substantive methods | All | 6-7 core methods |
+| Remote metadata | Full implementation | Not needed |
+| Connection management | Full implementation | Returns `NIXL_SUCCESS` |
+
+For the complete list of which methods are required for each capability flag combination, see the [Capability Matrix](./sb-api-reference#capability-matrix) in the SB API Reference.
diff --git a/docs/development/sb-api-reference.md b/docs/development/sb-api-reference.md
new file mode 100644
index 0000000000..f03b4dad4f
--- /dev/null
+++ b/docs/development/sb-api-reference.md
@@ -0,0 +1,910 @@
+---
+title: Southbound API Reference
+description: Complete reference for the NIXL Southbound API backend engine interface.
+---
+
+The Southbound API (SB API) is the standardized interface between NIXL's Transfer Agent and backend plug-ins. Every backend plug-in must implement this interface by inheriting from the `nixlBackendEngine` base class and overriding the required virtual methods.
+
+This reference documents all 25 methods in the SB API: 13 pure virtual methods that every backend must implement, 7 conditionally required methods based on capability flags, and 5 optional methods with default implementations. For a step-by-step tutorial on building a plug-in, see [Building a Backend Plugin](./building-a-backend-plugin). For the user-facing API, see the [C++ API Reference](../api-reference/cpp-api).
+
+## Base Class Hierarchy
+
+The SB API is defined through three base classes and one parameter struct, all declared in the `src/api/cpp/backend/` headers.
+
+### nixlBackendEngine
+
+The primary class that every backend plug-in inherits from. Defined in `backend_engine.h`.
+
+**Protected members** (accessible to child backends):
+
+| Member | Type | Description |
+|--------|------|-------------|
+| `initErr` | `bool` | Set to `true` in the constructor if initialization fails. Defaults to `false`. |
+| `localAgent` | `const std::string` | Name of the local agent that owns this backend instance. Read-only. |
+| `enableTelemetry_` | `const bool` | Whether telemetry is enabled for this backend. Read-only. |
+
+**Protected helper methods:**
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `setInitParam` | `nixl_status_t setInitParam(const std::string &key, const std::string &value)` | Store a custom parameter. Returns `NIXL_ERR_NOT_ALLOWED` if the key already exists. |
+| `getInitParam` | `nixl_status_t getInitParam(const std::string &key, std::string &value) const` | Retrieve a custom parameter. Returns `NIXL_ERR_INVALID_PARAM` if the key is not found. |
+| `addTelemetryEvent` | `void addTelemetryEvent(const std::string &event_name, uint64_t value)` | Record a telemetry event. No-op if telemetry is disabled or queue is full (max 1000 events). |
+
+**Public helper methods:**
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `getInitErr` | `bool getInitErr() const noexcept` | Check whether construction failed. |
+| `getType` | `const nixl_backend_t& getType() const noexcept` | Get the backend type string (e.g., `"UCX"`, `"POSIX"`). |
+| `getCustomParams` | `const nixl_b_params_t& getCustomParams() const noexcept` | Get the key-value parameters map. |
+| `getTelemetryEvents` | `std::vector getTelemetryEvents()` | Move-return accumulated telemetry events (clears internal buffer). |
+
+**Constructor:**
+
+```cpp
+explicit nixlBackendEngine(const nixlBackendInitParams *init_params);
+```
+
+The constructor initializes `backendType`, `customParams`, `localAgent`, and `enableTelemetry_` from the provided init params. The backend is **non-copyable and non-movable**.
+
+**Destructor:**
+
+```cpp
+virtual ~nixlBackendEngine() = default;
+```
+
+The agent destroys a backend engine only after its registrations have been deregistered and its transfer request handles have been released. The backend destructor must still clean up any backend-owned resources that remain, such as connections, registered-memory state, and I/O queues; it should not depend on every earlier cleanup callback having completed successfully.
+
+### nixlBackendMD
+
+Base class for backend metadata objects. Both registered memory metadata and remote identifier metadata inherit from this class. Defined in `backend_aux.h`.
+
+**Protected field:**
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `isPrivateMD` | `bool` | Distinguishes private (registered memory) from public (remote identifier) metadata. |
+
+**Constructor:**
+
+```cpp
+nixlBackendMD(bool isPrivate);
+```
+
+**Destructor:**
+
+```cpp
+virtual ~nixlBackendMD();
+```
+
+
+The `nixlBackendMD` pointer is opaque to the NIXL agent. The agent stores it during `registerMem()` and passes it back during transfer operations. Your backend is responsible for casting it to your concrete metadata subclass.
+
+
+### nixlBackendReqH
+
+Base class for transfer request handles. Backend-specific request state inherits from this class. Defined in `backend_aux.h`.
+
+**Constructor:**
+
+```cpp
+nixlBackendReqH();
+```
+
+**Destructor:**
+
+```cpp
+virtual ~nixlBackendReqH();
+```
+
+The agent creates request handles via `prepXfer()` and destroys them via `releaseReqH()`. Your backend stores whatever state it needs for tracking in-progress transfers in your subclass.
+
+### nixlBackendInitParams
+
+Parameter container passed to the backend constructor. Not inherited -- this is a plain data class. Defined in `backend_aux.h`.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `localAgent` | `std::string` | Name of the agent creating this backend. |
+| `type` | `nixl_backend_t` | Backend type identifier (e.g., `"UCX"`, `"POSIX"`). |
+| `customParams` | `nixl_b_params_t*` | Pointer to the key-value parameter map. |
+| `enableProgTh` | `bool` | Whether the progress thread is enabled. |
+| `pthrDelay` | `nixlTime::us_t` | Progress thread delay between iterations (microseconds). |
+| `syncMode` | `nixl_thread_sync_t` | Thread synchronization mode. |
+| `enableTelemetry_` | `bool` | Whether telemetry collection is enabled. |
+
+## Capability Matrix
+
+The following table shows which methods are required based on the capability flags your backend reports. Methods marked under "Always Required" must be implemented by every backend. Methods under capability columns are required only when that capability returns `true`.
+
+| Method | Always Required | supportsLocal | supportsRemote | supportsNotif |
+|--------|:-:|:-:|:-:|:-:|
+| Constructor / Destructor | x | | | |
+| `supportsRemote()` | x | | | |
+| `supportsLocal()` | x | | | |
+| `supportsNotif()` | x | | | |
+| `getSupportedMems()` | x | | | |
+| `registerMem()` | x | | | |
+| `deregisterMem()` | x | | | |
+| `connect()` | x | | | |
+| `disconnect()` | x | | | |
+| `unloadMD()` | x | | | |
+| `prepXfer()` | x | | | |
+| `postXfer()` | x | | | |
+| `checkXfer()` | x | | | |
+| `releaseReqH()` | x | | | |
+| `loadLocalMD()` | | x | | |
+| `getPublicData()` | | | x | |
+| `getConnInfo()` | | | x | |
+| `loadRemoteConnInfo()` | | | x | |
+| `loadRemoteMD()` | | | x | |
+| `getNotifs()` | | | | x |
+| `genNotif()` | | | | x |
+| `prepMemView()` (2 overloads) | | | | |
+| `releaseMemView()` | | | | |
+| `queryMem()` | | | | |
+| `estimateXferCost()` | | | | |
+
+The last 5 methods are truly optional and have default implementations that return `NIXL_ERR_NOT_SUPPORTED` or no-op.
+
+
+A network backend (e.g., UCX) should set `supportsRemote` and `supportsNotif` to `true`, and preferably `supportsLocal` as well so that another backend is not needed for local transfers. A storage backend (e.g., GDS, POSIX) should set `supportsLocal` to `true`; `supportsNotif` is optional.
+
+
+## Descriptor Types
+
+The SB API uses several descriptor types for memory registration and transfer operations. These are defined across `nixl_descriptors.h` and `backend_aux.h`.
+
+### nixlBlobDesc
+
+Registration descriptor used by `registerMem()` and `loadRemoteMD()`. Extends `nixlBasicDesc` with a metadata blob.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `addr` | `uintptr_t` | Start address of the buffer (or offset for file/object) |
+| `len` | `size_t` | Length of the buffer in bytes |
+| `devId` | `uint64_t` | Device ID, block ID, or file descriptor |
+| `metaInfo` | `nixl_blob_t` | Optional metadata string (e.g., file path, bucket ID) |
+
+### nixlMetaDesc
+
+Transfer descriptor used within `nixl_meta_dlist_t`. Extends `nixlBasicDesc` with a backend metadata pointer.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `addr` | `uintptr_t` | Start address of the buffer |
+| `len` | `size_t` | Length of the buffer in bytes |
+| `devId` | `uint64_t` | Device ID |
+| `metadataP` | `nixlBackendMD*` | Pointer to backend metadata for this descriptor |
+
+### nixlRemoteMetaDesc
+
+Remote transfer descriptor. Extends `nixlMetaDesc` with a remote agent name.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| (inherits all from `nixlMetaDesc`) | | |
+| `remoteAgent` | `std::string` | Name of the remote agent that owns this memory |
+
+### Descriptor List Types
+
+| Type Alias | Underlying Type | Used In |
+|------------|----------------|---------|
+| `nixl_reg_dlist_t` | `nixlDescList` | `queryMem()` |
+| `nixl_meta_dlist_t` | `nixlDescList` | `prepXfer()`, `postXfer()`, `prepMemView()` (local) |
+| `nixl_remote_meta_dlist_t` | `nixlDescList` | `prepMemView()` (remote) |
+| `notif_list_t` | `std::vector>` | `getNotifs()` -- pairs of (agent name, notification) |
+| `nixl_opt_b_args_t` | `nixlBackendOptionalArgs` | `prepXfer()`, `postXfer()` |
+
+### nixl_opt_b_args_t Fields
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `notifMsg` | `nixl_blob_t` | empty | Notification message to send after transfer completes |
+| `hasNotif` | `bool` | `false` | Whether a notification should be generated |
+| `customParam` | `nixl_blob_t` | empty | Custom backend parameter string |
+
+### Descriptor Field Meanings by Memory Type
+
+| mem type | addr | len | devID | str (metaInfo) |
+|----------|------|-----|-------|-----------------|
+| DRAM | buffer address | buffer length | 0 (or region) | -- |
+| VRAM | buffer address | buffer length | GPU ID | -- |
+| BLK | block offset | block length | Volume ID | -- |
+| FILE | offset in file | length (or 0) | file descriptor | Path (+ access mode) |
+| OBJ | offset in object | length (or 0) | key | Extended key (+ bucket ID) |
+
+## Capability Indicators
+
+
+These four methods are pure virtual and must be implemented by every backend. They determine which additional methods the NIXL agent will call on your backend.
+
+
+### supportsRemote
+
+Indicates whether this backend supports data transfers across nodes (between different agents on different machines).
+
+```cpp
+virtual bool supportsRemote() const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| **Returns** | `bool` | `true` if the backend supports remote (cross-node) operations |
+
+```cpp
+bool MyBackend::supportsRemote() const {
+ return true; // This is a network backend
+}
+```
+
+### supportsLocal
+
+Indicates whether this backend supports data transfers within the same node (local or loopback operations).
+
+```cpp
+virtual bool supportsLocal() const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| **Returns** | `bool` | `true` if the backend supports local (within-node) operations |
+
+```cpp
+bool MyBackend::supportsLocal() const {
+ return true; // This backend handles local file I/O
+}
+```
+
+### supportsNotif
+
+Indicates whether this backend supports sending and receiving notifications. Related notification methods (`getNotifs`, `genNotif`) are not pure virtual and return errors if called on a backend that does not support notifications.
+
+```cpp
+virtual bool supportsNotif() const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| **Returns** | `bool` | `true` if the backend supports notifications |
+
+```cpp
+bool MyBackend::supportsNotif() const {
+ return false; // Storage backends typically don't need notifications
+}
+```
+
+### getSupportedMems
+
+Returns the list of memory types this backend can handle. The agent uses this to route transfer requests to the appropriate backend.
+
+```cpp
+virtual nixl_mem_list_t getSupportedMems() const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| **Returns** | `nixl_mem_list_t` | Vector of supported `nixl_mem_t` values (e.g., `DRAM_SEG`, `VRAM_SEG`, `FILE_SEG`) |
+
+```cpp
+nixl_mem_list_t MyBackend::getSupportedMems() const {
+ return {DRAM_SEG, FILE_SEG};
+}
+```
+
+
+Based on these flags, the required methods change. A network backend should have `supportsRemote` and `supportsNotif` return `true` (and preferably `supportsLocal` as well). A storage backend should have `supportsLocal` return `true`.
+
+
+## Memory Management
+
+
+For descriptor identity and user-facing registration behavior, see [Northbound API Semantics](../api-reference/northbound-api#registration-behavior). For the C++ methods, see [C++ API Reference -- Memory Registration](../api-reference/cpp-api#registermem).
+
+
+The agent calls `registerMem()` and `deregisterMem()` while holding its exclusive lock. A backend may rely on this lock for state accessed only by these two callbacks. If transfer callbacks or backend-owned threads access the same state, the backend must provide its own synchronization.
+
+A descriptor list may be offered to more than one compatible backend. Each backend receives an independent series of calls and must manage only its own registrations; it must not depend on the order in which other backends register or deregister the same descriptor.
+
+### registerMem
+
+Registers one contiguous memory region with the backend. If the backend needs per-registration state, it creates a metadata object and returns it through `out`. The agent stores the pointer and passes it back during transfers and deregistration.
+
+```cpp
+virtual nixl_status_t registerMem(const nixlBlobDesc &mem,
+ const nixl_mem_t &nixl_mem,
+ nixlBackendMD* &out) = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `mem` | `const nixlBlobDesc&` | Memory descriptor (address, length, device ID, optional metadata string) |
+| `nixl_mem` | `const nixl_mem_t&` | Memory type (e.g., `DRAM_SEG`, `VRAM_SEG`, `FILE_SEG`) |
+| `out` | `nixlBackendMD*&` | [out] Pointer to backend-created metadata, when the backend needs it |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_status_t MyBackend::registerMem(const nixlBlobDesc &mem,
+ const nixl_mem_t &nixl_mem,
+ nixlBackendMD* &out) {
+ auto *md = new MyBackendMD(mem, nixl_mem);
+ out = md;
+ return NIXL_SUCCESS;
+}
+```
+
+Registration is transactional within one backend. If a later descriptor or its associated metadata setup fails, the agent calls `deregisterMem()` for descriptors that the backend already accepted from that list. A failing `registerMem()` must return a non-success status without publishing a usable `out` pointer or leaving resources that require a later cleanup call.
+
+If a backend treats `devId` as a unique lookup key, it must detect an active duplicate itself. It must reject the duplicate with `NIXL_ERR_INVALID_PARAM` without changing the existing registration or partially registering the new descriptor.
+
+### deregisterMem
+
+Deregisters a previously registered memory region and frees the resources associated with its metadata pointer.
+
+```cpp
+virtual nixl_status_t deregisterMem(nixlBackendMD* meta) = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `meta` | `nixlBackendMD*` | Metadata pointer returned by `registerMem()` |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_status_t MyBackend::deregisterMem(nixlBackendMD* meta) {
+ delete static_cast(meta);
+ return NIXL_SUCCESS;
+}
+```
+
+
+The agent passes only the metadata pointer to `deregisterMem()`; it does not pass the original descriptor or its `metaInfo`. After successful deregistration, the pointer is invalid and the agent will not pass it to the backend again.
+
+
+## Connection Management
+
+### connect
+
+Initiates a connection to a remote agent (or to itself for loopback). Some backends require a self-connection for local operations. The agent may call `connect` proactively or defer it until the first transfer.
+
+```cpp
+virtual nixl_status_t connect(const std::string &remote_agent) = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the remote agent to connect to |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_status_t MyBackend::connect(const std::string &remote_agent) {
+ // Establish connection using loaded remote connection info
+ return NIXL_SUCCESS;
+}
+```
+
+### disconnect
+
+Terminates a connection with a remote agent (or self for loopback). Called during metadata invalidation or agent destruction.
+
+```cpp
+virtual nixl_status_t disconnect(const std::string &remote_agent) = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the remote agent to disconnect from |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_status_t MyBackend::disconnect(const std::string &remote_agent) {
+ // Release connection resources
+ return NIXL_SUCCESS;
+}
+```
+
+### getConnInfo
+
+Provides the serialized connection information that remote agents need to communicate with this backend. Called once after backend creation.
+
+```cpp
+virtual nixl_status_t getConnInfo(std::string &str) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `str` | `std::string&` | [out] Serialized connection data |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::getConnInfo(std::string &str) const {
+ str = serializeMyConnectionInfo();
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsRemote()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`.
+
+
+### loadRemoteConnInfo
+
+Loads connection information received from a remote agent. This does not establish the connection -- it only stores the information for later use by `connect()`.
+
+```cpp
+virtual nixl_status_t loadRemoteConnInfo(const std::string &remote_agent,
+ const std::string &remote_conn_info);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the remote agent |
+| `remote_conn_info` | `const std::string&` | Serialized connection data from the remote agent |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::loadRemoteConnInfo(const std::string &remote_agent,
+ const std::string &remote_conn_info) {
+ connInfoMap_[remote_agent] = deserialize(remote_conn_info);
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsRemote()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`.
+
+
+## Metadata Management
+
+### getPublicData
+
+Serializes the remote identifier for a registered memory region. Remote agents use this serialized data to access this memory via `loadRemoteMD()`.
+
+```cpp
+virtual nixl_status_t getPublicData(const nixlBackendMD* meta,
+ std::string &str) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `meta` | `const nixlBackendMD*` | Metadata object from `registerMem()` |
+| `str` | `std::string&` | [out] Serialized remote identifier |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::getPublicData(const nixlBackendMD* meta,
+ std::string &str) const {
+ auto *md = static_cast(meta);
+ str = md->serializeForRemote();
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsRemote()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`.
+
+
+### loadRemoteMD
+
+Deserializes a remote memory identifier received from a remote agent. Creates a metadata object that the agent uses for remote transfers.
+
+```cpp
+virtual nixl_status_t loadRemoteMD(const nixlBlobDesc &input,
+ const nixl_mem_t &nixl_mem,
+ const std::string &remote_agent,
+ nixlBackendMD* &output);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `input` | `const nixlBlobDesc&` | Descriptor with the remote memory info (address, length, device, serialized metadata in `metaInfo`) |
+| `nixl_mem` | `const nixl_mem_t&` | Memory type of the remote region |
+| `remote_agent` | `const std::string&` | Name of the remote agent |
+| `output` | `nixlBackendMD*&` | [out] Pointer to the deserialized remote metadata object |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::loadRemoteMD(const nixlBlobDesc &input,
+ const nixl_mem_t &nixl_mem,
+ const std::string &remote_agent,
+ nixlBackendMD* &output) {
+ auto *md = new MyRemoteMD(input, remote_agent);
+ output = md;
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsRemote()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`.
+
+
+### loadLocalMD
+
+Loads metadata for local (within-node) operations. For simple backends, this can return the input pointer directly. For backends that need different metadata for initiator vs. target descriptors, this creates a separate target metadata object.
+
+```cpp
+virtual nixl_status_t loadLocalMD(nixlBackendMD* input,
+ nixlBackendMD* &output);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `input` | `nixlBackendMD*` | Metadata from `registerMem()` |
+| `output` | `nixlBackendMD*&` | [out] Metadata for target-side operations (can be same as input) |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::loadLocalMD(nixlBackendMD* input,
+ nixlBackendMD* &output) {
+ output = input; // Simple case: same metadata for initiator and target
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsLocal()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`.
+
+
+### unloadMD
+
+Releases resources associated with a metadata object that was created by `loadRemoteMD()` or `loadLocalMD()`. Always required.
+
+```cpp
+virtual nixl_status_t unloadMD(nixlBackendMD* input) = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `input` | `nixlBackendMD*` | Metadata object to release |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_status_t MyBackend::unloadMD(nixlBackendMD* input) {
+ delete static_cast(input);
+ return NIXL_SUCCESS;
+}
+```
+
+
+If `loadLocalMD()` returns the same pointer as input (identity return), `unloadMD()` must handle this case without double-freeing. Check whether the pointer is the same as the registered memory metadata before deleting.
+
+
+## Transfer Operations
+
+
+For the user-facing transfer workflow, see [C++ API Reference -- Transfer Operations](../api-reference/cpp-api#postxferreq).
+
+
+The agent calls `prepXfer()`, `postXfer()`, and `checkXfer()` while holding its shared/read lock. Backends must not treat that lock as exclusive serialization. A backend must synchronize state that these callbacks or its own worker threads can modify concurrently.
+
+### prepXfer
+
+Prepares a transfer request. The descriptor lists contain the backend metadata pointers associated with the requested regions, but their order is the request order and is unrelated to registration order.
+
+The backend must return `NIXL_SUCCESS` only if it accepts the complete local and remote descriptor lists; partial preparation is not valid. On success, it returns a backend-specific request handle (`nixlBackendReqH` subclass) that stores the transfer state. This method does not start the transfer. Resources held by the request handle remain the backend's responsibility until `releaseReqH()`.
+
+```cpp
+virtual nixl_status_t prepXfer(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH* &handle,
+ const nixl_opt_b_args_t* opt_args=nullptr) const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `const nixl_xfer_op_t&` | Transfer operation (`NIXL_READ` or `NIXL_WRITE`) |
+| `local` | `const nixl_meta_dlist_t&` | Local descriptor list with metadata pointers |
+| `remote` | `const nixl_meta_dlist_t&` | Remote descriptor list with metadata pointers |
+| `remote_agent` | `const std::string&` | Name of the remote agent (or self for loopback) |
+| `handle` | `nixlBackendReqH*&` | [out] Backend-created request handle |
+| `opt_args` | `const nixl_opt_b_args_t*` | Optional arguments (notification, custom params). Defaults to `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code |
+
+```cpp
+nixl_status_t MyBackend::prepXfer(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH* &handle,
+ const nixl_opt_b_args_t* opt_args) const {
+ handle = new MyReqHandle(operation, local, remote);
+ return NIXL_SUCCESS;
+}
+```
+
+### postXfer
+
+Posts (starts) an asynchronous transfer. This method should initiate the data movement and return immediately without waiting for completion. If the transfer is very small, it may complete synchronously and return `NIXL_SUCCESS` directly.
+
+```cpp
+virtual nixl_status_t postXfer(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH* &handle,
+ const nixl_opt_b_args_t* opt_args=nullptr) const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `const nixl_xfer_op_t&` | Transfer operation (`NIXL_READ` or `NIXL_WRITE`) |
+| `local` | `const nixl_meta_dlist_t&` | Local descriptor list with metadata pointers |
+| `remote` | `const nixl_meta_dlist_t&` | Remote descriptor list with metadata pointers |
+| `remote_agent` | `const std::string&` | Name of the remote agent (or self for loopback) |
+| `handle` | `nixlBackendReqH*&` | Request handle from `prepXfer()` (may be updated) |
+| `opt_args` | `const nixl_opt_b_args_t*` | Optional arguments (notification, custom params). Defaults to `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` (complete), `NIXL_IN_PROG` (started), or error code |
+
+```cpp
+nixl_status_t MyBackend::postXfer(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH* &handle,
+ const nixl_opt_b_args_t* opt_args) const {
+ auto *req = static_cast(handle);
+ req->submit(); // Start async I/O
+ return NIXL_IN_PROG;
+}
+```
+
+### checkXfer
+
+Checks the status of an in-progress transfer. The backend may use this call to internally progress its I/O engine.
+
+```cpp
+virtual nixl_status_t checkXfer(nixlBackendReqH* handle) const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixlBackendReqH*` | Request handle from `prepXfer()` |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` (complete), `NIXL_IN_PROG` (still running), or error code |
+
+```cpp
+nixl_status_t MyBackend::checkXfer(nixlBackendReqH* handle) const {
+ auto *req = static_cast(handle);
+ return req->isComplete() ? NIXL_SUCCESS : NIXL_IN_PROG;
+}
+```
+
+### releaseReqH
+
+Releases a transfer request handle. If the transfer is still in progress, the backend should attempt to abort it. This method should be non-blocking.
+
+```cpp
+virtual nixl_status_t releaseReqH(nixlBackendReqH* handle) const = 0;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixlBackendReqH*` | Request handle to release |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or error code if abort failed |
+
+```cpp
+nixl_status_t MyBackend::releaseReqH(nixlBackendReqH* handle) const {
+ delete static_cast(handle);
+ return NIXL_SUCCESS;
+}
+```
+
+
+The method is named `releaseReqH`, not `releaseXferReq` as mentioned in some older documentation. Always use the name from `backend_engine.h`.
+
+
+
+A transfer request is prepped once but can be posted multiple times (after reaching `NIXL_SUCCESS` state each time). There is no ordering guarantee across transfer requests. The user is responsible for avoiding concurrent writes to the same memory.
+
+
+### estimateXferCost
+
+Estimates the duration and cost of a transfer operation. This method is optional and allows the agent to make informed decisions about backend selection.
+
+```cpp
+virtual nixl_status_t
+estimateXferCost(const nixl_xfer_op_t &operation,
+ const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote,
+ const std::string &remote_agent,
+ nixlBackendReqH *const &handle,
+ std::chrono::microseconds &duration,
+ std::chrono::microseconds &err_margin,
+ nixl_cost_t &method,
+ const nixl_opt_args_t *extra_params = nullptr) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `operation` | `const nixl_xfer_op_t&` | Transfer operation (`NIXL_READ` or `NIXL_WRITE`) |
+| `local` | `const nixl_meta_dlist_t&` | Local descriptor list |
+| `remote` | `const nixl_meta_dlist_t&` | Remote descriptor list |
+| `remote_agent` | `const std::string&` | Name of the remote agent |
+| `handle` | `nixlBackendReqH *const&` | Request handle from `prepXfer()` |
+| `duration` | `std::chrono::microseconds&` | [out] Estimated transfer duration |
+| `err_margin` | `std::chrono::microseconds&` | [out] Error margin on the estimate |
+| `method` | `nixl_cost_t&` | [out] Estimation method used |
+| `extra_params` | `const nixl_opt_args_t*` | Optional extra parameters. Defaults to `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_NOT_SUPPORTED` (default) |
+
+```cpp
+nixl_status_t MyBackend::estimateXferCost(
+ const nixl_xfer_op_t &operation, const nixl_meta_dlist_t &local,
+ const nixl_meta_dlist_t &remote, const std::string &remote_agent,
+ nixlBackendReqH *const &handle, std::chrono::microseconds &duration,
+ std::chrono::microseconds &err_margin, nixl_cost_t &method,
+ const nixl_opt_args_t *extra_params) const {
+ duration = std::chrono::microseconds(100); // 100us estimate
+ err_margin = std::chrono::microseconds(50);
+ method = nixl_cost_t::ANALYTICAL_BACKEND;
+ return NIXL_SUCCESS;
+}
+```
+
+## Notification Handling
+
+### getNotifs
+
+Retrieves notifications received from remote agents (or local via loopback). The agent iterates over all notification-capable backends and merges results.
+
+```cpp
+virtual nixl_status_t getNotifs(notif_list_t ¬if_list);
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `notif_list` | `notif_list_t&` | [out] Vector of (agent name, notification message) pairs |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::getNotifs(notif_list_t ¬if_list) {
+ // Drain received notifications into the list
+ for (auto ¬if : pendingNotifs_) {
+ notif_list.push_back(notif);
+ }
+ pendingNotifs_.clear();
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsNotif()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`. The backend must extract the source agent name from each received notification.
+
+
+### genNotif
+
+Generates a standalone notification to a remote agent. This notification is not bound to any transfer and does not provide ordering guarantees.
+
+```cpp
+virtual nixl_status_t genNotif(const std::string &remote_agent,
+ const std::string &msg) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `remote_agent` | `const std::string&` | Name of the target agent |
+| `msg` | `const std::string&` | Notification message payload |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_BACKEND` (default) |
+
+```cpp
+nixl_status_t MyBackend::genNotif(const std::string &remote_agent,
+ const std::string &msg) const {
+ sendNotification(remote_agent, msg);
+ return NIXL_SUCCESS;
+}
+```
+
+
+Required only if `supportsNotif()` returns `true`. The default implementation returns `NIXL_ERR_BACKEND`.
+
+
+## Optional Methods
+
+These methods have default implementations and can be overridden to add functionality. They all return `NIXL_ERR_NOT_SUPPORTED` or perform a no-op by default.
+
+### prepMemView (remote)
+
+Prepares a memory view for remote buffers. Memory views provide an alternative access pattern for reading remote memory.
+
+```cpp
+virtual nixl_status_t
+prepMemView(const nixl_remote_meta_dlist_t &descs,
+ nixlMemViewH &handle,
+ const nixl_opt_b_args_t *opt_args = nullptr) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_remote_meta_dlist_t&` | Remote memory descriptor list |
+| `handle` | `nixlMemViewH&` | [out] Memory view handle (typedef for `void*`) |
+| `opt_args` | `const nixl_opt_b_args_t*` | Optional arguments. Defaults to `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_NOT_SUPPORTED` (default) |
+
+```cpp
+nixl_status_t MyBackend::prepMemView(const nixl_remote_meta_dlist_t &descs,
+ nixlMemViewH &handle,
+ const nixl_opt_b_args_t *opt_args) const {
+ handle = createRemoteView(descs);
+ return NIXL_SUCCESS;
+}
+```
+
+### prepMemView (local)
+
+Prepares a memory view for local buffers.
+
+```cpp
+virtual nixl_status_t
+prepMemView(const nixl_meta_dlist_t &descs,
+ nixlMemViewH &handle,
+ const nixl_opt_b_args_t *opt_args = nullptr) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_meta_dlist_t&` | Local memory descriptor list |
+| `handle` | `nixlMemViewH&` | [out] Memory view handle (typedef for `void*`) |
+| `opt_args` | `const nixl_opt_b_args_t*` | Optional arguments. Defaults to `nullptr`. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_NOT_SUPPORTED` (default) |
+
+```cpp
+nixl_status_t MyBackend::prepMemView(const nixl_meta_dlist_t &descs,
+ nixlMemViewH &handle,
+ const nixl_opt_b_args_t *opt_args) const {
+ handle = createLocalView(descs);
+ return NIXL_SUCCESS;
+}
+```
+
+### releaseMemView
+
+Releases a memory view handle previously created by `prepMemView()`.
+
+```cpp
+virtual void releaseMemView(nixlMemViewH handle) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `handle` | `nixlMemViewH` | Memory view handle to release |
+| **Returns** | `void` | -- |
+
+```cpp
+void MyBackend::releaseMemView(nixlMemViewH handle) const {
+ delete static_cast(handle);
+}
+```
+
+### queryMem
+
+Queries information about a list of memory or storage descriptors. File and object backends can override this to provide metadata such as file size, existence, or permissions.
+
+```cpp
+virtual nixl_status_t
+queryMem(const nixl_reg_dlist_t &descs,
+ std::vector &resp) const;
+```
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `descs` | `const nixl_reg_dlist_t&` | Registration descriptor list to query |
+| `resp` | `std::vector&` | [out] Response vector (one entry per descriptor). Empty optional if no data available. |
+| **Returns** | `nixl_status_t` | `NIXL_SUCCESS` or `NIXL_ERR_NOT_SUPPORTED` (default) |
+
+```cpp
+nixl_status_t MyBackend::queryMem(const nixl_reg_dlist_t &descs,
+ std::vector &resp) const {
+ for (size_t i = 0; i < descs.descCount(); i++) {
+ nixl_b_params_t info;
+ info["size"] = std::to_string(getFileSize(descs[i]));
+ resp.push_back(info);
+ }
+ return NIXL_SUCCESS;
+}
+```
+
+
+All optional methods default to `NIXL_ERR_NOT_SUPPORTED` or no-op. Override them only if your backend has the functionality to support them.
+
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_01_init_dark.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_01_init_dark.d2
new file mode 100644
index 0000000000..913991df15
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_01_init_dark.d2
@@ -0,0 +1,57 @@
+# NIXL Basic Two Peers Transfer — Phase 1: Initialization (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> target: "1.1 create_agent(\"target\", port=5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> initiator: "1.2 create_agent(\"initiator\", port=0)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+target -> target: "1.3 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+initiator -> initiator: "1.4 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> target: "1.5 allocate tensor (ones, 10x16 float32)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> initiator: "1.6 allocate tensor (zeros, 10x16 float32)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+target -> target: "1.7 register_memory(target_tensor)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> initiator: "1.8 register_memory(initiator_tensor)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_01_init_light.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_01_init_light.d2
new file mode 100644
index 0000000000..01e05b6fa7
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_01_init_light.d2
@@ -0,0 +1,54 @@
+# NIXL Basic Two Peers Transfer — Phase 1: Initialization (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+conductor -> target: "1.1 create_agent(\"target\", port=5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> initiator: "1.2 create_agent(\"initiator\", port=0)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+target -> target: "1.3 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+initiator -> initiator: "1.4 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> target: "1.5 allocate tensor (ones, 10x16 float32)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> initiator: "1.6 allocate tensor (zeros, 10x16 float32)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+target -> target: "1.7 register_memory(target_tensor)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> initiator: "1.8 register_memory(initiator_tensor)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_02_metadata_dark.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_02_metadata_dark.d2
new file mode 100644
index 0000000000..e294035fef
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_02_metadata_dark.d2
@@ -0,0 +1,51 @@
+# NIXL Basic Two Peers Transfer — Phase 2: Metadata & Descriptor Exchange (Dark)
+
+shape: sequence_diagram
+
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+initiator -> target: "2.1 fetch_remote_metadata(\"target\", ip, 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> target: "2.2 send_local_metadata(ip, 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> initiator: "2.3 check_remote_metadata(\"target\") [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+target -> target: "2.4 get_xfer_descs(target_rows)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+target -> target: "2.5 get_serialized_descs(target_descs)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+target -> initiator: "2.6 send_notif(\"initiator\", serialized_descs)" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+initiator -> initiator: "2.7 get_new_notifs() [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+initiator -> initiator: "2.8 deserialize_descs(notifs[\"target\"])" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_02_metadata_light.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_02_metadata_light.d2
new file mode 100644
index 0000000000..8d5bbb5528
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_02_metadata_light.d2
@@ -0,0 +1,49 @@
+# NIXL Basic Two Peers Transfer — Phase 2: Metadata & Descriptor Exchange (Light)
+
+shape: sequence_diagram
+
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+initiator -> target: "2.1 fetch_remote_metadata(\"target\", ip, 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> target: "2.2 send_local_metadata(ip, 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> initiator: "2.3 check_remote_metadata(\"target\") [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+target -> target: "2.4 get_xfer_descs(target_rows)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+target -> target: "2.5 get_serialized_descs(target_descs)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+target -> initiator: "2.6 send_notif(\"initiator\", serialized_descs)" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+initiator -> initiator: "2.7 get_new_notifs() [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+initiator -> initiator: "2.8 deserialize_descs(notifs[\"target\"])" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_03_transfer_dark.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_03_transfer_dark.d2
new file mode 100644
index 0000000000..39b7c521e5
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_03_transfer_dark.d2
@@ -0,0 +1,55 @@
+# NIXL Basic Two Peers Transfer — Phase 3: Transfer Execution (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> initiator: "3.1 initialize_xfer(READ, local, remote, \"target\", \"Done_reading\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> initiator: "3.2 resolve backend (UCX auto-select)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> initiator: "3.3 transfer(xfer_handle)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> target: "3.4 RDMA read — data transfer (ones tensor)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> initiator: "3.5 check_xfer_state(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+initiator -> conductor: "3.6 state: DONE" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+initiator -> target: "3.7 send_notif(\"target\", \"Done_reading\")" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_03_transfer_light.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_03_transfer_light.d2
new file mode 100644
index 0000000000..c6cc76e0bc
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_03_transfer_light.d2
@@ -0,0 +1,52 @@
+# NIXL Basic Two Peers Transfer — Phase 3: Transfer Execution (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+conductor -> initiator: "3.1 initialize_xfer(READ, local, remote, \"target\", \"Done_reading\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> initiator: "3.2 resolve backend (UCX auto-select)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> initiator: "3.3 transfer(xfer_handle)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+initiator -> target: "3.4 RDMA read — data transfer (ones tensor)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> initiator: "3.5 check_xfer_state(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+initiator -> conductor: "3.6 state: DONE" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+initiator -> target: "3.7 send_notif(\"target\", \"Done_reading\")" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_04_teardown_dark.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_04_teardown_dark.d2
new file mode 100644
index 0000000000..709ca4557a
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_04_teardown_dark.d2
@@ -0,0 +1,53 @@
+# NIXL Basic Two Peers Transfer — Phase 4: Teardown (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+initiator -> initiator: "4.1 remove_remote_agent(\"target\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+initiator -> initiator: "4.2 release_xfer_handle(handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+initiator -> target: "4.3 invalidate_local_metadata(ip, 5555)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+initiator -> initiator: "4.4 deregister_memory(initiator_reg)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+target -> target: "4.5 deregister_memory(target_reg)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> initiator: "4.6 delete agent(\"initiator\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+conductor -> target: "4.7 delete agent(\"target\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/basic-transfer/nixl_basic_transfer_04_teardown_light.d2 b/docs/diagrams/basic-transfer/nixl_basic_transfer_04_teardown_light.d2
new file mode 100644
index 0000000000..bfe86f1cbe
--- /dev/null
+++ b/docs/diagrams/basic-transfer/nixl_basic_transfer_04_teardown_light.d2
@@ -0,0 +1,50 @@
+# NIXL Basic Two Peers Transfer — Phase 4: Teardown (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+initiator: "Initiator Agent\n(Auto Port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+target: "Target Agent\n(Port 5555)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+initiator -> initiator: "4.1 remove_remote_agent(\"target\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+initiator -> initiator: "4.2 release_xfer_handle(handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+initiator -> target: "4.3 invalidate_local_metadata(ip, 5555)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+initiator -> initiator: "4.4 deregister_memory(initiator_reg)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+target -> target: "4.5 deregister_memory(target_reg)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> initiator: "4.6 delete agent(\"initiator\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+conductor -> target: "4.7 delete agent(\"target\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/data-flow/nixl_desc_hierarchy_dark.d2 b/docs/diagrams/data-flow/nixl_desc_hierarchy_dark.d2
new file mode 100644
index 0000000000..4a97763276
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_desc_hierarchy_dark.d2
@@ -0,0 +1,57 @@
+# NIXL Descriptor Class Hierarchy — Dark Mode
+# Render: d2 --theme 200 --pad 20 docs/diagrams/data-flow/nixl_desc_hierarchy_dark.d2 docs/figures/data-flow/nixl_desc_hierarchy_dark.svg
+
+direction: down
+
+nixlBasicDesc: "nixlBasicDesc" {
+ shape: class
+ style.font-size: 15
+ style.stroke: "#313244"
+ style.stroke-width: 2
+ style.fill: "#CDD6F4"
+ style.font-color: "#1E1E2E"
+
+ addr: "addr : uintptr_t"
+ len: "len : size_t"
+ devId: "devId : uint64_t"
+}
+
+nixlBlobDesc: "nixlBlobDesc" {
+ shape: class
+ style.font-size: 15
+ style.stroke: "#313244"
+ style.stroke-width: 2
+ style.fill: "#CDD6F4"
+ style.font-color: "#1E1E2E"
+
+ metaInfo: "metaInfo : nixl_blob_t"
+}
+
+nixlRemoteDesc: "nixlRemoteDesc" {
+ shape: class
+ style.font-size: 15
+ style.stroke: "#313244"
+ style.stroke-width: 2
+ style.fill: "#CDD6F4"
+ style.font-color: "#1E1E2E"
+
+ remoteAgent: "remoteAgent : std::string"
+}
+
+nixlBasicDesc -> nixlBlobDesc: "extends" {
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ target-arrowhead: {
+ shape: triangle
+ style.filled: false
+ }
+}
+
+nixlBasicDesc -> nixlRemoteDesc: "extends" {
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ target-arrowhead: {
+ shape: triangle
+ style.filled: false
+ }
+}
diff --git a/docs/diagrams/data-flow/nixl_desc_hierarchy_light.d2 b/docs/diagrams/data-flow/nixl_desc_hierarchy_light.d2
new file mode 100644
index 0000000000..cf988041d6
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_desc_hierarchy_light.d2
@@ -0,0 +1,57 @@
+# NIXL Descriptor Class Hierarchy — Light Mode
+# Render: d2 --theme 0 --pad 20 docs/diagrams/data-flow/nixl_desc_hierarchy_light.d2 docs/figures/data-flow/nixl_desc_hierarchy_light.svg
+
+direction: down
+
+nixlBasicDesc: "nixlBasicDesc" {
+ shape: class
+ style.font-size: 15
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+ style.font-color: "#333333"
+
+ addr: "addr : uintptr_t"
+ len: "len : size_t"
+ devId: "devId : uint64_t"
+}
+
+nixlBlobDesc: "nixlBlobDesc" {
+ shape: class
+ style.font-size: 15
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+ style.font-color: "#333333"
+
+ metaInfo: "metaInfo : nixl_blob_t"
+}
+
+nixlRemoteDesc: "nixlRemoteDesc" {
+ shape: class
+ style.font-size: 15
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+ style.font-color: "#333333"
+
+ remoteAgent: "remoteAgent : std::string"
+}
+
+nixlBasicDesc -> nixlBlobDesc: "extends" {
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ target-arrowhead: {
+ shape: triangle
+ style.filled: false
+ }
+}
+
+nixlBasicDesc -> nixlRemoteDesc: "extends" {
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ target-arrowhead: {
+ shape: triangle
+ style.filled: false
+ }
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_01_init_dark.d2 b/docs/diagrams/data-flow/nixl_flow_01_init_dark.d2
new file mode 100644
index 0000000000..6db05b6a90
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_01_init_dark.d2
@@ -0,0 +1,55 @@
+# NIXL Data Flow — Phase 1: Initialization (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> agent_a: "1.1 create_agent(\"Node1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_b: "1.2 create_agent(\"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_a: "1.3 create_transfer_backend(UCX)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_b: "1.4 create_transfer_backend(UCX)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "1.5 register_memory(VRAM segments)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_b -> agent_b: "1.6 register_memory(VRAM segments)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_01_init_light.d2 b/docs/diagrams/data-flow/nixl_flow_01_init_light.d2
new file mode 100644
index 0000000000..4881ff9184
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_01_init_light.d2
@@ -0,0 +1,51 @@
+# NIXL Data Flow — Phase 1: Initialization (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> agent_a: "1.1 create_agent(\"Node1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_b: "1.2 create_agent(\"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_a: "1.3 create_transfer_backend(UCX)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_b: "1.4 create_transfer_backend(UCX)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "1.5 register_memory(VRAM segments)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_b -> agent_b: "1.6 register_memory(VRAM segments)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_02_metadata_dark.d2 b/docs/diagrams/data-flow/nixl_flow_02_metadata_dark.d2
new file mode 100644
index 0000000000..d788183a15
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_02_metadata_dark.d2
@@ -0,0 +1,61 @@
+# NIXL Data Flow — Phase 2: Metadata Exchange (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+agent_a -> agent_a: "2.1 get_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_b -> agent_b: "2.2 get_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> metadata_server: "2.3 send_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_b -> metadata_server: "2.4 send_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> metadata_server: "2.5 fetch_remote_metadata(\"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+metadata_server -> agent_a: "2.6 return Node2 metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_a -> agent_b: "2.7 make_connection(\"Node2\") [optional]" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.stroke-dash: 5
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_02_metadata_light.d2 b/docs/diagrams/data-flow/nixl_flow_02_metadata_light.d2
new file mode 100644
index 0000000000..a8628464d6
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_02_metadata_light.d2
@@ -0,0 +1,57 @@
+# NIXL Data Flow — Phase 2: Metadata Exchange (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+agent_a -> agent_a: "2.1 get_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_b -> agent_b: "2.2 get_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> metadata_server: "2.3 send_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_b -> metadata_server: "2.4 send_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> metadata_server: "2.5 fetch_remote_metadata(\"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+metadata_server -> agent_a: "2.6 return Node2 metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_a -> agent_b: "2.7 make_connection(\"Node2\") [optional]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.stroke-dash: 5
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_03_transfer_dark.d2 b/docs/diagrams/data-flow/nixl_flow_03_transfer_dark.d2
new file mode 100644
index 0000000000..846ebed251
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_03_transfer_dark.d2
@@ -0,0 +1,57 @@
+# NIXL Data Flow — Phase 3: Transfer (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> agent_a: "3.1 create_xfer_req(WR, local, remote, \"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "3.2 resolve backend (auto-select)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent_a: "3.3 post_xfer_req(handle)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_b: "3.4 data transfer (RDMA write)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent_a: "3.5 get_xfer_status(handle)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> conductor: "3.6 status: complete" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_03_transfer_light.d2 b/docs/diagrams/data-flow/nixl_flow_03_transfer_light.d2
new file mode 100644
index 0000000000..cb3b0cef0d
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_03_transfer_light.d2
@@ -0,0 +1,53 @@
+# NIXL Data Flow — Phase 3: Transfer (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> agent_a: "3.1 create_xfer_req(WR, local, remote, \"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "3.2 resolve backend (auto-select)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent_a: "3.3 post_xfer_req(handle)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_b: "3.4 data transfer (RDMA write)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent_a: "3.5 get_xfer_status(handle)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> conductor: "3.6 status: complete" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_04_teardown_dark.d2 b/docs/diagrams/data-flow/nixl_flow_04_teardown_dark.d2
new file mode 100644
index 0000000000..3ce7a28e39
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_04_teardown_dark.d2
@@ -0,0 +1,55 @@
+# NIXL Data Flow — Phase 4: Teardown (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+agent_a -> metadata_server: "4.1 invalidate_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_a -> agent_a: "4.2 deregister_memory()" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent_a: "4.3 delete agent(\"Node1\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_b -> metadata_server: "4.4 invalidate_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_b -> agent_b: "4.5 deregister_memory()" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent_b: "4.6 delete agent(\"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_04_teardown_light.d2 b/docs/diagrams/data-flow/nixl_flow_04_teardown_light.d2
new file mode 100644
index 0000000000..7795bb5d36
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_04_teardown_light.d2
@@ -0,0 +1,51 @@
+# NIXL Data Flow — Phase 4: Teardown (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+agent_a -> metadata_server: "4.1 invalidate_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_a -> agent_a: "4.2 deregister_memory()" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent_a: "4.3 delete agent(\"Node1\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_b -> metadata_server: "4.4 invalidate_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_b -> agent_b: "4.5 deregister_memory()" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent_b: "4.6 delete agent(\"Node2\")" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_05_scaling_dark.d2 b/docs/diagrams/data-flow/nixl_flow_05_scaling_dark.d2
new file mode 100644
index 0000000000..767c77fcb5
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_05_scaling_dark.d2
@@ -0,0 +1,60 @@
+# NIXL Data Flow — Phase 5: Dynamic Scaling (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_c: "NIXL Agent\n(Node 3 — new)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+ style.stroke-dash: 5
+}
+
+conductor -> agent_c: "5.1 create_agent(\"Node3\") [new node]" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_c -> agent_c: "5.2 register_memory + backends" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_c -> metadata_server: "5.3 send_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> metadata_server: "5.4 fetch_remote_metadata(\"Node3\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_a: "5.5 invalidate_remote_agent(\"NodeX\") [on failure]" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+ style.stroke-dash: 5
+}
diff --git a/docs/diagrams/data-flow/nixl_flow_05_scaling_light.d2 b/docs/diagrams/data-flow/nixl_flow_05_scaling_light.d2
new file mode 100644
index 0000000000..7846a73398
--- /dev/null
+++ b/docs/diagrams/data-flow/nixl_flow_05_scaling_light.d2
@@ -0,0 +1,55 @@
+# NIXL Data Flow — Phase 5: Dynamic Scaling (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(Node 1)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(Node 2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+metadata_server: "Metadata Server\n(etcd / Redis)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+agent_c: "NIXL Agent\n(Node 3 — new)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+ style.stroke-dash: 5
+}
+
+conductor -> agent_c: "5.1 create_agent(\"Node3\") [new node]" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_c -> agent_c: "5.2 register_memory + backends" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_c -> metadata_server: "5.3 send_local_metadata()" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> metadata_server: "5.4 fetch_remote_metadata(\"Node3\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_a: "5.5 invalidate_remote_agent(\"NodeX\") [on failure]" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+ style.stroke-dash: 5
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_01_init_dark.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_01_init_dark.d2
new file mode 100644
index 0000000000..32ac700ca4
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_01_init_dark.d2
@@ -0,0 +1,59 @@
+# NIXL ETCD Metadata Exchange — Phase 1: Initialization (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> conductor: "1.1 set NIXL_ETCD_ENDPOINTS=\"http://localhost:2379\"" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent_a: "1.2 create_agent(\"EtcdAgent1\", config)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_b: "1.3 create_agent(\"EtcdAgent2\", config)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "1.4 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_b -> agent_b: "1.5 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> agent_a: "1.6 register_memory(DRAM, buf1 = 0xaa)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_b -> agent_b: "1.7 register_memory(DRAM, buf2 = 0xbb)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_01_init_light.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_01_init_light.d2
new file mode 100644
index 0000000000..3e5c8bda92
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_01_init_light.d2
@@ -0,0 +1,55 @@
+# NIXL ETCD Metadata Exchange — Phase 1: Initialization (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> conductor: "1.1 set NIXL_ETCD_ENDPOINTS=\"http://localhost:2379\"" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent_a: "1.2 create_agent(\"EtcdAgent1\", config)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent_b: "1.3 create_agent(\"EtcdAgent2\", config)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "1.4 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_b -> agent_b: "1.5 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> agent_a: "1.6 register_memory(DRAM, buf1 = 0xaa)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_b -> agent_b: "1.7 register_memory(DRAM, buf2 = 0xbb)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_02_publish_dark.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_02_publish_dark.d2
new file mode 100644
index 0000000000..7fa1a19503
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_02_publish_dark.d2
@@ -0,0 +1,41 @@
+# NIXL ETCD Metadata Exchange — Phase 2: Publish Metadata (Dark)
+
+shape: sequence_diagram
+
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+agent_a -> agent_a: "2.1 serialize metadata (descs + UCX rkeys)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> etcd: "2.2 sendLocalMD() — PUT /nixl/agents/EtcdAgent1/metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_b -> agent_b: "2.3 serialize metadata (descs + UCX rkeys)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_b -> etcd: "2.4 sendLocalMD() — PUT /nixl/agents/EtcdAgent2/metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_02_publish_light.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_02_publish_light.d2
new file mode 100644
index 0000000000..81d6d4c8ad
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_02_publish_light.d2
@@ -0,0 +1,38 @@
+# NIXL ETCD Metadata Exchange — Phase 2: Publish Metadata (Light)
+
+shape: sequence_diagram
+
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+agent_a -> agent_a: "2.1 serialize metadata (descs + UCX rkeys)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> etcd: "2.2 sendLocalMD() — PUT /nixl/agents/EtcdAgent1/metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_b -> agent_b: "2.3 serialize metadata (descs + UCX rkeys)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_b -> etcd: "2.4 sendLocalMD() — PUT /nixl/agents/EtcdAgent2/metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_03_fetch_dark.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_03_fetch_dark.d2
new file mode 100644
index 0000000000..dc0e28e2f3
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_03_fetch_dark.d2
@@ -0,0 +1,61 @@
+# NIXL ETCD Metadata Exchange — Phase 3: Fetch & Discover (Dark)
+
+shape: sequence_diagram
+
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+agent_a -> etcd: "3.1 fetchRemoteMD(\"EtcdAgent2\") — GET key" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+etcd -> agent_a: "3.2 return EtcdAgent2 metadata blob" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_a -> agent_a: "3.3 load metadata into remote sections" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> etcd: "3.4 create Watcher on /nixl/agents/EtcdAgent2/" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.stroke-dash: 5
+}
+agent_b -> etcd: "3.5 fetchRemoteMD(\"EtcdAgent1\") — GET key" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+etcd -> agent_b: "3.6 return EtcdAgent1 metadata blob" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_b -> agent_b: "3.7 load metadata into remote sections" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_b -> etcd: "3.8 create Watcher on /nixl/agents/EtcdAgent1/" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.stroke-dash: 5
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_03_fetch_light.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_03_fetch_light.d2
new file mode 100644
index 0000000000..1357b02cad
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_03_fetch_light.d2
@@ -0,0 +1,58 @@
+# NIXL ETCD Metadata Exchange — Phase 3: Fetch & Discover (Light)
+
+shape: sequence_diagram
+
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+agent_a -> etcd: "3.1 fetchRemoteMD(\"EtcdAgent2\") — GET key" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+etcd -> agent_a: "3.2 return EtcdAgent2 metadata blob" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_a -> agent_a: "3.3 load metadata into remote sections" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> etcd: "3.4 create Watcher on /nixl/agents/EtcdAgent2/" {
+ style.font-size: 20
+ style.stroke: "#999999"
+ style.stroke-dash: 5
+}
+agent_b -> etcd: "3.5 fetchRemoteMD(\"EtcdAgent1\") — GET key" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+etcd -> agent_b: "3.6 return EtcdAgent1 metadata blob" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_b -> agent_b: "3.7 load metadata into remote sections" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_b -> etcd: "3.8 create Watcher on /nixl/agents/EtcdAgent1/" {
+ style.font-size: 20
+ style.stroke: "#999999"
+ style.stroke-dash: 5
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_04_transfer_dark.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_04_transfer_dark.d2
new file mode 100644
index 0000000000..ad0fbc4983
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_04_transfer_dark.d2
@@ -0,0 +1,59 @@
+# NIXL ETCD Metadata Exchange — Phase 4: Transfer (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> agent_a: "4.1 createXferReq(WRITE, src, dst, \"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "4.2 resolve backend (UCX auto-select)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent_a: "4.3 postXferReq(handle)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_b: "4.4 RDMA write — data transfer" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent_a: "4.5 getXferStatus(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> conductor: "4.6 status: SUCCESS" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_a -> agent_b: "4.7 notification: \"notification\"" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+agent_b -> agent_b: "4.8 getNotifs() — receive notification" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_04_transfer_light.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_04_transfer_light.d2
new file mode 100644
index 0000000000..b336d7d000
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_04_transfer_light.d2
@@ -0,0 +1,56 @@
+# NIXL ETCD Metadata Exchange — Phase 4: Transfer (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+conductor -> agent_a: "4.1 createXferReq(WRITE, src, dst, \"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_a: "4.2 resolve backend (UCX auto-select)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent_a: "4.3 postXferReq(handle)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent_a -> agent_b: "4.4 RDMA write — data transfer" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent_a: "4.5 getXferStatus(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> conductor: "4.6 status: SUCCESS" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+agent_a -> agent_b: "4.7 notification: \"notification\"" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+agent_b -> agent_b: "4.8 getNotifs() — receive notification" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_05_invalidation_dark.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_05_invalidation_dark.d2
new file mode 100644
index 0000000000..066836f3fd
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_05_invalidation_dark.d2
@@ -0,0 +1,50 @@
+# NIXL ETCD Metadata Exchange — Phase 5: Invalidation & Teardown (Dark)
+
+shape: sequence_diagram
+
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+agent_a -> agent_a: "5.1 releaseXferReq(handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_a -> agent_a: "5.2 deregisterMem(dlist1)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_b -> agent_b: "5.3 deregisterMem(dlist2)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent_a -> etcd: "5.4 invalidateLocalMD() — DELETE /nixl/agents/EtcdAgent1/" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+etcd -> agent_b: "5.5 Watcher fires DELETE event for EtcdAgent1" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+ style.stroke-dash: 3
+}
+agent_b -> agent_b: "5.6 discard cached EtcdAgent1 metadata" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/etcd-metadata/nixl_etcd_metadata_05_invalidation_light.d2 b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_05_invalidation_light.d2
new file mode 100644
index 0000000000..e0fc09c7c1
--- /dev/null
+++ b/docs/diagrams/etcd-metadata/nixl_etcd_metadata_05_invalidation_light.d2
@@ -0,0 +1,47 @@
+# NIXL ETCD Metadata Exchange — Phase 5: Invalidation & Teardown (Light)
+
+shape: sequence_diagram
+
+agent_a: "NIXL Agent\n(\"EtcdAgent1\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+etcd: "etcd Server\n(localhost:2379)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+agent_b: "NIXL Agent\n(\"EtcdAgent2\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+agent_a -> agent_a: "5.1 releaseXferReq(handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent_a -> agent_a: "5.2 deregisterMem(dlist1)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_b -> agent_b: "5.3 deregisterMem(dlist2)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent_a -> etcd: "5.4 invalidateLocalMD() — DELETE /nixl/agents/EtcdAgent1/" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+etcd -> agent_b: "5.5 Watcher fires DELETE event for EtcdAgent1" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+ style.stroke-dash: 3
+}
+agent_b -> agent_b: "5.6 discard cached EtcdAgent1 metadata" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_01_init_dark.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_01_init_dark.d2
new file mode 100644
index 0000000000..a5d5a5e7f2
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_01_init_dark.d2
@@ -0,0 +1,56 @@
+# NIXL GDS Direct Storage — Phase 1: Initialization (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> agent: "1.1 create_agent(\"GDSTester\", backends=[])" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "1.2 get_plugin_list() — verify GDS available" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent: "1.3 create_backend(\"GDS\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent: "1.4 allocate DRAM buf1 (16*4096 B, fill 0xba)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent: "1.5 allocate DRAM buf2 (16*4096 B, empty)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "1.6 register_memory(DRAM: buf1, buf2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> storage: "1.7 open/create file (O_RDWR | O_CREAT)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent -> agent: "1.8 register_memory(FILE: fd, size)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_01_init_light.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_01_init_light.d2
new file mode 100644
index 0000000000..cdc5861343
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_01_init_light.d2
@@ -0,0 +1,53 @@
+# NIXL GDS Direct Storage — Phase 1: Initialization (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> agent: "1.1 create_agent(\"GDSTester\", backends=[])" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "1.2 get_plugin_list() — verify GDS available" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent: "1.3 create_backend(\"GDS\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent: "1.4 allocate DRAM buf1 (16*4096 B, fill 0xba)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> agent: "1.5 allocate DRAM buf2 (16*4096 B, empty)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "1.6 register_memory(DRAM: buf1, buf2)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+conductor -> storage: "1.7 open/create file (O_RDWR | O_CREAT)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent -> agent: "1.8 register_memory(FILE: fd, size)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_02_write_dark.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_02_write_dark.d2
new file mode 100644
index 0000000000..80165dc577
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_02_write_dark.d2
@@ -0,0 +1,50 @@
+# NIXL GDS Direct Storage — Phase 2: Write Transfer (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> agent: "2.1 initialize_xfer(WRITE, DRAM buf1, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "2.2 resolve backend (GDS auto-select)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent: "2.3 transfer(xfer_handle_write)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> storage: "2.4 GDS direct write — DRAM buf1 to file" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent: "2.5 check_xfer_state(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent -> conductor: "2.6 state: DONE" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_02_write_light.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_02_write_light.d2
new file mode 100644
index 0000000000..083a79c3f3
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_02_write_light.d2
@@ -0,0 +1,47 @@
+# NIXL GDS Direct Storage — Phase 2: Write Transfer (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> agent: "2.1 initialize_xfer(WRITE, DRAM buf1, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "2.2 resolve backend (GDS auto-select)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent: "2.3 transfer(xfer_handle_write)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> storage: "2.4 GDS direct write — DRAM buf1 to file" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent: "2.5 check_xfer_state(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent -> conductor: "2.6 state: DONE" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_03_read_dark.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_03_read_dark.d2
new file mode 100644
index 0000000000..8529ebd501
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_03_read_dark.d2
@@ -0,0 +1,50 @@
+# NIXL GDS Direct Storage — Phase 3: Read Transfer (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> agent: "3.1 initialize_xfer(READ, DRAM buf2, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "3.2 resolve backend (GDS auto-select)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> agent: "3.3 transfer(xfer_handle_read)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+storage -> agent: "3.4 GDS direct read — file to DRAM buf2" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent: "3.5 check_xfer_state(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent -> conductor: "3.6 state: DONE" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_03_read_light.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_03_read_light.d2
new file mode 100644
index 0000000000..b72166ab82
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_03_read_light.d2
@@ -0,0 +1,47 @@
+# NIXL GDS Direct Storage — Phase 3: Read Transfer (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> agent: "3.1 initialize_xfer(READ, DRAM buf2, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "3.2 resolve backend (GDS auto-select)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> agent: "3.3 transfer(xfer_handle_read)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+storage -> agent: "3.4 GDS direct read — file to DRAM buf2" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+conductor -> agent: "3.5 check_xfer_state(handle) [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent -> conductor: "3.6 state: DONE" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_04_verify_dark.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_04_verify_dark.d2
new file mode 100644
index 0000000000..39a7fbdf1e
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_04_verify_dark.d2
@@ -0,0 +1,52 @@
+# NIXL GDS Direct Storage — Phase 4: Verify & Teardown (Dark)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+conductor -> conductor: "4.1 verify_transfer(buf1, buf2) — compare 0xba" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "4.2 release_xfer_handle(write_handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent -> agent: "4.3 release_xfer_handle(read_handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent -> agent: "4.4 deregister_memory(DRAM descs)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+agent -> agent: "4.5 deregister_memory(FILE descs)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+conductor -> conductor: "4.6 free_passthru(buf1, buf2)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+conductor -> storage: "4.7 close(fd)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/gds-direct/nixl_gds_direct_04_verify_light.d2 b/docs/diagrams/gds-direct/nixl_gds_direct_04_verify_light.d2
new file mode 100644
index 0000000000..d506a7dd72
--- /dev/null
+++ b/docs/diagrams/gds-direct/nixl_gds_direct_04_verify_light.d2
@@ -0,0 +1,49 @@
+# NIXL GDS Direct Storage — Phase 4: Verify & Teardown (Light)
+
+shape: sequence_diagram
+
+conductor: Conductor {
+ style.font-size: 20
+ style.stroke: "#333333"
+ style.fill: "#F5F5F5"
+}
+agent: "NIXL Agent\n(\"GDSTester\")" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(NVMe / File)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+conductor -> conductor: "4.1 verify_transfer(buf1, buf2) — compare 0xba" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+agent -> agent: "4.2 release_xfer_handle(write_handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent -> agent: "4.3 release_xfer_handle(read_handle)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+agent -> agent: "4.4 deregister_memory(DRAM descs)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+agent -> agent: "4.5 deregister_memory(FILE descs)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+conductor -> conductor: "4.6 free_passthru(buf1, buf2)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
+conductor -> storage: "4.7 close(fd)" {
+ style.font-size: 20
+ style.stroke: "#CC0000"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_01_init_dark.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_01_init_dark.d2
new file mode 100644
index 0000000000..ce69e35511
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_01_init_dark.d2
@@ -0,0 +1,60 @@
+# NIXL Remote Storage — Phase 1: Initialization (Dark)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+client -> client: "1.1 create_agent(\"client\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+client -> client: "1.2 create_backend(\"GDS_MT\" / \"POSIX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+client -> client: "1.3 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+client -> client: "1.4 register_memory(VRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+client -> client: "1.5 register_memory(FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+
+server -> server: "1.6 create_agent(\"server\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+server -> server: "1.7 create_backend(\"GDS_MT\" / \"POSIX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+server -> server: "1.8 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+server -> server: "1.9 register_memory(DRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> server: "1.10 register_memory(FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_01_init_light.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_01_init_light.d2
new file mode 100644
index 0000000000..b68f53f884
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_01_init_light.d2
@@ -0,0 +1,58 @@
+# NIXL Remote Storage — Phase 1: Initialization (Light)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+client -> client: "1.1 create_agent(\"client\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+client -> client: "1.2 create_backend(\"GDS_MT\" / \"POSIX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+client -> client: "1.3 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+client -> client: "1.4 register_memory(VRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+client -> client: "1.5 register_memory(FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+
+server -> server: "1.6 create_agent(\"server\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+server -> server: "1.7 create_backend(\"GDS_MT\" / \"POSIX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+server -> server: "1.8 create_backend(\"UCX\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+server -> server: "1.9 register_memory(DRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> server: "1.10 register_memory(FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_02_metadata_dark.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_02_metadata_dark.d2
new file mode 100644
index 0000000000..75f159a033
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_02_metadata_dark.d2
@@ -0,0 +1,36 @@
+# NIXL Remote Storage — Phase 2: Metadata Exchange (Dark)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+client -> server: "2.1 send_local_metadata(server_ip, port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+client -> server: "2.2 fetch_remote_metadata(\"server\", ip, port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> client: "2.3 return server metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+client -> client: "2.4 check_remote_metadata(\"server\") [poll]" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_02_metadata_light.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_02_metadata_light.d2
new file mode 100644
index 0000000000..7d4cee37ac
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_02_metadata_light.d2
@@ -0,0 +1,34 @@
+# NIXL Remote Storage — Phase 2: Metadata Exchange (Light)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+client -> server: "2.1 send_local_metadata(server_ip, port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+client -> server: "2.2 fetch_remote_metadata(\"server\", ip, port)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> client: "2.3 return server metadata" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-dash: 3
+}
+client -> client: "2.4 check_remote_metadata(\"server\") [poll]" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_03_write_dark.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_03_write_dark.d2
new file mode 100644
index 0000000000..a34a35b929
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_03_write_dark.d2
@@ -0,0 +1,68 @@
+# NIXL Remote Storage — Phase 3: Remote Write Request (Dark)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+client -> client: "3.1 get_serialized_descs(VRAM descs)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+client -> server: "3.2 send_notif(\"WRTE\" + iterations + descs)" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+server -> server: "3.3 deserialize_descs(received data)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+server -> client: "3.4 initialize_xfer(READ, DRAM, client VRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> client: "3.5 transfer — network read from client" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> storage: "3.6 initialize_xfer(WRITE, DRAM, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> storage: "3.7 transfer — storage write" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> server: "3.8 repeat 3.4–3.7 (pipelined)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.stroke-dash: 5
+}
+server -> client: "3.9 send_notif(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+client -> client: "3.10 check_remote_xfer_done(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_03_write_light.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_03_write_light.d2
new file mode 100644
index 0000000000..370127334d
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_03_write_light.d2
@@ -0,0 +1,65 @@
+# NIXL Remote Storage — Phase 3: Remote Write Request (Light)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+client -> client: "3.1 get_serialized_descs(VRAM descs)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+client -> server: "3.2 send_notif(\"WRTE\" + iterations + descs)" {
+ style.font-size: 20
+ style.stroke: "#D4880F"
+}
+server -> server: "3.3 deserialize_descs(received data)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+server -> client: "3.4 initialize_xfer(READ, DRAM, client VRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> client: "3.5 transfer — network read from client" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> storage: "3.6 initialize_xfer(WRITE, DRAM, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> storage: "3.7 transfer — storage write" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> server: "3.8 repeat 3.4–3.7 (pipelined)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.stroke-dash: 5
+}
+server -> client: "3.9 send_notif(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#D4880F"
+}
+client -> client: "3.10 check_remote_xfer_done(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_04_read_dark.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_04_read_dark.d2
new file mode 100644
index 0000000000..00a4fa442c
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_04_read_dark.d2
@@ -0,0 +1,68 @@
+# NIXL Remote Storage — Phase 4: Remote Read Request (Dark)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+client -> client: "4.1 get_serialized_descs(VRAM descs)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+client -> server: "4.2 send_notif(\"READ\" + iterations + descs)" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+server -> server: "4.3 deserialize_descs(received data)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
+server -> storage: "4.4 initialize_xfer(READ, DRAM, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> storage: "4.5 transfer — storage read" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> client: "4.6 initialize_xfer(WRITE, DRAM, client VRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> client: "4.7 transfer — network write to client" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> server: "4.8 repeat 4.4–4.7 (pipelined)" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+ style.stroke-dash: 5
+}
+server -> client: "4.9 send_notif(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#F5A623"
+}
+client -> client: "4.10 check_remote_xfer_done(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#AAAAAA"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_04_read_light.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_04_read_light.d2
new file mode 100644
index 0000000000..26dd7084e9
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_04_read_light.d2
@@ -0,0 +1,65 @@
+# NIXL Remote Storage — Phase 4: Remote Read Request (Light)
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+client -> client: "4.1 get_serialized_descs(VRAM descs)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+client -> server: "4.2 send_notif(\"READ\" + iterations + descs)" {
+ style.font-size: 20
+ style.stroke: "#D4880F"
+}
+server -> server: "4.3 deserialize_descs(received data)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
+server -> storage: "4.4 initialize_xfer(READ, DRAM, FILE)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> storage: "4.5 transfer — storage read" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> client: "4.6 initialize_xfer(WRITE, DRAM, client VRAM)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+}
+server -> client: "4.7 transfer — network write to client" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+server -> server: "4.8 repeat 4.4–4.7 (pipelined)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.stroke-dash: 5
+}
+server -> client: "4.9 send_notif(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#D4880F"
+}
+client -> client: "4.10 check_remote_xfer_done(\"COMPLETE\")" {
+ style.font-size: 20
+ style.stroke: "#888888"
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_read_dark.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_read_dark.d2
new file mode 100644
index 0000000000..dbfe7e24db
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_read_dark.d2
@@ -0,0 +1,71 @@
+# NIXL Remote Storage — Read Pipeline (Dark)
+# Shows pipelined overlap: storage read N runs concurrently with network write N-1
+
+shape: sequence_diagram
+
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+
+server -> storage: "Storage Read 1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+
+server -> client: "Network Write 1" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
+server -> storage: "Storage Read 2 ← overlaps with Write 1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+
+server -> client: "Network Write 2" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
+server -> storage: "Storage Read 3 ← overlaps with Write 2" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+
+server -> client: "Network Write 3" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
+server -> storage: "Storage Read N ← overlaps with Write N-1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+ style.stroke-dash: 5
+}
+
+server -> client: "Network Write N" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_read_light.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_read_light.d2
new file mode 100644
index 0000000000..607771cb7f
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_read_light.d2
@@ -0,0 +1,68 @@
+# NIXL Remote Storage — Read Pipeline (Light)
+# Shows pipelined overlap: storage read N runs concurrently with network write N-1
+
+shape: sequence_diagram
+
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+
+server -> storage: "Storage Read 1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+
+server -> client: "Network Write 1" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
+server -> storage: "Storage Read 2 ← overlaps with Write 1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+
+server -> client: "Network Write 2" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
+server -> storage: "Storage Read 3 ← overlaps with Write 2" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+
+server -> client: "Network Write 3" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
+server -> storage: "Storage Read N ← overlaps with Write N-1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+ style.stroke-dash: 5
+}
+
+server -> client: "Network Write N" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_write_dark.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_write_dark.d2
new file mode 100644
index 0000000000..2593b08944
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_write_dark.d2
@@ -0,0 +1,71 @@
+# NIXL Remote Storage — Write Pipeline (Dark)
+# Shows pipelined overlap: network read N runs concurrently with storage write N-1
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#1A2A0D"
+ style.font-color: "#EEEEEE"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#888888"
+ style.fill: "#222222"
+ style.font-color: "#EEEEEE"
+}
+
+client -> server: "Network Read 1" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
+
+server -> storage: "Storage Write 1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+client -> server: "Network Read 2 ← overlaps with Write 1" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
+
+server -> storage: "Storage Write 2" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+client -> server: "Network Read 3 ← overlaps with Write 2" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+}
+
+server -> storage: "Storage Write 3" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+client -> server: "Network Read N ← overlaps with Write N-1" {
+ style.font-size: 20
+ style.stroke: "#6C8EBF"
+ style.stroke-width: 3
+ style.stroke-dash: 5
+}
+
+server -> storage: "Storage Write N" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
diff --git a/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_write_light.d2 b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_write_light.d2
new file mode 100644
index 0000000000..a7b754d438
--- /dev/null
+++ b/docs/diagrams/remote-storage/nixl_remote_storage_pipeline_write_light.d2
@@ -0,0 +1,68 @@
+# NIXL Remote Storage — Write Pipeline (Light)
+# Shows pipelined overlap: network read N runs concurrently with storage write N-1
+
+shape: sequence_diagram
+
+client: "Client\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+server: "Storage Server\n(NIXL Agent)" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 2
+ style.fill: "#F0F7E6"
+}
+storage: "Local Storage\n(GDS / POSIX)" {
+ style.font-size: 20
+ style.stroke: "#666666"
+ style.fill: "#F5F5F5"
+}
+
+client -> server: "Network Read 1" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
+
+server -> storage: "Storage Write 1" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+client -> server: "Network Read 2 ← overlaps with Write 1" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
+
+server -> storage: "Storage Write 2" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+client -> server: "Network Read 3 ← overlaps with Write 2" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+}
+
+server -> storage: "Storage Write 3" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
+client -> server: "Network Read N ← overlaps with Write N-1" {
+ style.font-size: 20
+ style.stroke: "#4A7ABF"
+ style.stroke-width: 3
+ style.stroke-dash: 5
+}
+
+server -> storage: "Storage Write N" {
+ style.font-size: 20
+ style.stroke: "#74B900"
+ style.stroke-width: 3
+}
diff --git a/docs/diagrams/render.sh b/docs/diagrams/render.sh
new file mode 100755
index 0000000000..29cf66f9e0
--- /dev/null
+++ b/docs/diagrams/render.sh
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Check d2 availability (per D-16)
+if ! command -v d2 &>/dev/null; then
+ echo "Error: d2 CLI not found."
+ echo "Install: https://d2lang.com/tour/install or brew install d2"
+ exit 1
+fi
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FIGURES_DIR="$SCRIPT_DIR/../figures"
+
+# Topic → list of diagram base names (without _light/_dark suffix)
+declare -A TOPICS
+TOPICS[data-flow]="nixl_flow_01_init nixl_flow_02_metadata nixl_flow_03_transfer nixl_flow_04_teardown nixl_flow_05_scaling nixl_desc_hierarchy"
+TOPICS[basic-transfer]="nixl_basic_transfer_01_init nixl_basic_transfer_02_metadata nixl_basic_transfer_03_transfer nixl_basic_transfer_04_teardown"
+TOPICS[gds-direct]="nixl_gds_direct_01_init nixl_gds_direct_02_write nixl_gds_direct_03_read nixl_gds_direct_04_verify"
+TOPICS[etcd-metadata]="nixl_etcd_metadata_01_init nixl_etcd_metadata_02_publish nixl_etcd_metadata_03_fetch nixl_etcd_metadata_04_transfer nixl_etcd_metadata_05_invalidation"
+TOPICS[remote-storage]="nixl_remote_storage_01_init nixl_remote_storage_02_metadata nixl_remote_storage_03_write nixl_remote_storage_04_read nixl_remote_storage_pipeline_read nixl_remote_storage_pipeline_write"
+
+render_diagram() {
+ local topic="$1"
+ local name="$2"
+ local src_dir="$SCRIPT_DIR/$topic"
+ local out_dir="$FIGURES_DIR/$topic"
+ mkdir -p "$out_dir"
+
+ for variant in light dark; do
+ local src="$src_dir/${name}_${variant}.d2"
+ local out="$out_dir/${name}_${variant}.svg"
+ if [[ ! -f "$src" ]]; then
+ echo "Warning: $src not found, skipping."
+ continue
+ fi
+ local theme=0
+ [[ "$variant" == "dark" ]] && theme=200
+ echo "Rendering: $src -> $out (theme $theme)"
+ d2 --theme "$theme" --pad 20 "$src" "$out"
+ done
+}
+
+render_topic() {
+ local topic="$1"
+ if [[ -z "${TOPICS[$topic]+x}" ]]; then
+ echo "Error: unknown topic '$topic'"
+ echo "Available topics: ${!TOPICS[*]}"
+ exit 1
+ fi
+ for diagram in ${TOPICS[$topic]}; do
+ render_diagram "$topic" "$diagram"
+ done
+}
+
+if [[ $# -eq 0 ]]; then
+ # Render all topics
+ for topic in "${!TOPICS[@]}"; do
+ render_topic "$topic"
+ done
+elif [[ $# -eq 1 ]]; then
+ # Render one topic
+ render_topic "$1"
+elif [[ $# -eq 2 ]]; then
+ # Render one diagram within a topic
+ render_diagram "$1" "$2"
+else
+ echo "Usage: $0 [topic] [diagram_name]"
+ echo "Topics: ${!TOPICS[*]}"
+ exit 1
+fi
+
+echo "Done."
diff --git a/docs/examples/basic-transfer.md b/docs/examples/basic-transfer.md
new file mode 100644
index 0000000000..07d65b0192
--- /dev/null
+++ b/docs/examples/basic-transfer.md
@@ -0,0 +1,404 @@
+---
+title: Basic Two-Peer Transfer
+description: Create two Transfer Agents, register memory, exchange metadata, and execute an asynchronous data transfer between them using the UCX backend.
+---
+
+The examples below are taken from the `examples/` directory in the [NIXL repository](https://github.com/ai-dynamo/nixl), annotated with inline explanations. For the conceptual overview of NIXL's transfer workflow, see [Quick Start](../getting-started/quick-start).
+
+**What you'll learn:** How to create two Transfer Agents, register memory regions, exchange metadata via a side channel, and execute an asynchronous data transfer between them.
+
+This example runs two agents in the same process (or across two processes/machines) and performs a data transfer using the UCX backend. The workflow follows the standard NIXL lifecycle: create agents, register memory, exchange metadata, execute transfer, verify, and tear down.
+
+The four phases below -- initialization, metadata exchange, transfer, and teardown -- cover the complete two-peer transfer lifecycle. Each phase includes a sequence diagram followed by a code walkthrough.
+
+### Initialization
+
+
+```python title="Python"
+import torch
+from nixl._api import nixl_agent, nixl_agent_config
+
+# Step 1: Configure and create two agents
+# enable_prog_thread=True enables the progress thread for async transfers
+# enable_notifs=True enables notification support between agents
+# listen_port sets the port for the target agent (0 = auto-assign for initiator)
+target_config = nixl_agent_config(True, True, 5555)
+initiator_config = nixl_agent_config(True, True, 0)
+
+target_agent = nixl_agent("target", target_config)
+initiator_agent = nixl_agent("initiator", initiator_config)
+
+# Step 2: Allocate memory and register with NIXL
+# The target has data (ones), the initiator starts empty (zeros)
+torch.set_default_device("cpu")
+target_tensor = torch.ones((10, 16), dtype=torch.float32)
+initiator_tensor = torch.zeros((10, 16), dtype=torch.float32)
+
+# register_memory tells NIXL about the memory regions so they can be
+# used in transfers -- the backend pins/registers them as needed
+target_reg = target_agent.register_memory(target_tensor)
+initiator_reg = initiator_agent.register_memory(initiator_tensor)
+
+# Step 3: Create transfer descriptors from tensor rows
+# Each row becomes a separate transfer descriptor
+target_rows = [target_tensor[i, :] for i in range(target_tensor.shape[0])]
+target_descs = target_agent.get_xfer_descs(target_rows)
+target_desc_str = target_agent.get_serialized_descs(target_descs)
+
+initiator_rows = [initiator_tensor[i, :] for i in range(initiator_tensor.shape[0])]
+initiator_descs = initiator_agent.get_xfer_descs(initiator_rows)
+
+# Step 4: Exchange metadata between agents
+# In a real deployment, agents run on different machines and exchange
+# metadata over the network (TCP, ETCD, etc.)
+initiator_agent.fetch_remote_metadata("target", "127.0.0.1", 5555)
+initiator_agent.send_local_metadata("127.0.0.1", 5555)
+
+# Wait for the target's metadata to be available
+ready = False
+while not ready:
+ ready = initiator_agent.check_remote_metadata("target")
+
+# The target sends its descriptors to the initiator via notification
+target_agent.send_notif("initiator", target_desc_str)
+
+# Initiator receives the target's descriptors
+notifs = initiator_agent.get_new_notifs()
+while len(notifs) == 0:
+ notifs = initiator_agent.get_new_notifs()
+target_descs = initiator_agent.deserialize_descs(notifs["target"][0])
+
+# Step 5: Create and post the transfer request
+# READ means the initiator reads data FROM the target
+# "Done_reading" is a notification sent to the target when complete
+xfer_handle = initiator_agent.initialize_xfer(
+ "READ", initiator_descs, target_descs, "target", "Done_reading"
+)
+
+# Step 6: Execute the transfer and poll for completion
+state = initiator_agent.transfer(xfer_handle)
+
+while True:
+ state = initiator_agent.check_xfer_state(xfer_handle)
+ if state == "ERR":
+ raise RuntimeError("Transfer failed")
+ elif state == "DONE":
+ break
+
+# Step 7: Verify the data was transferred correctly
+assert torch.allclose(initiator_tensor, torch.ones((10, 16)))
+
+# Step 8: Clean up resources
+initiator_agent.remove_remote_agent("target")
+initiator_agent.release_xfer_handle(xfer_handle)
+initiator_agent.invalidate_local_metadata("127.0.0.1", 5555)
+initiator_agent.deregister_memory(initiator_reg)
+target_agent.deregister_memory(target_reg)
+```
+
+```cpp title="C++"
+#include
+#include
+#include "nixl.h"
+
+int main() {
+ // Step 1: Create two agents with progress threads enabled
+ nixlAgentConfig cfg;
+ cfg.useProgThread = true;
+
+ nixlAgent A1("Agent001", cfg);
+ nixlAgent A2("Agent002", cfg);
+
+ // Step 2: Create UCX backends for both agents
+ nixl_b_params_t init1, init2;
+ nixl_mem_list_t mems1, mems2;
+
+ A1.getPluginParams("UCX", mems1, init1);
+ A2.getPluginParams("UCX", mems2, init2);
+
+ nixlBackendH *bknd1, *bknd2;
+ A1.createBackend("UCX", init1, bknd1);
+ A2.createBackend("UCX", init2, bknd2);
+
+ nixl_opt_args_t extra_params1, extra_params2;
+ extra_params1.backends.push_back(bknd1);
+ extra_params2.backends.push_back(bknd2);
+
+ // Step 3: Allocate memory and register with NIXL
+ // Agent1 has source data (0xbb), Agent2 has empty destination (0x00)
+ nixl_reg_dlist_t dlist1(DRAM_SEG), dlist2(DRAM_SEG);
+ size_t len = 256;
+ void* addr1 = calloc(1, len);
+ void* addr2 = calloc(1, len);
+
+ memset(addr1, 0xbb, len);
+ memset(addr2, 0, len);
+
+ // Create descriptors pointing to the allocated buffers
+ nixlBlobDesc buff1, buff2;
+ buff1.addr = (uintptr_t)addr1;
+ buff1.len = len;
+ buff1.devId = 0;
+ dlist1.addDesc(buff1);
+
+ buff2.addr = (uintptr_t)addr2;
+ buff2.len = len;
+ buff2.devId = 0;
+ dlist2.addDesc(buff2);
+
+ // Register memory regions with their respective agents
+ A1.registerMem(dlist1, &extra_params1);
+ A2.registerMem(dlist2, &extra_params2);
+
+ // Step 4: Exchange metadata between agents
+ // In a single-process example, we serialize and load directly
+ std::string meta1, meta2;
+ A1.getLocalMD(meta1);
+ A2.getLocalMD(meta2);
+
+ std::string remote_name;
+ A1.loadRemoteMD(meta2, remote_name);
+
+ // Step 5: Set up transfer descriptors
+ // Transfer 8 bytes from offset 16 in Agent1 to offset 8 in Agent2
+ size_t req_size = 8;
+
+ nixl_xfer_dlist_t src_descs(DRAM_SEG);
+ nixlBasicDesc src;
+ src.addr = (uintptr_t)(((char*)addr1) + 16);
+ src.len = req_size;
+ src.devId = 0;
+ src_descs.addDesc(src);
+
+ nixl_xfer_dlist_t dst_descs(DRAM_SEG);
+ nixlBasicDesc dst;
+ dst.addr = (uintptr_t)(((char*)addr2) + 8);
+ dst.len = req_size;
+ dst.devId = 0;
+ dst_descs.addDesc(dst);
+
+ // Step 6: Create and post the transfer request
+ // NIXL_WRITE writes from source (Agent1) to destination (Agent2)
+ // A notification "notification" is sent to Agent2 on completion
+ nixlXferReqH *req_handle;
+ extra_params1.notif = "notification";
+
+ A1.createXferReq(NIXL_WRITE, src_descs, dst_descs,
+ "Agent002", req_handle, &extra_params1);
+
+ nixl_status_t status = A1.postXferReq(req_handle);
+
+ // Step 7: Poll for transfer completion and notification delivery
+ nixl_notifs_t notif_map;
+ int n_notifs = 0;
+
+ while (status != NIXL_SUCCESS || n_notifs == 0) {
+ if (status != NIXL_SUCCESS)
+ status = A1.getXferStatus(req_handle);
+ if (n_notifs == 0)
+ A2.getNotifs(notif_map);
+ n_notifs = notif_map.size();
+ }
+
+ std::cout << "Transfer verified" << std::endl;
+
+ // Step 8: Clean up -- release handle, deregister memory, invalidate metadata
+ A1.releaseXferReq(req_handle);
+ A1.deregisterMem(dlist1, &extra_params1);
+ A2.deregisterMem(dlist2, &extra_params2);
+ A1.invalidateRemoteMD("Agent002");
+
+ free(addr1);
+ free(addr2);
+
+ std::cout << "Test done" << std::endl;
+}
+```
+
+```rust title="Rust"
+use nixl_sys::{
+ Agent, MemType, NixlRegistration, NotificationMap, OptArgs,
+ SystemStorage, XferDescList, XferOp,
+};
+use std::error::Error;
+use std::thread;
+use std::time::Duration;
+
+fn main() -> Result<(), Box> {
+ // Step 1: Create two agents in the same process
+ let agent1 = Agent::new("Agent001")?;
+ let agent2 = Agent::new("Agent002")?;
+
+ // Step 2: Create UCX backends for both agents
+ let (_, params1) = agent1.get_plugin_params("UCX")?;
+ let (_, params2) = agent2.get_plugin_params("UCX")?;
+
+ let backend1 = agent1.create_backend("UCX", ¶ms1)?;
+ let backend2 = agent2.create_backend("UCX", ¶ms2)?;
+
+ // Step 3: Allocate and register memory
+ // Agent1 memory is filled with 0xaa, Agent2 with 0xbb
+ let buffer_size = 1024;
+
+ let mut storage1 = SystemStorage::new(buffer_size)?;
+ storage1.memset(0xaa);
+
+ let mut storage2 = SystemStorage::new(buffer_size)?;
+ storage2.memset(0xbb);
+
+ // Register memory with backends specified in OptArgs
+ let mut opt_args1 = OptArgs::new()?;
+ opt_args1.add_backend(&backend1)?;
+
+ let mut opt_args2 = OptArgs::new()?;
+ opt_args2.add_backend(&backend2)?;
+
+ storage1.register(&agent1, Some(&opt_args1))?;
+ storage2.register(&agent2, Some(&opt_args2))?;
+
+ // Step 4: Exchange metadata between agents
+ // Serialize local metadata and load it into the remote agent
+ let md1 = agent1.get_local_md()?;
+ let md2 = agent2.get_local_md()?;
+
+ agent1.load_remote_md(&md2)?;
+ agent2.load_remote_md(&md1)?;
+
+ // Step 5: Create transfer descriptors
+ // Transfer 4 bytes from the start of Agent1's buffer to Agent2's buffer
+ let req_size = 4;
+
+ let mut src_desc = XferDescList::new(MemType::Dram)?;
+ let src_ptr = unsafe { storage1.as_ptr() as usize };
+ src_desc.add_desc(src_ptr, req_size, 0)?;
+
+ let mut dst_desc = XferDescList::new(MemType::Dram)?;
+ let dst_ptr = unsafe { storage2.as_ptr() as usize };
+ dst_desc.add_desc(dst_ptr, req_size, 0)?;
+
+ // Step 6: Create and post the transfer request
+ // XferOp::Write sends data from Agent1 to Agent2
+ let mut xfer_opt_args = OptArgs::new()?;
+ xfer_opt_args.add_backend(&backend1)?;
+ xfer_opt_args.set_notification_message(b"notification")?;
+ xfer_opt_args.set_has_notification(true)?;
+
+ let xfer_req = agent1.create_xfer_req(
+ XferOp::Write,
+ &src_desc,
+ &dst_desc,
+ "Agent002",
+ Some(&xfer_opt_args),
+ )?;
+
+ agent1.post_xfer_req(&xfer_req, Some(&xfer_opt_args))?;
+
+ // Step 7: Wait for transfer completion and notification
+ let mut completed = false;
+ let mut received_notification = false;
+ let start = std::time::Instant::now();
+
+ while (!completed || !received_notification)
+ && start.elapsed() < Duration::from_secs(5)
+ {
+ if !completed {
+ if let Ok(status) = agent1.get_xfer_status(&xfer_req) {
+ if status.is_success() {
+ completed = true;
+ }
+ }
+ }
+
+ if !received_notification {
+ let mut notifs = NotificationMap::new()?;
+ agent2.get_notifications(&mut notifs, None)?;
+ if !notifs.is_empty()? {
+ received_notification = true;
+ }
+ }
+
+ if !completed || !received_notification {
+ thread::sleep(Duration::from_millis(100));
+ }
+ }
+
+ // Step 8: Verify the data was transferred
+ let data2 = storage2.as_slice();
+ assert_eq!(data2[0], 0xaa, "Memory should contain Agent1's pattern");
+
+ println!("Transfer and notification verified");
+ Ok(())
+}
+```
+
+
+**Expected output:**
+
+```
+Transfer verified
+Test done
+```
+
+
+For a step-by-step explanation of the NIXL transfer workflow (initialization, registration, metadata exchange, transfer, teardown), see [Quick Start](../getting-started/quick-start).
+
diff --git a/docs/examples/etcd-metadata-exchange.md b/docs/examples/etcd-metadata-exchange.md
new file mode 100644
index 0000000000..09df8221d9
--- /dev/null
+++ b/docs/examples/etcd-metadata-exchange.md
@@ -0,0 +1,262 @@
+---
+title: etcd Metadata Exchange
+description: Use etcd for automatic distributed metadata exchange between Transfer Agents, eliminating manual side-channel metadata serialization.
+---
+
+The examples below are taken from the `examples/` directory in the [NIXL repository](https://github.com/ai-dynamo/nixl), annotated with inline explanations. For the conceptual overview of NIXL's transfer workflow, see [Quick Start](../getting-started/quick-start).
+
+**What you'll learn:** How to use etcd for automatic distributed metadata exchange between Transfer Agents, eliminating the need for manual side-channel metadata serialization.
+
+When etcd is configured, agents publish their metadata (connection info and registered memory descriptors) to an etcd server and discover remote agents by name. This replaces the manual `getLocalMD`/`loadRemoteMD` exchange shown in the basic transfer example, making it suitable for production deployments with many agents.
+
+The five phases below -- initialization, publish, fetch, transfer, and invalidation -- cover the complete etcd metadata exchange lifecycle. Each phase includes a sequence diagram followed by a code walkthrough.
+
+### Initialization
+
+/metadata`. No IP/port arguments are needed -- the absence of address arguments signals etcd mode.
+
+### Fetch & Discover from etcd
+
+
+```cpp title="C++"
+#include
+#include
+#include
+#include
+#include "nixl.h"
+
+int main() {
+ // Step 1: Configure etcd endpoint
+ // NIXL reads NIXL_ETCD_ENDPOINTS from the environment.
+ // If not set, configure it programmatically.
+ if (!getenv("NIXL_ETCD_ENDPOINTS")) {
+ setenv("NIXL_ETCD_ENDPOINTS", "http://localhost:2379", 1);
+ }
+
+ // Step 2: Create agents -- configuration is the same as without etcd
+ nixlAgentConfig cfg;
+ cfg.useProgThread = true;
+
+ nixlAgent A1("EtcdAgent1", cfg);
+ nixlAgent A2("EtcdAgent2", cfg);
+
+ // Step 3: Create UCX backends and register memory
+ nixl_b_params_t init1, init2;
+ nixl_mem_list_t mems1, mems2;
+
+ A1.getPluginParams("UCX", mems1, init1);
+ A2.getPluginParams("UCX", mems2, init2);
+
+ nixlBackendH *ucx1, *ucx2;
+ A1.createBackend("UCX", init1, ucx1);
+ A2.createBackend("UCX", init2, ucx2);
+
+ nixl_opt_args_t extra_params1, extra_params2;
+ extra_params1.backends.push_back(ucx1);
+ extra_params2.backends.push_back(ucx2);
+
+ // Allocate and register memory buffers
+ nixl_reg_dlist_t dlist1(DRAM_SEG), dlist2(DRAM_SEG);
+ size_t buffer_size = 1024;
+
+ void* addr1 = malloc(buffer_size);
+ void* addr2 = malloc(buffer_size);
+ memset(addr1, 0xaa, buffer_size);
+ memset(addr2, 0xbb, buffer_size);
+
+ nixlBlobDesc desc1, desc2;
+ desc1.addr = (uintptr_t)addr1;
+ desc1.len = buffer_size;
+ desc1.devId = 0;
+ dlist1.addDesc(desc1);
+
+ desc2.addr = (uintptr_t)addr2;
+ desc2.len = buffer_size;
+ desc2.devId = 0;
+ dlist2.addDesc(desc2);
+
+ A1.registerMem(dlist1, &extra_params1);
+ A2.registerMem(dlist2, &extra_params2);
+
+ // Step 4: Publish metadata to etcd
+ // sendLocalMD() publishes this agent's metadata to the etcd server.
+ // Remote agents can then discover it by name.
+ A1.sendLocalMD();
+ A2.sendLocalMD();
+
+ // Allow time for etcd to process the metadata
+ std::this_thread::sleep_for(std::chrono::seconds(5));
+
+ // Step 5: Fetch remote metadata from etcd
+ // fetchRemoteMD() retrieves the named agent's metadata from etcd.
+ // No manual serialization or side-channel needed.
+ A1.fetchRemoteMD("EtcdAgent2");
+ A2.fetchRemoteMD("EtcdAgent1");
+
+ // Step 6: Execute a transfer (same as non-etcd workflow)
+ size_t req_size = 8;
+
+ nixl_xfer_dlist_t src_descs(DRAM_SEG);
+ nixlBasicDesc src;
+ src.addr = (uintptr_t)(((char*)addr1) + 16);
+ src.len = req_size;
+ src.devId = 0;
+ src_descs.addDesc(src);
+
+ nixl_xfer_dlist_t dst_descs(DRAM_SEG);
+ nixlBasicDesc dst_desc;
+ dst_desc.addr = (uintptr_t)(((char*)addr2) + 8);
+ dst_desc.len = req_size;
+ dst_desc.devId = 0;
+ dst_descs.addDesc(dst_desc);
+
+ nixlXferReqH *req_handle;
+ extra_params1.notif = "notification";
+
+ A1.createXferReq(NIXL_WRITE, src_descs, dst_descs,
+ "EtcdAgent2", req_handle, &extra_params1);
+
+ nixl_status_t status = A1.postXferReq(req_handle);
+
+ // Step 7: Wait for completion
+ nixl_notifs_t notif_map;
+ int n_notifs = 0;
+
+ while (status != NIXL_SUCCESS || n_notifs == 0) {
+ if (status != NIXL_SUCCESS)
+ status = A1.getXferStatus(req_handle);
+ if (n_notifs == 0)
+ A2.getNotifs(notif_map);
+ n_notifs = notif_map.size();
+ }
+
+ std::cout << "Transfer verified" << std::endl;
+
+ // Step 8: Clean up
+ A1.releaseXferReq(req_handle);
+ A1.deregisterMem(dlist1, &extra_params1);
+ A2.deregisterMem(dlist2, &extra_params2);
+
+ // Invalidate metadata in etcd so other agents know we're gone
+ A1.invalidateLocalMD();
+
+ free(addr1);
+ free(addr2);
+
+ std::cout << "Example completed." << std::endl;
+}
+```
+
+```python title="Python"
+# The Python basic_two_peers.py example works with etcd when the
+# NIXL_ETCD_ENDPOINTS environment variable is set. The Python bindings
+# handle etcd metadata exchange transparently -- no code changes needed.
+#
+# To run the basic transfer example with etcd:
+
+# Terminal 1 (target):
+# NIXL_ETCD_ENDPOINTS=http://localhost:2379 \
+# python basic_two_peers.py --mode target --ip 127.0.0.1
+
+# Terminal 2 (initiator):
+# NIXL_ETCD_ENDPOINTS=http://localhost:2379 \
+# python basic_two_peers.py --mode initiator --ip 127.0.0.1
+
+# When NIXL_ETCD_ENDPOINTS is set, the agent automatically publishes
+# metadata to etcd and fetches remote metadata from etcd instead of
+# using direct TCP connections for metadata exchange.
+```
+
+
+
+No Rust etcd example is currently available. The Rust bindings support etcd when `NIXL_ETCD_ENDPOINTS` is set in the environment, following the same transparent behavior as Python.
+
+
+**Expected output:**
+
+```
+NIXL Etcd Metadata Example
+==========================
+
+1. Sending local metadata to etcd...
+
+2. Fetching remote metadata from etcd...
+Transfer verified
+
+Example completed.
+```
+
+
+For full etcd setup instructions including server deployment, namespace configuration, and connection tuning, see [Metadata Exchange with etcd](../user-guide/etcd-metadata-exchange).
+
diff --git a/docs/examples/gds-direct-storage.md b/docs/examples/gds-direct-storage.md
new file mode 100644
index 0000000000..ece55aece6
--- /dev/null
+++ b/docs/examples/gds-direct-storage.md
@@ -0,0 +1,176 @@
+---
+title: GDS Direct Storage
+description: Transfer data directly between GPU memory and file storage using NIXL's GPUDirect Storage (GDS) backend, bypassing the CPU.
+---
+
+This example is taken from the `examples/` directory in the [NIXL repository](https://github.com/ai-dynamo/nixl), annotated with inline explanations.
+
+**What you'll learn:** How to use NIXL's GPUDirect Storage (GDS) backend to transfer data directly between GPU memory and file storage, bypassing the CPU.
+
+This example demonstrates writing data from a DRAM buffer to a file using the GDS backend, then reading it back into a second buffer for verification. GDS enables direct data paths between storage and memory, reducing CPU overhead for large data transfers.
+
+The four phases below -- initialization, write transfer, read transfer, and verification -- cover the complete GDS direct storage lifecycle. Each phase includes a sequence diagram followed by a code walkthrough.
+
+### Initialization
+
+
+GDS requires the cuFile library and a supported filesystem (ext4, XFS, or GDS-compatible). Only the Python GDS example is currently available -- no C++ or Rust variants exist.
+
+
+**Expected output:**
+
+```
+Initiator done
+Initiator done
+Test Complete.
+```
+
+
+For `CUFILE_ENV_PATH_JSON` and other GDS configuration, see [Environment Variables](../resources/environment-variables#gds-gpudirect-storage).
+
diff --git a/docs/examples/nixl-ep.md b/docs/examples/nixl-ep.md
new file mode 100644
index 0000000000..f81dc6c3e7
--- /dev/null
+++ b/docs/examples/nixl-ep.md
@@ -0,0 +1,167 @@
+---
+title: "NIXL-EP: Expert-Parallel Communication"
+description: Expert-parallel communication for Mixture of Experts (MoE) models built on NIXL's device API with elastic scaling, RDMA and NVLink support.
+---
+
+NIXL EP is a complete example implementation of expert-parallel communication for Mixture of Experts (MoE) models built on top of [NIXL](https://github.com/ai-dynamo/nixl)'s device API. It provides elastic scaling capabilities, enabling dynamic addition and removal of processes (ranks) during runtime without disrupting existing connections. Source: [`examples/device/ep/`](https://github.com/ai-dynamo/nixl/tree/main/examples/device/ep)
+
+## Features
+
+- **Dispatch and Combine support**: Dispatch and combine operations for MoE inference
+- **RDMA and NVLink support**: Utilizes NIXL's abstractions for both RDMA and NVLink transports
+- **Elastic Scaling**: Dynamically add or remove ranks during runtime
+
+## Building NIXL-EP
+
+Configure pkg-config paths (only needed when dependencies are installed to non-default paths):
+
+```bash
+export PKG_CONFIG_PATH=/lib/pkgconfig:$PKG_CONFIG_PATH
+export PKG_CONFIG_PATH=/lib/pkgconfig:$PKG_CONFIG_PATH
+export PKG_CONFIG_PATH=/lib/x86_64-linux-gnu/pkgconfig:$PKG_CONFIG_PATH
+```
+
+Set the NIXL plug-in directory and library path:
+
+```bash
+export NIXL_PLUGIN_DIR=/lib/x86_64-linux-gnu/plugins
+export LD_LIBRARY_PATH=/lib:$LD_LIBRARY_PATH
+```
+
+Build and install with NIXL-EP enabled:
+
+```bash
+meson setup build \
+ -Ducx_path= \
+ -Dprefix= \
+ -Dbuildtype=release \
+ -Dbuild_nixl_ep=true
+
+cd build
+ninja install
+```
+
+Add NIXL-EP to your Python path:
+
+```bash
+export PYTHONPATH=/examples/device/ep
+```
+
+## Python API
+
+```python title="Python"
+import nixl_ep
+
+# Initialize buffer with dynamic rank support
+buffer = nixl_ep.Buffer(rank=rank, explicitly_destroy=True)
+buffer.update_memory_buffers(num_ranks, num_experts_per_rank, num_rdma_bytes)
+buffer.connect_ranks(initial_ranks)
+
+# Dispatch tokens to experts
+recv_x, recv_count, handle, event, hook = buffer.dispatch(
+ x, topk_idx, num_max_dispatch_tokens_per_rank, num_experts
+)
+
+# ... process tokens through experts ...
+
+# Combine results back
+combined_x, event, hook = buffer.combine(
+ x, topk_idx, topk_weights, handle
+)
+
+# Later: Connect new ranks dynamically (elastic scaling)
+buffer.connect_ranks(new_ranks)
+
+# Disconnect ranks when scaling down
+buffer.disconnect_ranks(departing_ranks)
+
+# Explicit cleanup
+buffer.destroy()
+```
+
+### Key Methods
+
+| Method | Description |
+|--------|-------------|
+| `Buffer(disable_ll_nvlink=False, explicitly_destroy=False, rank=0, group=None, comm=None, tcp_store_group=None)` | Initialize the NIXL communication buffer |
+| `update_memory_buffers(num_ranks, num_experts_per_rank, num_rdma_bytes)` | Allocate remote memory for communication |
+| `connect_ranks(remote_ranks)` | Establish NIXL connections to new peers (can be called multiple times) |
+| `disconnect_ranks(remote_ranks)` | Clean up connections to departing peers |
+| `dispatch(x, topk_idx, num_max_dispatch_tokens_per_rank, num_experts, ...)` | Low-latency dispatch with NIXL device API |
+| `combine(x, topk_idx, topk_weights, handle, ...)` | Low-latency combine (reduce with weights) via NIXL device API |
+| `get_rdma_size_hint(num_max_dispatch_tokens_per_rank, hidden, num_ranks, num_experts)` | Get minimum RDMA buffer size requirement |
+| `clean_buffer(num_max_dispatch_tokens_per_rank, hidden, num_experts)` | Zero-initialize buffer (required before reuse) |
+| `destroy()` | Explicitly release resources (when `explicitly_destroy=True`) |
+
+## C++ Core
+
+The C++ implementation in `csrc/` provides the high-performance backend:
+
+- `nixl_ep.hpp` / `nixl_ep.cpp` -- Core Buffer implementation with Transfer Agent management
+- `kernels/` -- CUDA kernels for dispatch and combine operations
+- `config.hpp` -- Configuration constants
+- `event.hpp` -- CUDA event handling
+
+The C++ core is exposed to Python via pybind11. Direct C++ usage requires building with meson and linking against the NIXL library.
+
+## Elastic Scaling
+
+NIXL-EP supports dynamic addition and removal of ranks during runtime. The `connect_ranks()` method can be called multiple times to add new peers, and `disconnect_ranks()` removes departing peers.
+
+Example scaling plan from the test suite:
+
+```json
+[
+ [0, 1, 2, 3],
+ [0, 1, 2, 3, 4, 5, 6, 7],
+ [0, 1, 2, 3, 4, 5]
+]
+```
+
+- **Phase 0**: Initial state with ranks 0-3
+- **Phase 1**: Ranks 4-7 added dynamically
+- **Phase 2**: Ranks 6-7 removed dynamically
+
+## Testing
+
+The elastic test suite in `tests/elastic/` validates dynamic scaling capabilities.
+
+Single-node example (8 ranks, 4 to 8 expansion):
+
+```bash
+python3 tests/elastic/elastic.py \
+ --plan tests/elastic/single_expansion.json \
+ --num-processes 8
+```
+
+Multi-node example with 2 nodes:
+
+**Node 1** (launches the first phase with 4 ranks):
+
+```bash
+python3 tests/elastic/elastic.py \
+ --plan tests/elastic/single_expansion.json \
+ --num-processes 4
+```
+
+**Node 2** (joins the second phase with additional 4 ranks):
+
+```bash
+python3 tests/elastic/elastic.py \
+ --plan tests/elastic/single_expansion.json \
+ --num-processes 4 \
+ --tcp-server $MASTER_IP
+```
+
+### Available Test Plans
+
+| Plan File | Description |
+|-----------|-------------|
+| `no_expansion.json` | Static 4 ranks (baseline) |
+| `single_expansion.json` | 4 to 8 ranks (single expansion) |
+| `double_expansion.json` | 4 to 6 to 8 ranks (two expansions) |
+| `expansion_contraction.json` | 4 to 8 to 6 ranks (scale up then down) |
+
+
+NIXL-EP builds on the [NIXL Device API](../api-reference/device-api). For general NIXL concepts, see [Architecture](../getting-started/architecture).
+
diff --git a/docs/examples/remote-storage.md b/docs/examples/remote-storage.md
new file mode 100644
index 0000000000..a7cf3c4ecc
--- /dev/null
+++ b/docs/examples/remote-storage.md
@@ -0,0 +1,238 @@
+---
+title: Remote Storage Transfer
+description: A client/server storage transfer system using NIXL with POSIX and GDS backends for local and remote storage operations.
+---
+
+This example demonstrates a high-performance storage transfer system built on NIXL that supports both local and remote storage operations. It uses POSIX and GDS (GPUDirect Storage) backends with UCX-based networking. Source: [`examples/python/remote_storage_example/`](https://github.com/ai-dynamo/nixl/tree/main/examples/python/remote_storage_example)
+
+## Features
+
+- **Flexible Storage Backends**: GDS for high-performance, POSIX fallback, automatic selection
+- **Transfer Modes**: Local and remote memory-to-storage, bidirectional READ/WRITE, batch processing
+- **Network Communication**: UCX-based data transfer, metadata exchange, async notifications
+
+## Overview
+
+The system operates in two modes. The **server** waits for requests from clients to READ/WRITE from its storage to a remote node. The **client** initiates transfers and performs both local and remote operations with storage servers.
+
+The four phases below -- initialization, metadata exchange, remote write, and remote read -- cover the complete remote storage transfer lifecycle. Each phase includes a sequence diagram followed by a code walkthrough.
+
+
+In production, initialization and metadata exchange happen once at startup. Only the transfer phases (write/read) repeat per request.
+
+
+### Initialization
+
+ " per line)
+my_agent.send_local_metadata(server_ip, server_port)
+my_agent.fetch_remote_metadata(server_name, server_ip, server_port)
+
+# Poll until metadata is available
+while my_agent.check_remote_metadata(server_name) is False:
+ time.sleep(1.0)
+```
+
+### Remote Write Request
+
+ \
+ --fileprefix \
+ --name \
+ [--buf_size ] \
+ [--batch_size ]
+```
+
+The `--agents_file` is a list of storage servers the client connects to. The file should have agents separated by line, with ` ` on each line.
+
+The `--fileprefix` specifies a path to run local storage transfers on. The `--name` sets the Transfer Agent name for this client.
+
+### Running as Server
+
+```bash
+python nixl_p2p_storage_example.py --role server \
+ --fileprefix \
+ --name
+
+
+
+
+
+
+
+
+
+
+
+
+### nixlBasicDesc
+
+Base descriptor for a single contiguous memory region.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `addr` | `uintptr_t` | Start address of the buffer |
+| `len` | `size_t` | Length of the buffer in bytes |
+| `devId` | `uint64_t` | Device ID, block ID, or file ID |
+
+```cpp
+nixlBasicDesc desc(reinterpret_cast
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Both agents are created with configuration for progress threads and notifications. The target holds source data (ones), while the initiator starts empty (zeros). Both register their memory regions with NIXL so the UCX backend can pin them for RDMA.
+
+### Metadata & Descriptor Exchange
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The initiator fetches the target's metadata via direct TCP, then the target serializes its transfer descriptors and sends them to the initiator using NIXL's notification system. The initiator polls for and deserializes the received descriptors.
+
+### Transfer Execution
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The initiator creates a READ transfer request to pull data from the target's tensor. The transfer is posted asynchronously and the initiator polls for completion. On success, a "Done_reading" notification is sent to the target.
+
+### Teardown
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The initiator removes the remote agent reference, releases the transfer handle, and invalidates metadata. Both agents deregister their memory regions and are destroyed.
+
+### Code
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The `NIXL_ETCD_ENDPOINTS` environment variable enables etcd mode -- no code changes needed. Both agents are created with UCX backends and register DRAM memory, identical to the non-etcd workflow.
+
+### Publish Metadata to etcd
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Each agent serializes its memory descriptors and UCX connection info, then publishes the blob to etcd under `/nixl/agents/
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Each agent fetches the other's metadata from etcd by name. The metadata is loaded into local remote sections for transfer use. NIXL automatically creates a persistent watcher on the remote agent's etcd key to detect disconnections.
+
+### Transfer
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The transfer is identical to the non-etcd workflow -- etcd is only used for metadata exchange. The cached metadata enables direct RDMA transfers between agents.
+
+### Invalidation & Teardown
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+When an agent goes offline, `invalidateLocalMD()` deletes its etcd key. This triggers a DELETE event on watchers, causing remote agents to automatically discard cached metadata.
+
+### Code
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+A single agent is created with the GDS backend. Two DRAM buffers are allocated -- buf1 filled with a 0xba test pattern, buf2 left empty for read-back. Both DRAM buffers and a file descriptor are registered with NIXL.
+
+### Write Transfer (DRAM to File)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+A WRITE transfer moves data from DRAM buf1 directly to the file via the GDS backend, bypassing the CPU. The transfer is posted asynchronously and polled for completion.
+
+### Read Transfer (File to DRAM)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+A READ transfer moves data from the file back into DRAM buf2 via GDS. This completes the round-trip needed for verification.
+
+### Verify & Teardown
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The round-trip transfer is verified by comparing buf1 and buf2 (both should contain the 0xba pattern). Transfer handles are released, memory is deregistered, buffers are freed, and the file is closed.
+
+### Code
+
+```python title="Python"
+import os
+import nixl._utils as nixl_utils
+from nixl._api import nixl_agent, nixl_agent_config
+
+# Step 1: Create an agent and initialize the GDS backend
+# The GDS backend must be available (requires cuFile library)
+agent_config = nixl_agent_config(backends=[])
+agent = nixl_agent("GDSTester", agent_config)
+
+# Verify GDS plugin is available
+plugin_list = agent.get_plugin_list()
+assert "GDS" in plugin_list, "GDS plugin not available"
+
+agent.create_backend("GDS")
+
+# Step 2: Allocate DRAM buffers
+# buf1 holds the source data (filled with 0xba pattern)
+# buf2 is the destination for read-back verification
+buf_size = 16 * 4096
+addr1 = nixl_utils.malloc_passthru(buf_size)
+addr2 = nixl_utils.malloc_passthru(buf_size)
+nixl_utils.ba_buf(addr1, buf_size) # Fill buf1 with 0xba
+
+# Step 3: Register DRAM memory regions with the agent
+agent_strings = [(addr1, buf_size, 0, "a"), (addr2, buf_size, 0, "b")]
+agent_reg_descs = agent.get_reg_descs(agent_strings, "DRAM")
+
+xfer1_descs = agent.get_xfer_descs([(addr1, buf_size, 0)], "DRAM")
+xfer2_descs = agent.get_xfer_descs([(addr2, buf_size, 0)], "DRAM")
+
+agent.register_memory(agent_reg_descs)
+
+# Step 4: Register file-backed memory
+# Open (or create) a file and register it as a FILE memory region
+# The GDS backend handles direct storage I/O
+file_path = "/path/to/gds_test_file"
+fd = os.open(file_path, os.O_RDWR | os.O_CREAT)
+
+file_list = [(0, buf_size, fd, "b")]
+file_descs = agent.register_memory(file_list, "FILE")
+xfer_files = file_descs.trim()
+
+# Step 5: Write DRAM buffer to file via GDS
+# WRITE transfers data from the first descriptor list to the second
+xfer_handle_write = agent.initialize_xfer(
+ "WRITE", xfer1_descs, xfer_files, "GDSTester"
+)
+
+state = agent.transfer(xfer_handle_write)
+while True:
+ state = agent.check_xfer_state(xfer_handle_write)
+ if state == "ERR":
+ raise RuntimeError("Write transfer failed")
+ elif state == "DONE":
+ break
+
+# Step 6: Read file data back into the second DRAM buffer
+# READ transfers data from the file back into DRAM for verification
+xfer_handle_read = agent.initialize_xfer(
+ "READ", xfer2_descs, xfer_files, "GDSTester"
+)
+
+state = agent.transfer(xfer_handle_read)
+while True:
+ state = agent.check_xfer_state(xfer_handle_read)
+ if state == "ERR":
+ raise RuntimeError("Read transfer failed")
+ elif state == "DONE":
+ break
+
+# Step 7: Verify the round-trip transfer
+# buf2 should now contain the same 0xba pattern as buf1
+nixl_utils.verify_transfer(addr1, addr2, buf_size)
+
+# Step 8: Clean up
+agent.release_xfer_handle(xfer_handle_write)
+agent.release_xfer_handle(xfer_handle_read)
+agent.deregister_memory(agent_reg_descs)
+agent.deregister_memory(file_descs)
+
+nixl_utils.free_passthru(addr1)
+nixl_utils.free_passthru(addr2)
+os.close(fd)
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Both nodes create a NIXL agent, register storage backends (GDS_MT preferred, POSIX fallback), and register a UCX backend for network transfers. The client registers VRAM segments (GPU memory) and file descriptors. The server registers DRAM segments and file descriptors.
+
+```python title="Python"
+# Both client and server
+my_agent = nixl_agent(agent_name, agent_config)
+
+# Register storage + network backends
+my_agent.create_backend("GDS_MT") # or "POSIX" as fallback
+my_agent.create_backend("UCX")
+
+# Client: VRAM + FILE, Server: DRAM + FILE
+nixl_mem_reg_descs = my_agent.register_memory(mem_list, "VRAM") # or "DRAM"
+nixl_file_reg_descs = my_agent.register_memory(file_list, "FILE")
+```
+
+### Metadata Exchange
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The client reads a list of storage servers from a file and connects to each one. For each server, the client publishes its own metadata and fetches the server's metadata, then polls until the exchange completes.
+
+```python title="Python"
+# For each server in agents_file ("
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The client serializes its VRAM descriptors and sends a `WRTE` notification to the server. The server deserializes the descriptors and executes a pipelined loop: **network read** (UCX read from client VRAM into server DRAM) followed by **storage write** (GDS/POSIX write from DRAM to local file). These two operations overlap across iterations for throughput. On completion, the server sends a `COMPLETE` notification back.
+
+```python title="Python"
+# Client: send write request
+descs_str = my_agent.get_serialized_descs(my_mem_descs)
+my_agent.send_notif(server_name, b"WRTE" + iterations_str + descs_str)
+
+# Client: wait for completion
+while not my_agent.check_remote_xfer_done(server_name, b"COMPLETE"):
+ continue
+```
+
+```python title="Python"
+# Server: pipelined write handler (network read → storage write)
+sent_descs = my_agent.deserialize_descs(received_data)
+
+# Each iteration: read from client network, then write to local storage
+# Operations are pipelined across iterations using thread pool
+handle = my_agent.initialize_xfer("READ", dram_descs, client_vram_descs, client_name)
+my_agent.transfer(handle) # network read
+
+handle = my_agent.initialize_xfer("WRITE", dram_descs, file_descs, self_name)
+my_agent.transfer(handle) # storage write
+
+my_agent.send_notif(client_name, b"COMPLETE")
+```
+
+### Remote Read Request
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The client sends a `READ` notification to the server. The server executes the reverse pipeline: **storage read** (GDS/POSIX read from local file into server DRAM) followed by **network write** (UCX write from server DRAM to client VRAM). Again, operations overlap across iterations. On completion, the server notifies the client.
+
+```python title="Python"
+# Client: send read request
+descs_str = my_agent.get_serialized_descs(my_mem_descs)
+my_agent.send_notif(server_name, b"READ" + iterations_str + descs_str)
+
+# Client: wait for completion
+while not my_agent.check_remote_xfer_done(server_name, b"COMPLETE"):
+ continue
+```
+
+```python title="Python"
+# Server: pipelined read handler (storage read → network write)
+sent_descs = my_agent.deserialize_descs(received_data)
+
+# Each iteration: read from local storage, then write to client network
+# Operations are pipelined across iterations using thread pool
+handle = my_agent.initialize_xfer("READ", dram_descs, file_descs, self_name)
+my_agent.transfer(handle) # storage read
+
+handle = my_agent.initialize_xfer("WRITE", dram_descs, client_vram_descs, client_name)
+my_agent.transfer(handle) # network write
+
+my_agent.send_notif(client_name, b"COMPLETE")
+```
+
+### Pipelining
+
+To improve throughput, the server pipelines storage and network operations across iterations. While one iteration's network transfer is in flight, the next iteration's storage operation begins concurrently using a thread pool.
+
+#### Read Pipeline (Storage Read → Network Write)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+#### Write Pipeline (Network Read → Storage Write)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The pipeline is implemented using a `ThreadPoolExecutor` with two workers -- one for storage and one for network. The first and last iterations are special-cased (no overlap), while middle iterations submit both operations concurrently:
+
+```python title="Python"
+with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
+ # Middle iterations: storage and network run in parallel
+ executor.submit(execute_transfer, agent, dram, file, self, "READ") # storage
+ executor.submit(execute_transfer, agent, dram, vram, client, "WRITE") # network
+```
+
+## Usage
+
+### Running as Client
+
+```bash
+python nixl_p2p_storage_example.py --role client \
+ --agents_file
+
+