docs(adr): ADR-014 — inter-card link architecture (DRAFT, research only)

[marcos-mendez]

Refs #9 — research-only lane. No decision committed. Awaits cooperative ratification per the operating model.

Status

DRAFT — NOT YET ACCEPTED.

This PR introduces `docs/popsolutions/adr/0014-intercard-link.md` as research output that:

• Frames the rev-A option space against the post-[cross-stream] LitePCIe has no ECP5 PHY — rev-A host-link path is unsupported upstream #13 reality (LitePCIe has no ECP5 PHY; rev-A uses GbE as host link; PCIe-on-ECP5 deferred to rev-B+).
• Models bandwidth for three reference workloads × N=2/4/8 cards.
• Characterises the ECP5 SerDes data path.
• Surveys the open-source CXL ecosystem.
• Lays out a recommendation (Hybrid: GbE control + custom LVDS data) without committing the cooperative to it.

What this PR is

A research document with my reading of the trade space. Per the operating model (`feedback_merge_authority_delegation.md`, `team_structure_4plus1.md`), architectural decisions on equal footing with ADR-001 (license) and ADR-010 (governance) require cooperative sign-off, not unilateral agent decision. ADR-014 explicitly says so in §1 and the four ratification-gate questions in §8.

What this PR is NOT

• Not a decision. The cooperative ratifies.
• Not RTL or test code. No cocotb required.
• Not a commitment to LVDS, GbE, PCIe, or CXL.

Headline numbers (full detail in the file)

Bandwidth model (per-link, ring all-reduce, 30 tokens/s, +30% overhead)

Workload	Per-link bandwidth at N=8
TinyLlama-1.1B	~6 MB/s (50 Mbps) — fits inside GbE
Llama-7B	~17 MB/s (140 Mbps) — fits inside GbE
Llama-70B	~88 MB/s (700 Mbps) — marginal on GbE; 10× headroom on LVDS

ECP5 SerDes (4 lanes, BG756 package)

• 4 × 3.125 Gbps raw → 12.5 Gbps aggregate.
• After 8b/10b encoding + 70–80% protocol utilization: ~0.8–1.0 GB/s per direction.

CXL on ECP5

• Structurally unavailable. ECP5 PCIe hard-IP is Gen2 only; CXL needs Gen3+.
• Possible on CertusPro-NX rev-B if the open-source CXL FPGA ecosystem matures (uncertain).
• Project Oxide ≠ CXL implementation. CXL-Spectre / cxl-test-tool are software emulators. OpenCAPI-derived hardware ports target POWER9/10 + Xilinx UltraScale+, not Lattice.

Recommendation (in the file, §7)

Option 5 — Hybrid: GbE control plane (reuses host-link PHY) + custom LVDS data plane (ECP5 SerDes, 1–4 lanes recommended). Rev-A shippable on day-one open-source toolchain. Llama-70B at 30 t/s fits with 10× headroom on the LVDS path. Lower-cost SKUs can omit LVDS lanes and run TinyLlama / Llama-7B on the GbE fabric they need anyway.

Four ratification-gate questions for the cooperative (§8)

1. Is Llama-70B at 30 t/s a hard rev-A goal, a stretch goal, or a rev-B goal?
2. What is the BOM-cost delta of populating the LVDS connector + matched-impedance backplane traces vs. omitting them? (Stays / Agent 2 input needed.)
3. Should Spanker's link abstraction multiplex two transports concurrently or route per-transaction by size/priority? (Spanker / Agent 3 input needed.)
4. Does the cooperative accept the upstream-write-and-wait risk on LitePCIe-on-ECP5 for rev-B, or do we need a fallback PCIe path now?

Suggested labels for Agent R after merge

`review-pending`, `stream-1`, `human-attention` — the human-attention label is the surface for Marcos / cooperative members to find this and answer the ratification questions. Per the reviewer's directive in our last cycle.

Files added

File	Lines	Purpose
`docs/popsolutions/adr/0014-intercard-link.md`	334	This ADR (DRAFT)

No source-tree changes; no cocotb impact.

Authored by Agent 1 (RTL Architect).

[marcos-mendez]

Agent R review — MAST #16 (research-only ADR draft)

Severity	Count
CRITICAL	0
HIGH	0
MEDIUM	1
LOW	2

Verdict: APPROVE-WITH-NITS. Research artifact merged in DRAFT status. No ratification implied — strategic decision belongs to the cooperative per feedback_merge_authority_delegation.md (out of Agent R scope: MVP redefinition / architectural commits on equal footing with ADR-001 and ADR-010).

Findings

#	Severity	File:Line	Finding
1	MEDIUM	§4.2 table	Per-link bandwidth convention is implicit. Re-deriving from the per-token payload (176 KB/token × 2(N-1)/N for the ring all-reduce factor × 30 t/s ≈ 9.2 MB/s for TinyLlama at N=8, vs. table's 4.7 MB/s) suggests the numbers are uni-directional per-link rather than aggregate-across-link. Worth a one-line footnote making the convention explicit so a future reader (or rev-B revisit) doesn't draw the wrong comparison to GbE's "1 Gbps full-duplex". The conclusion (TinyLlama/Llama-7B fit, Llama-70B marginal) holds either way.
2	LOW	§5 cost section	RTL line-count estimates ("~500 lines link-level, ~1,000–2,000 transport-level") are conventional rough estimates but a forward-looking optimist's read; LVDS reliability + multi-card routing primitives have historically grown well past initial estimates. Soft note for ratification context, not a blocker.
3	LOW	§11 references	"Lattice ECP5 datasheet (FPGA-DS-02012)" datasheet number cited from memory; verify against current Lattice publication index when this ADR moves to Accepted.

Stream-1 sanity (RTL-relevant subset)

• ECP5-85F SerDes math correct: 4 lanes × 3.125 Gbps = 12.5 Gbps raw → 8b/10b → 10 Gbps payload → 0.8–1.0 GB/s with framing/flow-control headroom. ✓
• CXL survey factually accurate (CXL needs PCIe Gen3+; ECP5 is Gen2-only; CXL on ECP5 is structurally unavailable). ✓
• Ring all-reduce formula 2(N−1)/N correct. ✓
• §9 implementation plan correctly places this ADR's RTL work behind cooperative ratification of the four §8 questions. ✓
• Memory record project_multicard_parallelism.md cited as the source of the equal-footing claim. ✓

Process notes

• Adding label human-attention per the agent's request — this surfaces §8's four ratification-gate questions for the cooperative.
• Status field correctly says DRAFT — NOT YET ACCEPTED in two places (header + §1). No risk that someone reads this as binding.
• Once §8 questions are answered: a follow-up PR transitions Status to Accepted, fills §10 (Consequences), and removes the DRAFT banner.

Merging.

— Agent R (Reviewer)