Md/s4#1940
Draft
aschwartz12 wants to merge 4 commits into
Draft
Conversation
Introduce the metadata-exchange backend layer without wiring it in yet: the nixlMetadataBackend contract, the nixlMetadataContext interface (the agent-side operations a backend needs), and the self-contained P2P (socket) backend that implements the contract against the context. Nothing constructs these yet; the agent still uses its inline path. The manager that owns and dispatches to backends, and the cutover that removes the inline path, come in the following commit. Splitting the introduction out keeps that cutover reviewable on its own.
Wire the backend layer in and remove the inline path. nixlAgentData now owns a nixlMDManager (built unconditionally) that owns the worker thread and dispatches each call to a backend by precedence: a peer address selects P2P, otherwise the configured centralized store (ETCD). Add the self-contained ETCD backend (owns its nixlEtcdClient). nixlAgent's exchange methods (sendLocalMD, sendLocalPartialMD, fetchRemoteMD, invalidateLocalMD) become thin wrappers over the manager; pure-cache operations are consolidated on nixlAgentData, which implements nixlMetadataContext for the backends. nixl_listener.cpp is deleted: its worker loop moves into the manager, its socket helpers into the P2P backend, and its etcd client into the ETCD backend. Public API and observable P2P/ETCD behavior are unchanged; gtest log expectations are updated to the new backend messages.
Add md_manager.cpp covering the manager's routing and the P2P and ETCD backends (send/fetch/invalidate by peer address and by name). ETCD cases are gated on a live NIXL_ETCD_ENDPOINTS endpoint.
Add a TCPStore metadata backend as a centralized, name-addressed store alongside ETCD, selected via NIXL_TCPSTORE_ENDPOINTS and mutually exclusive with NIXL_ETCD_ENDPOINTS. It is self-contained: it owns an in-house c10d TCPStore wire-protocol client (nixl_tcpstore_client) and links no libtorch, yet interoperates with a torch.distributed.TCPStore server. Fetch semantics match ETCD: fetchRemoteMD returns immediately and the caller polls checkRemoteMD, so behavior is uniform across P2P, ETCD, and TCPStore. Adds gtest coverage gated on a live store.
|
👋 Hi aschwartz12! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR4 - feat(metadata): self-contained TCPStore backend
Branch
md/s4(8fd9b6a9), stacked on PR3 (md/s3). Final slice of the4-PR metadata-manager stack. Adds a TCPStore metadata backend as a centralized,
name-addressed store alongside ETCD, selected via
NIXL_TCPSTORE_ENDPOINTS.The tip of this branch (
md/s4) reproduces the reviewed metadata-manager endstate: same git tree hash as
md/rebasedandmd/tcpstore, with all 15 coremetadata files byte-identical to
metadata_manager_pr5.What?
Adds a third metadata backend and its in-house c10d wire-protocol client:
nixl_tcpstore_metadata_backend.{cpp,h}— TCPStore backendnixl_tcpstore_client.{cpp,h}— hand-rolled c10d TCPStore client (no libtorch)nixl_md_manager.{cpp,h},nixl_agent.cpp, andnixl_metadata_backend.hfor backend selectiontest/gtest/md_manager.cppWhy?
Provides a lightweight centralized store for environments that already run or
prefer TCPStore, without pulling in a full etcd/gRPC dependency. Because PR1–PR2
established the backend contract and manager, this is one new backend class plus
its client and a selection branch — no further agent-side or public-API changes.
How?
Backend -
nixlTcpStoreMetadataBackendimplementsnixlMetadataBackendbehindthe same
prepare*methods as P2P/ETCD. Self-contained; synchronous store I/Oon the manager's worker.
Client -
nixl_tcpstore_clientimplements the c10d TCPStore wire protocol(set/get/check/delete). Interoperates with a
torch.distributed.TCPStoreserver.Selection -
makeBackendselects TCPStore whenNIXL_TCPSTORE_ENDPOINTSis set.ETCD and TCPStore are mutually exclusive (setting both is an error).
Fetch semantics - uniform with ETCD:
fetchRemoteMDreturns immediately and thecaller polls
checkRemoteMD.Scope - Additive. No change to P2P/ETCD behavior or the public API.
Size
~802 lines added across 10 files. Under the 1000-line target.
Test plan
MDManagerTcpStoreFixtureskips cleanly withoutNIXL_TCPSTORE_ENDPOINTS;P2P/ETCD suites are unaffected.
torch.distributed.TCPStoremaster:NIXL_TCPSTORE_ENDPOINTS=host:port ./builddir/test/gtest/nixl_gtest --gtest_filter=MDManagerTcpStoreFixture*Equivalence note
md/s4tree hash matchesmd/rebasedexactly. All core metadata manager filesmatch
metadata_manager_pr5; only upstream-drift hunks differ inmeson.build(version
1.4.0) and non-metadata changes innixl_agent.cpp(telemetry,prepMemViewlock) because this stack is rebased on currentnixl/main.Stack
md/s3)