feat(platform): add heartbeat module for replica health monitoring

.agents/skills/README.md

@@ -8,3 +8,13 @@ tooling without maintaining duplicate copies.
 
 Add or update skill directories here. Each skill directory must contain its own
 `SKILL.md`.
+
+## Skills
+
+| Skill | Description |
+|---|---|
+| `create-feature-request` | File a GitHub feature request following the repo template |
+| `create-issue` | File a GitHub bug or docs issue following the repo template |
+| `rebase-on-main` | Rebase the current feature branch on latest main and force-push, resolving conflicts by preserving the intent of both sides |
+| `retrigger-ci` | Retry a failing CI run until it passes when the failure is flaky and unrelated to our changes |
+| `platform-style-fix` | Check and fix clang-format style violations in `platform/include` and `platform/examples` |

platform/docs/README.md

@@ -22,21 +22,28 @@ platform/
   build/                            generated build output; not documented here
 ```
 
-## Module Areas
+## Documentation
 
+- [Architecture](architecture.md): instance roles, module system, channel model, and scope.
+- [Model Support Quick-Start](model-support-quickstart.md): wire your inference kernel into the coordinator→replica path, understand the `processFc` interface, and benchmark framework overhead.
+- [Simpler Integration](simpler-integration.md): how replica ranks launch Simpler, the `processFc` interface, and the path to in-device tensor channels.
+- [Model Support Interface](model-support-interface.md): what Model Support owns, provides, and will consume from the platform.
 - [System](system/README.md): engine lifecycle and cross-instance start/stop control.
 - [Channels](system/channels.md): payload, coordination, metadata, input, output, and message primitives.
-
-Configuration types, the service module, channel controller, and broadcast deployment are tracked in issue #32 and are not part of this initial PR.
+- [Modules](modules/README.md): configuration, service, channel controller, and broadcast deployment.
 
 ## Runtime Shape
 
 The current platform runtime is built around `serving::system::Engine`. The engine owns a set of `serving::modules::Module` instances, initializes them, starts them across instances through RPC, waits for termination, and finalizes them.
 
-This initial PR covers the following building blocks:
+This PR covers the following building blocks:
 
 - Engine lifecycle: cross-instance start/stop over RPC (`serving::system::Engine`).
 - Module base interface: initialize/run/terminate/await/finalize lifecycle with optional `taskr::Service` (`serving::modules::Module`).
 - Channel primitives: `Input`, `Output`, `Message`, and `MessageTypeRegistry` for host-side control traffic.
+- Configuration types: `Deployment`, `Partition`, `Task`, `Edge`, `Replica`, `RequestManager` with JSON serialization and verification.
+- `broadcastDeployment`: deployer-to-worker deployment config distribution over RPC.
+- `channelController`: desired-vs-actual reconciliation loop for host-side SPSC channels.
+- `service`: wraps `taskr::Runtime` for cooperative background services.
 
-Deployment graph representation, deployment broadcast, desired-state channel creation, dynamic scaling, topology-aware replacement, fault recovery, and Python bindings are not implemented in this PR.
+Dynamic scaling, fault recovery, `channelDispatcher`, `taskScheduler`, executor roles, heartbeat, `RuntimePlan` update protocol, in-device tensor channels, and Python bindings are deferred to follow-up PRs.

platform/docs/architecture.md

@@ -0,0 +1,69 @@
+# Platform Architecture
+
+## What the platform layer is
+
+The platform layer is the **host-side control plane** for a distributed PyPTO Serving deployment. It manages:
+
+- Instance lifecycle (start, stop, health)
+- Deployment configuration distribution
+- Host-side channel creation and teardown
+- Module initialization and service scheduling
+
+It is not a model execution layer. Model kernels, tensor movement, KV cache, batching, and token scheduling belong to Model Support and are executed through Simpler on each device. The platform exists so that Model Support does not have to manage MPI ranks, HiCR channels, or distributed lifecycle directly.
+
+## Instance roles
+
+Every MPI rank in a deployment plays one of four roles, determined by its instance ID and the deployment configuration:
+
+| Role | Count | Responsibility |
+|---|---|---|
+| **Deployer** | 1 (root) | Reads deployment config, broadcasts it to all other ranks, then transitions to a coordinator or replica role |
+| **Coordinator** | one per partition | Routes jobs to replicas, accumulates outputs, fires completion callbacks |
+| **Replica** | one or more per partition | Executes model computation via Simpler on a dedicated NPU device |
+| **Request Manager** | 1 | Entry point for client requests; maps to the partition owning the user-interface edge |
+
+Device assignment for replicas: `deviceId = instanceId - numPartitions`. Each replica rank owns exactly one NPU.
+
+## Module system
+
+All platform behaviour is expressed as `serving::modules::Module` instances owned by the `serving::system::Engine`. The engine drives a fixed lifecycle:
+
+```
+initialize() → run() → [service loop] → terminate() → await() → finalize()
+```
+
+Modules can register a periodic background service (via `taskr::Service`) that runs inside the service loop between the `run()` and `terminate()` phases. The engine coordinates the lifecycle of all instances over RPC, so every rank progresses through the same phases in lockstep.
+
+### Modules in this PR
+
+| Module | Purpose |
+|---|---|
+| `broadcastDeployment` | Deployer sends deployment JSON to all workers over RPC at initialize time |
+| `channelController` | Desired-vs-actual reconciliation loop; creates and tears down HiCR SPSC channels |
+| `service` | Wraps `taskr::Runtime`; owns and drives background `taskr::Service` instances |
+
+### Deferred modules (next PRs)
+
+| Module | Purpose |
+|---|---|
+| `channelDispatcher` | Polls subscribed input channels; dispatches messages to registered handlers |
+| `taskScheduler` | Registers named `taskr::Task` instances; drives taskr through the module lifecycle |
+| `roles::coordinator` | Job queue and replica dispatch; completion callback when all outputs are gathered |
+| `roles::replica` | Receives coordinator input, invokes `processFc`, returns outputs |
+| `heartbeat` | Periodic health check between coordinators and replicas |
+
+## Channel model
+
+All channels in this PR are **host-side HiCR SPSC channels** carrying variable-size payloads and fixed-size metadata. They are used for control traffic and small tensors.
+
+In-device tensor channels — where the hot path (prefill/decode token data) moves directly between NPU devices without staging through host memory — are **not implemented here**. They are the next major milestone and unblock the TP/PP tensor data path.
+
+## What is deliberately out of scope
+
+- Per-token scheduling (hot path)
+- KV cache management
+- Batching and sampling policy
+- Python bindings
+- `RuntimePlan` update protocol and watch/subscribe API
+- Dynamic scaling, drain, and fault recovery
+- Topology-aware placement

platform/docs/model-support-interface.md

@@ -0,0 +1,74 @@
+# Model Support Interface
+
+This document describes what the Model Support layer needs to know and control when integrating with the platform.
+
+## What the platform tells you
+
+At startup, after `broadcastDeployment` completes, every rank knows:
+
+- **Its role**: coordinator, replica, or request manager — determined by instance ID and the deployment config
+- **Its device index** (replicas only): `deviceId = instanceId - numPartitions`
+- **The full deployment graph**: partitions, tasks, edges, and which edges carry user-interface traffic
+
+Model Support should use this information to initialize Simpler on replica ranks before the engine starts.
+
+## What you provide
+
+### 1. Compiled Simpler artifacts
+
+Replica ranks need the five runtime library paths in `RuntimeBinaries`. The platform does not locate or validate them. Pass them via command-line arguments or configuration alongside the deployment JSON.
+
+### 2. A `processFc` per task function name
+
+For each task declared in the deployment configuration, Model Support registers a function:
+
+```cpp
+std::function<void(serving::modules::roles::TaskContext &context)>
+```
+
+This is the only point where Model Support code runs during execution. Everything else — job routing, input aggregation, output forwarding, channel lifecycle — is handled by the platform.
+
+### 3. Request/response edge names
+
+The deployment config declares a `RequestManager` with an input edge name and an output edge name. Model Support must ensure:
+
+- The input edge name matches the edge from which user requests arrive
+- The output edge name matches the edge to which results are written
+- The task that reads the input edge does not have any other inter-partition inputs
+- The task that writes the output edge does not have any other inter-partition outputs
+
+These constraints are verified by `Deployment::verify()` at startup.
+
+## What you do not own
+
+| Concern | Owner |
+|---|---|
+| MPI rank management | Platform (via HiCR `InstanceManager`) |
+| Channel creation and teardown | Platform (`channelController` module) |
+| Deployment broadcast | Platform (`broadcastDeployment` module) |
+| Job routing from coordinator to replica | Platform (`coordinator::Module`) |
+| Input aggregation and output forwarding | Platform (`replica::Module`) |
+| Engine lifecycle (init/run/terminate) | Platform (`Engine`) |
+| Replica health monitoring | Platform (heartbeat, future) |
+
+## RuntimePlan — future interface
+
+The current PR does not yet expose a `RuntimePlan`. A future milestone will provide a versioned, observable snapshot of what the platform has actually instantiated:
+
+- Which replicas are live, draining, or unavailable
+- Which channels exist and what their endpoints are
+- Safe `active → draining → removed` state transitions before any channel is deleted
+
+Model Support will consume this object to make routing and scheduling decisions (KV-locality-aware replica selection, drain-aware request assignment, channel handle lookup) without duplicating platform resource state. The same object will be queryable by operators and higher-level control loops for observability.
+
+Until `RuntimePlan` is implemented, Model Support should treat the deployment configuration as the ground truth of what is running.
+
+## Tensor channel hot path — future
+
+All channels are currently host-side. For the prefill/decode token data path, in-device tensor channels will carry tensor payloads directly between NPU devices without staging through host memory. When implemented:
+
+- Replica `processFc` will receive device-memory `LocalMemorySlot` handles rather than host buffers
+- `toDevice` / `toHost` staging in the process function becomes unnecessary
+- The `processFc` signature does not change
+
+Model Support should design the process function to be forward-compatible: check whether the incoming `LocalMemorySlot` is a host or device buffer and branch accordingly.

platform/docs/model-support-quickstart.md

@@ -0,0 +1,168 @@
+# Model Support Quick-Start
+
+This guide shows how to wire a model's inference kernel into the platform coordinator→replica path, using the `singlePartition` benchmark as the reference example.
+
+## 1. The `processFc` interface
+
+The platform invokes your inference code through a single callback:
+
+```cpp
+using processFc_t = std::function<void(serving::modules::roles::TaskContext &)>;
+```
+
+Inside the callback:
+
+```cpp
+void myInferenceFn(serving::modules::roles::TaskContext &ctx)
+{
+  // Read input tensor
+  auto slot = ctx.getInput("tokens");   // returns shared_ptr<HiCR::LocalMemorySlot>
+  const auto *data = static_cast<const uint8_t *>(slot->getPointer());
+  size_t      size = slot->getSize();
+
+  // Run inference ...
+  std::vector<float> logits = runModel(data, size);
+
+  // Write output (platform takes a copy; the buffer can go out of scope after this call)
+  ctx.setOutput("logits", logits.data(), logits.size() * sizeof(float));
+}
+```
+
+`TaskContext` API summary:
+
+| Method | Description |
+|--------|-------------|
+| `getInput(name)` | Returns the named input slot; `nullptr` if the dependency was not satisfied |
+| `setOutput(name, ptr, size)` | Copies `size` bytes from `ptr` into the named output channel |
+| `setOutput(name, slot)` | Moves an already-allocated `LocalMemorySlot` into the output (zero-copy path) |
+
+## 2. Defining the data graph
+
+Each edge in the deployment graph carries one tensor per job. Configure capacity (concurrent jobs in flight) and payload size to match your model's I/O shapes:
+
+```cpp
+// One input edge: 4096-byte token buffer, up to 4 jobs in flight
+auto tokensEdge = std::make_shared<serving::configuration::Edge>(
+    "tokens", /*capacity=*/4, /*payloadBytes=*/4096);
+tokensEdge->setPayloadCommunicationManager(commMgr);
+tokensEdge->setPayloadMemoryManager(memMgr);
+tokensEdge->setPayloadMemorySpace(memSpace);
+tokensEdge->setCoordinationCommunicationManager(commMgr);
+tokensEdge->setCoordinationMemoryManager(memMgr);
+tokensEdge->setCoordinationMemorySpace(memSpace);
+
+// One output edge: 8-byte result (uint64_t logit checksum in the example)
+auto logitsEdge = std::make_shared<serving::configuration::Edge>(
+    "logits", /*capacity=*/4, sizeof(uint64_t));
+// ... same manager/space setters ...
+
+serving::configuration::Deployment deployment;
+deployment.addChannel(tokensEdge);
+deployment.addChannel(logitsEdge);
+```
+
+Add one task describing the I/O contract:
+
+```cpp
+auto task = std::make_shared<serving::configuration::Task>(std::string("infer"));
+task->addInput("tokens");
+task->addOutput("logits");
+
+auto partition = std::make_shared<serving::configuration::Partition>("P0", instanceId);
+partition->addTask(task);
+partition->addReplica(std::make_shared<serving::configuration::Replica>(instanceId));
+deployment.addPartition(partition);
+```
+
+## 3. Wiring coordinator and replica (co-located)
+
+For a single-rank deployment the coordinator and replica live on the same MPI rank. They communicate over loopback MPI channels. The internal channel IDs (`1000`, `2000` below) must not conflict with any other channels in the system.
+
+```cpp
+// Internal edges derived from the graph edge templates
+const auto internalTokensEdge = makeInternalEdgeFromTemplate(
+    "tokens-coord-replica", *deployment.getEdges()[0]);
+const auto internalLogitsEdge = makeInternalEdgeFromTemplate(
+    "logits-replica-coord", *deployment.getEdges()[1]);
+
+// Coordinator side: sends tokens, receives logits
+auto tokensOut = channelController->addDesiredProducer(
+    instanceId, /*channelId=*/1000, internalTokensEdge, defaultChannelKeyBuilder).lock();
+auto logitsIn  = channelController->addDesiredConsumer(
+    instanceId, /*channelId=*/2000, internalLogitsEdge, defaultChannelKeyBuilder).lock();
+
+coordinator->addReplica(instanceId,
+    /*inputs=*/ {{"tokens", tokensOut}},
+    /*outputs=*/{{"logits",  logitsIn}});
+
+// Replica side: receives tokens, sends logits
+auto tokensIn  = channelController->addDesiredConsumer(
+    instanceId, /*channelId=*/1000, internalTokensEdge, defaultChannelKeyBuilder).lock();
+auto logitsOut = channelController->addDesiredProducer(
+    instanceId, /*channelId=*/2000, internalLogitsEdge, defaultChannelKeyBuilder).lock();
+
+replica->setCoordinator(instanceId,
+    /*outputs=*/{{"logits",  logitsOut}},
+    /*inputs=*/ {{"tokens",  tokensIn}});
+```
+
+## 4. Submitting jobs
+
+After `serving.initialize()` and `serving.run()`, jobs are submitted directly to the coordinator. The platform routes each job through the channel, invokes `processFc`, and delivers the output via the completion callback.
+
+```cpp
+// Register a completion callback before submitting
+coordinator->setCompletionCallback(
+    [](const serving::system::channels::Message::metadata_t &meta,
+       const std::vector<serving::modules::roles::coordinator::Module::JobOutput> &outputs)
+    {
+      // outputs[i].data is a LocalMemorySlot containing the named output
+      auto *result = static_cast<const float *>(outputs[0].data->getPointer());
+      // ... process result ...
+    });
+
+// Submit a job
+auto job = std::make_shared<serving::modules::roles::Job>(
+    jobFactory.createJob("infer", metadata));
+job->getInputDependency("tokens").storeData(inputBuffer.data(), inputBuffer.size());
+job->getInputDependency("tokens").setSatisfied(true);
+coordinator->submitJob(job);
+```
+
+`metadata.sequenceId` is returned verbatim in the completion callback and can be used to correlate requests with responses.
+
+## 5. Thread safety and memory ownership
+
+- `coordinator->submitJob()` is thread-safe; call it from any thread.
+- `storeData()` copies the buffer into platform-managed memory; the caller's buffer can be reused immediately after `storeData` returns.
+- The `LocalMemorySlot` passed to the completion callback is valid only for the duration of the callback. Copy the data out before returning.
+- `processFc` is invoked by the replica's internal thread. Do not share mutable state between `processFc` and the submission thread without synchronization.
+
+## 6. Performance characteristics
+
+The platform adds one round-trip through the MPI loopback channel per job. With a 1 ms `channelDispatcher` poll interval the overhead is ~1 ms/job independent of payload size.
+
+| Compute time | Platform overhead | Assessment |
+|:---:|:---:|:---:|
+| < 1 ms | > 100% | Framework dominates — optimize poll interval or batch |
+| ~10 ms | ~10% | Borderline — acceptable for many workloads |
+| ≥ 100 ms | < 1% | Not in critical path |
+
+Run the `singlePartition` benchmark to measure overhead on your hardware:
+
+```sh
+mpirun -np 1 --oversubscribe singlePartition <num_requests> <compute_us>
+
+# Example: 200 requests, 10 ms simulated compute
+mpirun -np 1 --oversubscribe singlePartition 200 10000
+```
+
+Output:
+```
+[Baseline]  200 calls | 2000.235 ms total | 10.001 ms/call
+[Platform]  200 jobs  | 2205.507 ms total | 11.028 ms/call
+[Overhead]  +1.026 ms/call (10.3% of compute time)
+            compute=10000 us/call → framework borderline critical path
+```
+
+To reduce overhead below 1 ms, lower the `channelDispatcher` poll interval or reduce payload copy volume via the zero-copy `setOutput(name, slot)` path.