diff --git a/.agents/skills/README.md b/.agents/skills/README.md index 7c8801a..6f5ced9 100644 --- a/.agents/skills/README.md +++ b/.agents/skills/README.md @@ -8,3 +8,13 @@ tooling without maintaining duplicate copies. Add or update skill directories here. Each skill directory must contain its own `SKILL.md`. + +## Skills + +| Skill | Description | +|---|---| +| `create-feature-request` | File a GitHub feature request following the repo template | +| `create-issue` | File a GitHub bug or docs issue following the repo template | +| `rebase-on-main` | Rebase the current feature branch on latest main and force-push, resolving conflicts by preserving the intent of both sides | +| `retrigger-ci` | Retry a failing CI run until it passes when the failure is flaky and unrelated to our changes | +| `platform-style-fix` | Check and fix clang-format style violations in `platform/include` and `platform/examples` | diff --git a/platform/docs/README.md b/platform/docs/README.md index 639ee50..b9adbfa 100644 --- a/platform/docs/README.md +++ b/platform/docs/README.md @@ -22,21 +22,28 @@ platform/ build/ generated build output; not documented here ``` -## Module Areas +## Documentation +- [Architecture](architecture.md): instance roles, module system, channel model, and scope. +- [Model Support Quick-Start](model-support-quickstart.md): wire your inference kernel into the coordinator→replica path, understand the `processFc` interface, and benchmark framework overhead. +- [Simpler Integration](simpler-integration.md): how replica ranks launch Simpler, the `processFc` interface, and the path to in-device tensor channels. +- [Model Support Interface](model-support-interface.md): what Model Support owns, provides, and will consume from the platform. - [System](system/README.md): engine lifecycle and cross-instance start/stop control. - [Channels](system/channels.md): payload, coordination, metadata, input, output, and message primitives. - -Configuration types, the service module, channel controller, and broadcast deployment are tracked in issue #32 and are not part of this initial PR. +- [Modules](modules/README.md): configuration, service, channel controller, and broadcast deployment. ## Runtime Shape The current platform runtime is built around `serving::system::Engine`. The engine owns a set of `serving::modules::Module` instances, initializes them, starts them across instances through RPC, waits for termination, and finalizes them. -This initial PR covers the following building blocks: +This PR covers the following building blocks: - Engine lifecycle: cross-instance start/stop over RPC (`serving::system::Engine`). - Module base interface: initialize/run/terminate/await/finalize lifecycle with optional `taskr::Service` (`serving::modules::Module`). - Channel primitives: `Input`, `Output`, `Message`, and `MessageTypeRegistry` for host-side control traffic. +- Configuration types: `Deployment`, `Partition`, `Task`, `Edge`, `Replica`, `RequestManager` with JSON serialization and verification. +- `broadcastDeployment`: deployer-to-worker deployment config distribution over RPC. +- `channelController`: desired-vs-actual reconciliation loop for host-side SPSC channels. +- `service`: wraps `taskr::Runtime` for cooperative background services. -Deployment graph representation, deployment broadcast, desired-state channel creation, dynamic scaling, topology-aware replacement, fault recovery, and Python bindings are not implemented in this PR. +Dynamic scaling, fault recovery, `channelDispatcher`, `taskScheduler`, executor roles, heartbeat, `RuntimePlan` update protocol, in-device tensor channels, and Python bindings are deferred to follow-up PRs. diff --git a/platform/docs/architecture.md b/platform/docs/architecture.md new file mode 100644 index 0000000..c50c6d5 --- /dev/null +++ b/platform/docs/architecture.md @@ -0,0 +1,69 @@ +# Platform Architecture + +## What the platform layer is + +The platform layer is the **host-side control plane** for a distributed PyPTO Serving deployment. It manages: + +- Instance lifecycle (start, stop, health) +- Deployment configuration distribution +- Host-side channel creation and teardown +- Module initialization and service scheduling + +It is not a model execution layer. Model kernels, tensor movement, KV cache, batching, and token scheduling belong to Model Support and are executed through Simpler on each device. The platform exists so that Model Support does not have to manage MPI ranks, HiCR channels, or distributed lifecycle directly. + +## Instance roles + +Every MPI rank in a deployment plays one of four roles, determined by its instance ID and the deployment configuration: + +| Role | Count | Responsibility | +|---|---|---| +| **Deployer** | 1 (root) | Reads deployment config, broadcasts it to all other ranks, then transitions to a coordinator or replica role | +| **Coordinator** | one per partition | Routes jobs to replicas, accumulates outputs, fires completion callbacks | +| **Replica** | one or more per partition | Executes model computation via Simpler on a dedicated NPU device | +| **Request Manager** | 1 | Entry point for client requests; maps to the partition owning the user-interface edge | + +Device assignment for replicas: `deviceId = instanceId - numPartitions`. Each replica rank owns exactly one NPU. + +## Module system + +All platform behaviour is expressed as `serving::modules::Module` instances owned by the `serving::system::Engine`. The engine drives a fixed lifecycle: + +``` +initialize() → run() → [service loop] → terminate() → await() → finalize() +``` + +Modules can register a periodic background service (via `taskr::Service`) that runs inside the service loop between the `run()` and `terminate()` phases. The engine coordinates the lifecycle of all instances over RPC, so every rank progresses through the same phases in lockstep. + +### Modules in this PR + +| Module | Purpose | +|---|---| +| `broadcastDeployment` | Deployer sends deployment JSON to all workers over RPC at initialize time | +| `channelController` | Desired-vs-actual reconciliation loop; creates and tears down HiCR SPSC channels | +| `service` | Wraps `taskr::Runtime`; owns and drives background `taskr::Service` instances | + +### Deferred modules (next PRs) + +| Module | Purpose | +|---|---| +| `channelDispatcher` | Polls subscribed input channels; dispatches messages to registered handlers | +| `taskScheduler` | Registers named `taskr::Task` instances; drives taskr through the module lifecycle | +| `roles::coordinator` | Job queue and replica dispatch; completion callback when all outputs are gathered | +| `roles::replica` | Receives coordinator input, invokes `processFc`, returns outputs | +| `heartbeat` | Periodic health check between coordinators and replicas | + +## Channel model + +All channels in this PR are **host-side HiCR SPSC channels** carrying variable-size payloads and fixed-size metadata. They are used for control traffic and small tensors. + +In-device tensor channels — where the hot path (prefill/decode token data) moves directly between NPU devices without staging through host memory — are **not implemented here**. They are the next major milestone and unblock the TP/PP tensor data path. + +## What is deliberately out of scope + +- Per-token scheduling (hot path) +- KV cache management +- Batching and sampling policy +- Python bindings +- `RuntimePlan` update protocol and watch/subscribe API +- Dynamic scaling, drain, and fault recovery +- Topology-aware placement diff --git a/platform/docs/model-support-interface.md b/platform/docs/model-support-interface.md new file mode 100644 index 0000000..1dd7cd6 --- /dev/null +++ b/platform/docs/model-support-interface.md @@ -0,0 +1,74 @@ +# Model Support Interface + +This document describes what the Model Support layer needs to know and control when integrating with the platform. + +## What the platform tells you + +At startup, after `broadcastDeployment` completes, every rank knows: + +- **Its role**: coordinator, replica, or request manager — determined by instance ID and the deployment config +- **Its device index** (replicas only): `deviceId = instanceId - numPartitions` +- **The full deployment graph**: partitions, tasks, edges, and which edges carry user-interface traffic + +Model Support should use this information to initialize Simpler on replica ranks before the engine starts. + +## What you provide + +### 1. Compiled Simpler artifacts + +Replica ranks need the five runtime library paths in `RuntimeBinaries`. The platform does not locate or validate them. Pass them via command-line arguments or configuration alongside the deployment JSON. + +### 2. A `processFc` per task function name + +For each task declared in the deployment configuration, Model Support registers a function: + +```cpp +std::function +``` + +This is the only point where Model Support code runs during execution. Everything else — job routing, input aggregation, output forwarding, channel lifecycle — is handled by the platform. + +### 3. Request/response edge names + +The deployment config declares a `RequestManager` with an input edge name and an output edge name. Model Support must ensure: + +- The input edge name matches the edge from which user requests arrive +- The output edge name matches the edge to which results are written +- The task that reads the input edge does not have any other inter-partition inputs +- The task that writes the output edge does not have any other inter-partition outputs + +These constraints are verified by `Deployment::verify()` at startup. + +## What you do not own + +| Concern | Owner | +|---|---| +| MPI rank management | Platform (via HiCR `InstanceManager`) | +| Channel creation and teardown | Platform (`channelController` module) | +| Deployment broadcast | Platform (`broadcastDeployment` module) | +| Job routing from coordinator to replica | Platform (`coordinator::Module`) | +| Input aggregation and output forwarding | Platform (`replica::Module`) | +| Engine lifecycle (init/run/terminate) | Platform (`Engine`) | +| Replica health monitoring | Platform (heartbeat, future) | + +## RuntimePlan — future interface + +The current PR does not yet expose a `RuntimePlan`. A future milestone will provide a versioned, observable snapshot of what the platform has actually instantiated: + +- Which replicas are live, draining, or unavailable +- Which channels exist and what their endpoints are +- Safe `active → draining → removed` state transitions before any channel is deleted + +Model Support will consume this object to make routing and scheduling decisions (KV-locality-aware replica selection, drain-aware request assignment, channel handle lookup) without duplicating platform resource state. The same object will be queryable by operators and higher-level control loops for observability. + +Until `RuntimePlan` is implemented, Model Support should treat the deployment configuration as the ground truth of what is running. + +## Tensor channel hot path — future + +All channels are currently host-side. For the prefill/decode token data path, in-device tensor channels will carry tensor payloads directly between NPU devices without staging through host memory. When implemented: + +- Replica `processFc` will receive device-memory `LocalMemorySlot` handles rather than host buffers +- `toDevice` / `toHost` staging in the process function becomes unnecessary +- The `processFc` signature does not change + +Model Support should design the process function to be forward-compatible: check whether the incoming `LocalMemorySlot` is a host or device buffer and branch accordingly. diff --git a/platform/docs/model-support-quickstart.md b/platform/docs/model-support-quickstart.md new file mode 100644 index 0000000..ee80054 --- /dev/null +++ b/platform/docs/model-support-quickstart.md @@ -0,0 +1,168 @@ +# Model Support Quick-Start + +This guide shows how to wire a model's inference kernel into the platform coordinator→replica path, using the `singlePartition` benchmark as the reference example. + +## 1. The `processFc` interface + +The platform invokes your inference code through a single callback: + +```cpp +using processFc_t = std::function; +``` + +Inside the callback: + +```cpp +void myInferenceFn(serving::modules::roles::TaskContext &ctx) +{ + // Read input tensor + auto slot = ctx.getInput("tokens"); // returns shared_ptr + const auto *data = static_cast(slot->getPointer()); + size_t size = slot->getSize(); + + // Run inference ... + std::vector logits = runModel(data, size); + + // Write output (platform takes a copy; the buffer can go out of scope after this call) + ctx.setOutput("logits", logits.data(), logits.size() * sizeof(float)); +} +``` + +`TaskContext` API summary: + +| Method | Description | +|--------|-------------| +| `getInput(name)` | Returns the named input slot; `nullptr` if the dependency was not satisfied | +| `setOutput(name, ptr, size)` | Copies `size` bytes from `ptr` into the named output channel | +| `setOutput(name, slot)` | Moves an already-allocated `LocalMemorySlot` into the output (zero-copy path) | + +## 2. Defining the data graph + +Each edge in the deployment graph carries one tensor per job. Configure capacity (concurrent jobs in flight) and payload size to match your model's I/O shapes: + +```cpp +// One input edge: 4096-byte token buffer, up to 4 jobs in flight +auto tokensEdge = std::make_shared( + "tokens", /*capacity=*/4, /*payloadBytes=*/4096); +tokensEdge->setPayloadCommunicationManager(commMgr); +tokensEdge->setPayloadMemoryManager(memMgr); +tokensEdge->setPayloadMemorySpace(memSpace); +tokensEdge->setCoordinationCommunicationManager(commMgr); +tokensEdge->setCoordinationMemoryManager(memMgr); +tokensEdge->setCoordinationMemorySpace(memSpace); + +// One output edge: 8-byte result (uint64_t logit checksum in the example) +auto logitsEdge = std::make_shared( + "logits", /*capacity=*/4, sizeof(uint64_t)); +// ... same manager/space setters ... + +serving::configuration::Deployment deployment; +deployment.addChannel(tokensEdge); +deployment.addChannel(logitsEdge); +``` + +Add one task describing the I/O contract: + +```cpp +auto task = std::make_shared(std::string("infer")); +task->addInput("tokens"); +task->addOutput("logits"); + +auto partition = std::make_shared("P0", instanceId); +partition->addTask(task); +partition->addReplica(std::make_shared(instanceId)); +deployment.addPartition(partition); +``` + +## 3. Wiring coordinator and replica (co-located) + +For a single-rank deployment the coordinator and replica live on the same MPI rank. They communicate over loopback MPI channels. The internal channel IDs (`1000`, `2000` below) must not conflict with any other channels in the system. + +```cpp +// Internal edges derived from the graph edge templates +const auto internalTokensEdge = makeInternalEdgeFromTemplate( + "tokens-coord-replica", *deployment.getEdges()[0]); +const auto internalLogitsEdge = makeInternalEdgeFromTemplate( + "logits-replica-coord", *deployment.getEdges()[1]); + +// Coordinator side: sends tokens, receives logits +auto tokensOut = channelController->addDesiredProducer( + instanceId, /*channelId=*/1000, internalTokensEdge, defaultChannelKeyBuilder).lock(); +auto logitsIn = channelController->addDesiredConsumer( + instanceId, /*channelId=*/2000, internalLogitsEdge, defaultChannelKeyBuilder).lock(); + +coordinator->addReplica(instanceId, + /*inputs=*/ {{"tokens", tokensOut}}, + /*outputs=*/{{"logits", logitsIn}}); + +// Replica side: receives tokens, sends logits +auto tokensIn = channelController->addDesiredConsumer( + instanceId, /*channelId=*/1000, internalTokensEdge, defaultChannelKeyBuilder).lock(); +auto logitsOut = channelController->addDesiredProducer( + instanceId, /*channelId=*/2000, internalLogitsEdge, defaultChannelKeyBuilder).lock(); + +replica->setCoordinator(instanceId, + /*outputs=*/{{"logits", logitsOut}}, + /*inputs=*/ {{"tokens", tokensIn}}); +``` + +## 4. Submitting jobs + +After `serving.initialize()` and `serving.run()`, jobs are submitted directly to the coordinator. The platform routes each job through the channel, invokes `processFc`, and delivers the output via the completion callback. + +```cpp +// Register a completion callback before submitting +coordinator->setCompletionCallback( + [](const serving::system::channels::Message::metadata_t &meta, + const std::vector &outputs) + { + // outputs[i].data is a LocalMemorySlot containing the named output + auto *result = static_cast(outputs[0].data->getPointer()); + // ... process result ... + }); + +// Submit a job +auto job = std::make_shared( + jobFactory.createJob("infer", metadata)); +job->getInputDependency("tokens").storeData(inputBuffer.data(), inputBuffer.size()); +job->getInputDependency("tokens").setSatisfied(true); +coordinator->submitJob(job); +``` + +`metadata.sequenceId` is returned verbatim in the completion callback and can be used to correlate requests with responses. + +## 5. Thread safety and memory ownership + +- `coordinator->submitJob()` is thread-safe; call it from any thread. +- `storeData()` copies the buffer into platform-managed memory; the caller's buffer can be reused immediately after `storeData` returns. +- The `LocalMemorySlot` passed to the completion callback is valid only for the duration of the callback. Copy the data out before returning. +- `processFc` is invoked by the replica's internal thread. Do not share mutable state between `processFc` and the submission thread without synchronization. + +## 6. Performance characteristics + +The platform adds one round-trip through the MPI loopback channel per job. With a 1 ms `channelDispatcher` poll interval the overhead is ~1 ms/job independent of payload size. + +| Compute time | Platform overhead | Assessment | +|:---:|:---:|:---:| +| < 1 ms | > 100% | Framework dominates — optimize poll interval or batch | +| ~10 ms | ~10% | Borderline — acceptable for many workloads | +| ≥ 100 ms | < 1% | Not in critical path | + +Run the `singlePartition` benchmark to measure overhead on your hardware: + +```sh +mpirun -np 1 --oversubscribe singlePartition + +# Example: 200 requests, 10 ms simulated compute +mpirun -np 1 --oversubscribe singlePartition 200 10000 +``` + +Output: +``` +[Baseline] 200 calls | 2000.235 ms total | 10.001 ms/call +[Platform] 200 jobs | 2205.507 ms total | 11.028 ms/call +[Overhead] +1.026 ms/call (10.3% of compute time) + compute=10000 us/call → framework borderline critical path +``` + +To reduce overhead below 1 ms, lower the `channelDispatcher` poll interval or reduce payload copy volume via the zero-copy `setOutput(name, slot)` path. diff --git a/platform/docs/modules/README.md b/platform/docs/modules/README.md new file mode 100644 index 0000000..3e5be68 --- /dev/null +++ b/platform/docs/modules/README.md @@ -0,0 +1,52 @@ +# Modules + +Source paths: + +- `include/modules/module.hpp` +- `include/modules/subscription.hpp` +- `include/modules/service/module.hpp` +- `include/modules/channelController/module.hpp` +- `include/modules/broadcastDeployment/module.hpp` + +The modules layer defines reusable platform components that can be installed into `serving::system::Engine`. Modules use a common lifecycle and may optionally expose a periodic TaskR service. + +## Base Module + +`serving::modules::Module` is the abstract base class for all platform modules. + +Main responsibilities: + +- Provide a uniform lifecycle: `initialize()`, `run()`, `terminate()`, `await()`, and `finalize()`. +- Optionally create a `taskr::Service` when constructed with a positive interval. +- Let the engine discover and register periodic services through `hasService()` and `getService()`. + +The base module does not define deployment semantics itself. It only defines how a platform component participates in the runtime lifecycle. + +## Lifecycle + +The engine calls module methods in this order: + +1. `initialize()` before the instance is marked running. +2. `run()` when the deployer starts itself or sends start RPCs to workers. +3. `terminate()` after stop is requested. +4. `await()` after termination starts. +5. `finalize()` after module work has stopped. + +Periodic modules implement their work in the protected `service()` method. The base constructor wraps that method in a `taskr::Service` using the requested interval. + +## Subscription + +`serving::modules::Subscription` binds a message type, an input channel, and a callback: + +- Message type: `channels::Message::messageType_t`. +- Edge: `std::shared_ptr`. +- Handler: `std::function, const channels::Message &)>`. + +This type is a small ownership wrapper for message-driven modules. It does not poll or dispatch by itself. + +## Relation To Platform Design + +The module interface supports the issue #32 goal that platform management should be a side service, not a model-execution abstraction. Modules can bootstrap deployment state, reconcile channels, or run control services without placing platform calls in the token-level model path. + +## Future work +- More modules are coming \ No newline at end of file diff --git a/platform/docs/modules/broadcast-deployment.md b/platform/docs/modules/broadcast-deployment.md new file mode 100644 index 0000000..569bf56 --- /dev/null +++ b/platform/docs/modules/broadcast-deployment.md @@ -0,0 +1,51 @@ +# Broadcast Deployment Module + +Source path: `include/modules/broadcastDeployment/module.hpp` + +The broadcast deployment module distributes a serialized deployment configuration from a deployer instance to other instances through RPC. + +## Main Type + +`serving::modules::broadcastDeployment::Module` derives from `serving::modules::Module`. + +Constructor inputs: + +- `HiCR::InstanceManager` +- `HiCR::ComputeManager` +- `HiCR::frontend::RPCEngine` +- Deployer instance id. +- Current instance id. +- Optional `serving::configuration::Deployment` for the deployer. + +The deployer constructor receives the deployment object. Worker instances use the constructor without a deployment and fetch it during initialization. + +## RPC Flow + +All instances register an RPC target named `__SERVING_REQUEST_DEPLOYMENT_CONFIGURATION_RPC_NAME`. + +When a worker initializes: + +1. It requests the deployment RPC from the deployer instance. +2. It waits for the return value. +3. It parses the returned JSON string. +4. It constructs a local `configuration::Deployment` from that JSON. +5. It frees the RPC return-value memory slot. + +When the deployer initializes: + +- It listens for deployment requests from non-deployer instances. +- On request, it serializes `_deployment` to JSON and submits the serialized string as the RPC return value. + +## Relation To Platform Design + +This module implements the static bootstrap direction from issue #32. It ensures all instances can obtain the same desired deployment state before channels, services, coordinators, and replicas are initialized. + +## Example + +`examples/modules/broadcastDeployment/broadcastDeployment.cpp` parses a deployment only on the root instance, creates the broadcast module on every instance, initializes the engine, and prints the received deployment on each instance. + +## Current Limitations + +- Deployment distribution is init-only and does not publish incremental updates. +- There is no versioning or consistency protocol for dynamic reconfiguration. +- The deployer listens in a loop based on the current instance list and assumes startup-time distribution. diff --git a/platform/docs/modules/channel-controller.md b/platform/docs/modules/channel-controller.md new file mode 100644 index 0000000..9116022 --- /dev/null +++ b/platform/docs/modules/channel-controller.md @@ -0,0 +1,63 @@ +# Channel Controller Module + +Source path: `include/modules/channelController/module.hpp` + +The channel controller manages channel creation through a desired-state reconciliation loop. It turns desired producer and consumer registrations into initialized HiCR-backed `Output` and `Input` channels. + +## Main Type + +`serving::modules::channelController::Module` derives from `serving::modules::Module` and runs as a periodic service by default. + +Constructor inputs: + +- Current instance id. +- Ordered list of `HiCR::CommunicationManager *` objects. +- Global memory-slot exchange tag. +- Reconciliation interval in milliseconds. + +At least one communication manager is required. + +## Desired State + +The controller keeps separate desired and actual maps for producers and consumers. + +`addDesiredProducer(targetId, channelId, edge, keyBuilder)` creates a desired `Output` channel for an edge. + +`addDesiredConsumer(sourceId, channelId, edge, keyBuilder)` creates a desired `Input` channel for an edge. + +`removeDesiredProducer(edgeName)` and `removeDesiredConsumer(edgeName)` remove desired entries. + +`hasProducer(edgeName)` and `hasConsumer(edgeName)` report whether actual channels exist. + +`getProducer(edgeName)` and `getConsumer(edgeName)` return initialized actual channels. + +## Reconciliation Flow + +The private `reconcile()` method: + +1. Diffs desired producers/consumers against actual producers/consumers. +2. Collects memory slots required by missing channels. +3. Groups slots by communication manager. +4. Exchanges global memory slots through the configured managers. +5. Fences each manager on the exchange tag. +6. Initializes missing `Output` and `Input` channels. +7. Moves initialized channels into actual maps. +8. Removes stale actual channels when desired entries disappear. + +The module calls `reconcile()` during `initialize()` and from its periodic `service()` method. + +## Relation To Platform Design + +This module implements the channel-management direction from issue #32: model support can request channels by desired intent, while platform code handles lifecycle and resource exchange. + +The current implementation works at the HiCR channel level. Future integration can add a higher-level distinction between host control channels and device tensor payload channels while preserving the same desired-state reconciliation model. + +## Example + +`examples/modules/channelController/channelController.cpp` parses a deployment, assigns edge managers, registers desired local producer/consumer channels, waits until both are ready, runs a telephone-game message exchange, removes desired channels, and terminates. + +## Current Limitations + +- The TODO in the source notes that every instance currently needs to run this service for MPI-style backends. +- The controller does not yet expose a frontend that hides backend-specific details from callers. +- Placement policy is provided indirectly through edge HiCR manager assignments. diff --git a/platform/docs/modules/configuration.md b/platform/docs/modules/configuration.md new file mode 100644 index 0000000..c89a192 --- /dev/null +++ b/platform/docs/modules/configuration.md @@ -0,0 +1,92 @@ +# Configuration Module + +Source paths: + +- `include/modules/configuration/deployment.hpp` +- `include/modules/configuration/partition.hpp` +- `include/modules/configuration/task.hpp` +- `include/modules/configuration/replica.hpp` +- `include/modules/configuration/edge.hpp` +- `include/modules/configuration/requestManager.hpp` + +The configuration module represents the desired deployment graph for the platform layer. It describes partitions, host-side tasks, replicas, request-manager endpoints, and communication edges. + +## Main Types + +`serving::configuration::Deployment` is the top-level object. It owns: + +- A deployment name. +- A list of `Partition` objects. +- A list of `Edge` objects. +- A `RequestManager` object. +- Heartbeat settings. +- Control-buffer settings. + +`Partition` describes a logical stage of the application. It stores: + +- A partition name. +- The coordinator instance id. +- A list of `Task` objects. +- A list of `Replica` objects. + +`Task` describes a named function within a partition. It stores: + +- Function name. +- Input edge names. +- Output edge names. +- Intra-partition dependencies by function name. + +`Replica` identifies a concrete instance assigned to execute a partition. + +`Edge` describes a communication link. It stores: + +- Edge name. +- Producer partition. +- Consumer partition. +- Buffer capacity and size. +- Payload HiCR manager objects. +- Coordination HiCR manager objects. +- Prompt/result flags for request-manager boundary edges. + +`RequestManager` defines the external entry and result edges for the deployment. + +## Serialization + +All configuration objects serialize to and deserialize from `nlohmann::json`. Runtime-only HiCR manager pointers and memory spaces are not serialized; they are assigned after parsing by runtime setup code. + +The serialized deployment shape includes: + +- `Name` +- `Partitions` +- `Edges` +- `Request Manager` +- `Settings` + +## Validation + +`Deployment::verify()` performs graph sanity checks and fills edge producer/consumer metadata. + +Important checks include: + +- Task dependencies must refer to tasks in the same partition. +- Tasks cannot depend on themselves. +- Edge names must not be duplicated. +- Inputs and outputs must refer to defined edges. +- Each non-request-manager edge must have exactly one producer and one consumer. +- An edge cannot connect a partition to itself, except request-manager prompt/result edges. +- The request-manager input and output edges must be used. +- The partition consuming the external input must not also consume cross-partition inputs. +- The partition producing the external output must not also produce cross-partition outputs. + +After validation, request-manager boundary edges are marked as prompt or result edges. + +## Relation To Platform Design + +The deployment graph is the current implementation of the static deployment API described in issue #32. It gives the platform a desired-state view of partitions, tasks, replicas, and edges before dynamic scaling or fault recovery are added. + +## Current Limitations + +- Placement concepts such as host/device placement are not explicit JSON fields yet. +- Replicas and coordinator instance ids are optional because they may be assigned at runtime, but no placement optimizer is implemented here. +- Heartbeat settings are represented, but recovery policy is not implemented in this module. +- Edge payload/control placement is represented through runtime HiCR manager assignments rather than a high-level placement enum. diff --git a/platform/docs/modules/service.md b/platform/docs/modules/service.md new file mode 100644 index 0000000..8c3ca83 --- /dev/null +++ b/platform/docs/modules/service.md @@ -0,0 +1,41 @@ +# Service Module + +Source path: `include/modules/service/module.hpp` + +The service module adapts TaskR services into the platform module lifecycle. It lets the platform run one or more periodic control services even when there are no normal TaskR tasks. + +## Main Type + +`serving::modules::service::Module` derives from `serving::modules::Module` and owns a shared `taskr::Runtime`. + +It maintains a map from service name to non-owning `taskr::Service *`. + +## Behavior + +`addService(name, service)` registers a service with the module. Names must be unique. + +During `initialize()`: + +- The TaskR runtime is configured with `setFinishOnLastTask(false)` so service-only workloads do not finish immediately. +- Registered services are added to the TaskR runtime. +- The TaskR runtime is initialized. + +During `run()`: + +- The TaskR runtime starts running. + +During `terminate()`: + +- The TaskR runtime is switched back to `setFinishOnLastTask(true)` so it can exit. + +During `await()` and `finalize()`: + +- The module waits for TaskR completion, finalizes the runtime, and clears the service registry. + +## Example + +`examples/modules/service/service.cpp` creates a simple periodic `helloWorld` service, registers it with `service::Module`, installs the module into `serving::system::Engine`, and terminates from the root instance after a short delay. + +## Relation To Platform Design + +This module supports the passive platform-management role from issues #32 and #13. After deployment bootstrap, periodic services such as heartbeat, monitoring, channel reconciliation, or future scaling decisions can run through TaskR without becoming part of model execution. \ No newline at end of file diff --git a/platform/docs/simpler-integration.md b/platform/docs/simpler-integration.md new file mode 100644 index 0000000..16b90ef --- /dev/null +++ b/platform/docs/simpler-integration.md @@ -0,0 +1,72 @@ +# Simpler Integration + +## Which ranks launch Simpler + +Only **replica** ranks initialize a Simpler runtime instance. Coordinator and request manager ranks are pure host-side and never touch device memory or Simpler APIs. + +Device assignment: `deviceId = instanceId - numPartitions` + +With two partitions and one replica each, ranks 0 and 1 are coordinators, ranks 2 and 3 are replicas owning devices 0 and 1 respectively. + +## Initialization + +Each replica rank initializes Simpler in `main()` before the serving engine starts: + +```cpp +hllm::simpler::RuntimeBinaries bins{hostLib, aicpuLib, aicoreKernel, dispatcherLib, simplerLogLib}; +auto rt = std::make_unique(); +rt->init(bins, deviceId); +mnist::loadKernels(*rt, artifactDir); +``` + +`RuntimeBinaries` holds five paths to compiled runtime artifacts: + +| Field | File | Purpose | +|---|---|---| +| `host` | `libhost_runtime.so` | Core device runtime | +| `aicpu` | `libaicpu_kernel.so` | CPU-side kernel support | +| `aicore` | `aicore_kernel.o` | NPU core kernel binary | +| `dispatcher` | `libsimpler_aicpu_dispatcher.so` | On-device dispatcher | +| `simplerLog` | `libsimpler_log.so` | Logging (preloaded RTLD_GLOBAL) | + +Model Support is responsible for providing and locating these artifacts. The platform does not interpret or validate them. + +## `processFc` — the task execution callback + +The replica module accepts a user-provided function with signature: + +```cpp +std::function +``` + +The platform calls this function once per job, after all input dependencies have arrived. Inside the function, Model Support: + +1. Reads inputs from `context.getInput(edgeName)` — returns a `LocalMemorySlot` containing the host buffer +2. Stages inputs to device with `rt->toDevice(ptr, bytes)` +3. Dispatches a Simpler kernel with `rt->run(callableId, args, config)` +4. Stages outputs back with `rt->toHost(hostPtr, devPtr, bytes)` +5. Registers outputs with `context.setOutput(edgeName, ptr, size)` + +The platform guarantees that: +- All declared inputs are present and ready before the call +- Outputs registered via `setOutput` are forwarded to the coordinator after the call returns +- The function is called at most once per job; re-entrancy is not required + +## `SimplerRuntime` API surface + +```cpp +void init(const RuntimeBinaries &bins, int deviceId); +void loadCallable(const void *blob, size_t size, uint32_t id); +void run(uint32_t callableId, ChipStorageTaskArgs &args, ChipCallConfig &config); +void *toDevice(const void *hostPtr, size_t bytes); +void toHost(void *hostPtr, const void *devPtr, size_t bytes); +void *alloc(size_t bytes); +void free(void *devPtr); +void finalize(); +``` + +## Channel model and hot path + +All channels in the current platform are host-side. Tensor payloads passed through `processFc` are staged through host memory (`toDevice` / `toHost`). This is correct for control traffic and small tensors but adds latency on the hot path for large prefill/decode tensors. + +**In-device tensor channels** (direct NPU-to-NPU transfer without host staging) are the next milestone. When implemented, the `processFc` interface will remain the same — Model Support will simply receive device-memory `LocalMemorySlot` handles instead of host buffers, and the staging calls become unnecessary. diff --git a/platform/examples/include/channels/helpers.hpp b/platform/examples/include/channels/helpers.hpp new file mode 100644 index 0000000..735f943 --- /dev/null +++ b/platform/examples/include/channels/helpers.hpp @@ -0,0 +1,148 @@ +#pragma once + +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include +#include +#include + +#include + +struct localChannels_t +{ + std::vector inputs; + std::vector outputs; +}; + +struct singleLocalChannels_t +{ + localInput_t input; + localOutput_t output; +}; + +struct desiredSingleLocalChannels_t +{ + std::shared_ptr input; + std::shared_ptr output; + std::string inputName; + std::string outputName; + localInput_t inputInfo; + localOutput_t outputInfo; +}; + +__INLINE__ localChannels_t buildLocalChannelInfos(const serving::configuration::Deployment &deployment, + const HiCR::Instance::instanceId_t instanceId, + const serving::system::channels::keyBuilderFc_t &keyBuilder) +{ + localChannels_t channels; + buildLocalChannelsFromDeploymentWithIds(deployment, instanceId, keyBuilder, channels.inputs, channels.outputs); + return channels; +} + +__INLINE__ singleLocalChannels_t buildSingleLocalChannelInfos(const serving::configuration::Deployment &deployment, + const HiCR::Instance::instanceId_t instanceId, + const serving::system::channels::keyBuilderFc_t &keyBuilder) +{ + auto channels = buildLocalChannelInfos(deployment, instanceId, keyBuilder); + if (channels.inputs.size() != 1) HICR_THROW_LOGIC("Expected exactly one local input channel, got %lu.", channels.inputs.size()); + if (channels.outputs.size() != 1) HICR_THROW_LOGIC("Expected exactly one local output channel, got %lu.", channels.outputs.size()); + return singleLocalChannels_t{ + .input = channels.inputs.front(), + .output = channels.outputs.front(), + }; +} + +__INLINE__ std::shared_ptr createDesiredInput(const std::shared_ptr &channelController, + const localInput_t &inputInfo, + const serving::system::channels::keyBuilderFc_t &keyBuilder) +{ + auto input = channelController->addDesiredConsumer(inputInfo.sourceInstanceId, inputInfo.channelId, *inputInfo.edge, keyBuilder).lock(); + return input; +} + +__INLINE__ std::shared_ptr createDesiredOutput(const std::shared_ptr &channelController, + const localOutput_t &outputInfo, + const serving::system::channels::keyBuilderFc_t &keyBuilder) +{ + auto output = channelController->addDesiredProducer(outputInfo.targetInstanceId, outputInfo.channelId, *outputInfo.edge, keyBuilder).lock(); + return output; +} + +__INLINE__ desiredSingleLocalChannels_t createDesiredSingleLocalChannels(const serving::configuration::Deployment &deployment, + const HiCR::Instance::instanceId_t instanceId, + const serving::system::channels::keyBuilderFc_t &keyBuilder, + const std::shared_ptr &channelController) +{ + auto infos = buildSingleLocalChannelInfos(deployment, instanceId, keyBuilder); + auto input = createDesiredInput(channelController, infos.input, keyBuilder); + auto output = createDesiredOutput(channelController, infos.output, keyBuilder); + return desiredSingleLocalChannels_t{ + .input = input, + .output = output, + .inputName = infos.input.edge->getName(), + .outputName = infos.output.edge->getName(), + .inputInfo = infos.input, + .outputInfo = infos.output, + }; +} + +__INLINE__ void removeDesiredInputs(const std::shared_ptr &channelController, const std::vector &inputs) +{ + for (const auto &input : inputs) { channelController->removeDesiredConsumer(input.edge->getName()); } +} + +__INLINE__ void removeDesiredOutputs(const std::shared_ptr &channelController, const std::vector &outputs) +{ + for (const auto &output : outputs) { channelController->removeDesiredProducer(output.edge->getName()); } +} + +__INLINE__ void removeDesiredSingleLocalChannels(const std::shared_ptr &channelController, desiredSingleLocalChannels_t &channels) +{ + channelController->removeDesiredConsumer(channels.inputName); + channelController->removeDesiredProducer(channels.outputName); + channels.input = {}; + channels.output = {}; +} + +__INLINE__ void waitUntilReady(const std::shared_ptr &channel, const size_t intervalMs = 500) +{ + while (!channel->isReady()) { std::this_thread::sleep_for(std::chrono::milliseconds(intervalMs)); } +} + +__INLINE__ std::shared_ptr getEdgeByName(serving::configuration::Deployment &deployment, const std::string &edgeName) +{ + for (const auto &edge : deployment.getEdges()) + if (edge->getName() == edgeName) return edge; + HICR_THROW_LOGIC("Deployment has no edge named '%s'.", edgeName.c_str()); +} + +__INLINE__ std::string makeCoordinatorToReplicaEdgeName(const std::string &edgeName, const HiCR::Instance::instanceId_t coordinatorId, const HiCR::Instance::instanceId_t replicaId) +{ + return edgeName + "-coordinator" + std::to_string(coordinatorId) + "-replica" + std::to_string(replicaId); +} + +__INLINE__ std::string makeReplicaToCoordinatorEdgeName(const std::string &edgeName, const HiCR::Instance::instanceId_t replicaId, const HiCR::Instance::instanceId_t coordinatorId) +{ + return edgeName + "-replica" + std::to_string(replicaId) + "-coordinator" + std::to_string(coordinatorId); +} + +__INLINE__ serving::configuration::Edge makeInternalEdgeFromTemplate(const std::string &name, const serving::configuration::Edge &edgeTemplate) +{ + serving::configuration::Edge edge(name, edgeTemplate.getBufferCapacity(), edgeTemplate.getBufferSize()); + edge.setPayloadCommunicationManager(edgeTemplate.getPayloadCommunicationManager()); + edge.setPayloadMemoryManager(edgeTemplate.getPayloadMemoryManager()); + edge.setPayloadMemorySpace(edgeTemplate.getPayloadMemorySpace()); + edge.setCoordinationCommunicationManager(edgeTemplate.getCoordinationCommunicationManager()); + edge.setCoordinationMemoryManager(edgeTemplate.getCoordinationMemoryManager()); + edge.setCoordinationMemorySpace(edgeTemplate.getCoordinationMemorySpace()); + return edge; +} \ No newline at end of file diff --git a/platform/examples/include/deployment/helpers.hpp b/platform/examples/include/deployment/helpers.hpp new file mode 100644 index 0000000..5fe05b8 --- /dev/null +++ b/platform/examples/include/deployment/helpers.hpp @@ -0,0 +1,236 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include +#include + +struct localInput_t +{ + HiCR::Instance::instanceId_t sourceInstanceId = -1; + HiCR::Instance::instanceId_t targetInstanceId = -1; + serving::system::channels::channelId_t channelId = -1; + std::shared_ptr edge = {}; + std::shared_ptr channel = nullptr; // optional: used by legacy examples +}; + +struct localOutput_t +{ + HiCR::Instance::instanceId_t sourceInstanceId = -1; + HiCR::Instance::instanceId_t targetInstanceId = -1; + serving::system::channels::channelId_t channelId = -1; + std::shared_ptr edge = {}; + std::shared_ptr channel; // optional: used by legacy examples +}; + +__INLINE__ serving::system::channels::slotKeys_t defaultChannelKeyBuilder(const HiCR::Instance::instanceId_t sourceInstanceId, + const HiCR::Instance::instanceId_t targetInstanceId, + const serving::system::channels::channelId_t channelId) +{ + using key_t = HiCR::GlobalMemorySlot::globalKey_t; + const key_t src = (key_t(sourceInstanceId) & ((1ull << 20) - 1)) << 44; + const key_t dst = (key_t(targetInstanceId) & ((1ull << 20) - 1)) << 24; + const key_t ch = (key_t(channelId) & ((1ull << 20) - 1)) << 4; + auto makeKey = [&](const key_t slot) -> key_t { return src | dst | ch | (slot & 0xFull); }; + return { + .dataConsumerSizesBufferKey = makeKey(0), + .dataConsumerPayloadBufferKey = makeKey(1), + .dataConsumerCoordinationBufferForSizesKey = makeKey(2), + .dataConsumerCoordinationBufferForPayloadKey = makeKey(3), + .dataProducerCoordinationBufferForSizesKey = makeKey(4), + .dataProducerCoordinationBufferForPayloadKey = makeKey(5), + .metadataConsumerPayloadBufferKey = makeKey(6), + .metadataConsumerCoordinationBufferKey = makeKey(7), + .metadataProducerCoordinationBufferKey = makeKey(8), + }; +} + +__INLINE__ void assignEdgeManagers(serving::configuration::Deployment &deployment, + HiCR::CommunicationManager *communicationManager, + HiCR::MemoryManager *memoryManager, + const std::shared_ptr &memorySpace) +{ + for (const auto &edge : deployment.getEdges()) + { + edge->setPayloadCommunicationManager(communicationManager); + edge->setPayloadMemoryManager(memoryManager); + edge->setPayloadMemorySpace(memorySpace); + edge->setCoordinationCommunicationManager(communicationManager); + edge->setCoordinationMemoryManager(memoryManager); + edge->setCoordinationMemorySpace(memorySpace); + } +} + +__INLINE__ void inferEdgeEndpointsFromTasks(serving::configuration::Deployment &deployment) +{ + std::set edgeNameSet; + + for (const auto &edge : deployment.getEdges()) + { + const auto &edgeName = edge->getName(); + if (edgeNameSet.contains(edgeName)) HICR_THROW_LOGIC("Repeated edge name '%s' in deployment.", edgeName.c_str()); + edgeNameSet.insert(edgeName); + } + + std::map producerPartitionMap; + std::map consumerPartitionMap; + + for (const auto &partition : deployment.getPartitions()) + { + const auto &partitionName = partition->getName(); + + for (const auto &task : partition->getTasks()) + { + for (const auto &output : task->getOutputs()) + { + if (edgeNameSet.contains(output) == false) HICR_THROW_LOGIC("Task '%s' references undefined output edge '%s'.", task->getFunctionName().c_str(), output.c_str()); + if (producerPartitionMap.contains(output)) + HICR_THROW_LOGIC("Edge '%s' has multiple producer partitions ('%s' and '%s').", output.c_str(), producerPartitionMap.at(output).c_str(), partitionName.c_str()); + producerPartitionMap[output] = partitionName; + } + for (const auto &input : task->getInputs()) + { + if (edgeNameSet.contains(input) == false) HICR_THROW_LOGIC("Task '%s' references undefined input edge '%s'.", task->getFunctionName().c_str(), input.c_str()); + if (consumerPartitionMap.contains(input)) + HICR_THROW_LOGIC("Edge '%s' has multiple consumer partitions ('%s' and '%s').", input.c_str(), consumerPartitionMap.at(input).c_str(), partitionName.c_str()); + consumerPartitionMap[input] = partitionName; + } + } + } + + for (const auto &edge : deployment.getEdges()) + { + const auto &edgeName = edge->getName(); + + if (producerPartitionMap.contains(edgeName) == false) HICR_THROW_LOGIC("Edge '%s' is never produced by any task.", edgeName.c_str()); + if (consumerPartitionMap.contains(edgeName) == false) HICR_THROW_LOGIC("Edge '%s' is never consumed by any task.", edgeName.c_str()); + + const auto &producer = producerPartitionMap.at(edgeName); + const auto &consumer = consumerPartitionMap.at(edgeName); + + if (producer == consumer) HICR_THROW_LOGIC("Edge '%s' is both produced and consumed by partition '%s'.", edgeName.c_str(), producer.c_str()); + + edge->setProducer(producer); + edge->setConsumer(consumer); + } +} + +__INLINE__ void buildLocalChannelsFromDeploymentWithIds(const serving::configuration::Deployment &deployment, + const HiCR::Instance::instanceId_t myInstanceId, + const serving::system::channels::keyBuilderFc_t &keyBuilder, + std::vector &inputs, + std::vector &outputs) +{ + std::map partitionToInstance; + + for (const auto &partition : deployment.getPartitions()) partitionToInstance[partition->getName()] = partition->getCoordinatorInstanceId(); + + for (serving::configuration::Edge::edgeIndex_t edgeIdx = 0; edgeIdx < deployment.getEdges().size(); edgeIdx++) + { + const auto &edge = deployment.getEdges()[edgeIdx]; + const auto &producerPartition = edge->getProducer(); + const auto &consumerPartition = edge->getConsumer(); + + if (partitionToInstance.contains(producerPartition) == false) + HICR_THROW_LOGIC("Edge '%s' producer partition '%s' is not present in deployment partition map.", edge->getName().c_str(), producerPartition.c_str()); + if (partitionToInstance.contains(consumerPartition) == false) + HICR_THROW_LOGIC("Edge '%s' consumer partition '%s' is not present in deployment partition map.", edge->getName().c_str(), consumerPartition.c_str()); + + const auto sourceInstanceId = partitionToInstance.at(producerPartition); + const auto targetInstanceId = partitionToInstance.at(consumerPartition); + const auto channelId = static_cast(edgeIdx); + + if (myInstanceId == sourceInstanceId) + { + outputs.push_back(localOutput_t{ + .sourceInstanceId = sourceInstanceId, + .targetInstanceId = targetInstanceId, + .channelId = channelId, + .edge = edge, + .channel = std::make_shared(*edge, channelId, sourceInstanceId, targetInstanceId, keyBuilder), + }); + } + + if (myInstanceId == targetInstanceId) + { + inputs.push_back(localInput_t{ + .sourceInstanceId = sourceInstanceId, + .targetInstanceId = targetInstanceId, + .channelId = channelId, + .edge = edge, + .channel = std::make_shared(*edge, channelId, sourceInstanceId, targetInstanceId, keyBuilder), + }); + } + } +} + +// Compatibility helper for legacy examples that still consume raw channel vectors. +__INLINE__ void buildLocalChannelsFromDeployment(const serving::configuration::Deployment &deployment, + const HiCR::Instance::instanceId_t myInstanceId, + const serving::system::channels::keyBuilderFc_t &keyBuilder, + std::vector> &inputs, + std::vector> &outputs) +{ + std::vector localInputs; + std::vector localOutputs; + + buildLocalChannelsFromDeploymentWithIds(deployment, myInstanceId, keyBuilder, localInputs, localOutputs); + + inputs.reserve(inputs.size() + localInputs.size()); + outputs.reserve(outputs.size() + localOutputs.size()); + + for (const auto &in : localInputs) inputs.push_back(in.channel); + for (const auto &out : localOutputs) outputs.push_back(out.channel); +} + +__INLINE__ void readAndParseConfiguration(char *argv[], + serving::configuration::Deployment &deployment, + std::shared_ptr &instanceManager, + const size_t replicasPerPartition = 0) +{ + const std::string servingConfigFilePath = std::string(argv[1]); + std::ifstream servingConfigFs(servingConfigFilePath); + auto servingConfigJs = nlohmann::json::parse(servingConfigFs); + + deployment.deserialize(servingConfigJs); + + inferEdgeEndpointsFromTasks(deployment); + + // The coordinator is co-located with the first replica, so each partition needs + // 1 instance for the coordinator + (replicasPerPartition - 1) extra replica instances. + const auto extraReplicasPerPartition = replicasPerPartition > 0 ? replicasPerPartition - 1 : 0; + const auto instancesRequired = deployment.getPartitions().size() + (deployment.getPartitions().size() * extraReplicasPerPartition); + + if (instanceManager->getInstances().size() < instancesRequired) + { + fprintf(stderr, "Error: %zu instances provided, but %lu are required\n", instanceManager->getInstances().size(), instancesRequired); + instanceManager->abort(-1); + } + + auto instance = instanceManager->getInstances().begin(); + + for (auto &partition : deployment.getPartitions()) + { + const auto partitionInstanceId = (*instance)->getId(); + ++instance; + partition->setCoordinatorInstanceId(partitionInstanceId); + for (size_t i = 0; i < replicasPerPartition; i++) + { + // First replica is co-located with the coordinator; subsequent ones get their own instance. + const auto replicaInstanceId = (i == 0) ? partitionInstanceId : (*instance)->getId(); + if (i > 0) ++instance; + partition->addReplica(std::make_shared(replicaInstanceId)); + } + } +} \ No newline at end of file diff --git a/platform/examples/include/runtime/helpers.hpp b/platform/examples/include/runtime/helpers.hpp new file mode 100644 index 0000000..3548dce --- /dev/null +++ b/platform/examples/include/runtime/helpers.hpp @@ -0,0 +1,82 @@ +#pragma once +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +struct Runtime +{ + hwloc_topology_t hwlocTopologyObject; + std::shared_ptr instanceManager; + std::shared_ptr communicationManager; + std::shared_ptr memoryManager; + std::shared_ptr workerComputeManager; + std::shared_ptr taskComputeManager; + std::shared_ptr rpcEngine; + std::shared_ptr taskr; + std::shared_ptr bufferMemorySpace; + std::vector> computeResources; + std::unique_ptr serving; +}; + +__INLINE__ nlohmann::json makeDefaultTaskrConfig() +{ + nlohmann::json taskrConfig; + taskrConfig["Task Worker Inactivity Time (Ms)"] = 500; + taskrConfig["Task Suspend Interval Time (Ms)"] = 500; + taskrConfig["Minimum Active Task Workers"] = 1; + taskrConfig["Service Worker Count"] = 1; + taskrConfig["Make Task Workers Run Services"] = true; + return taskrConfig; +} + +__INLINE__ Runtime makeRuntime(int *argc, char ***argv, const size_t computeResourceCount = 2) +{ + Runtime runtime; + + hwloc_topology_init(&runtime.hwlocTopologyObject); + HiCR::backend::hwloc::TopologyManager hwlocTopologyManager(&runtime.hwlocTopologyObject); + + const auto topology = hwlocTopologyManager.queryTopology(); + auto device = *topology.getDevices().begin(); + auto memorySpaces = device->getMemorySpaceList(); + runtime.bufferMemorySpace = *memorySpaces.begin(); + auto computeResourcesIt = device->getComputeResourceList().begin(); + for (size_t i = 0; i < computeResourceCount; i++) + { + runtime.computeResources.push_back(*computeResourcesIt); + ++computeResourcesIt; + } + auto rpcComputeResource = *runtime.computeResources.begin(); + + runtime.instanceManager = std::shared_ptr(HiCR::backend::mpi::InstanceManager::createDefault(argc, argv)); + runtime.communicationManager = std::make_shared(); + runtime.memoryManager = std::make_shared(); + runtime.workerComputeManager = std::make_shared(); + runtime.taskComputeManager = std::make_shared(); + + runtime.rpcEngine = std::make_shared( + *runtime.communicationManager, *runtime.instanceManager, *runtime.memoryManager, *runtime.workerComputeManager, runtime.bufferMemorySpace, rpcComputeResource); + runtime.rpcEngine->initialize(); + + if (runtime.computeResources.size() != 2) { HICR_THROW_RUNTIME("Too many CR"); } + runtime.taskr = std::make_shared(runtime.taskComputeManager.get(), runtime.workerComputeManager.get(), runtime.computeResources, makeDefaultTaskrConfig()); + + runtime.serving = std::make_unique(runtime.instanceManager, runtime.taskComputeManager, runtime.rpcEngine, runtime.instanceManager->getRootInstanceId()); + return runtime; +} \ No newline at end of file diff --git a/platform/examples/include/runtime/moduleDeploymentRunner.hpp b/platform/examples/include/runtime/moduleDeploymentRunner.hpp new file mode 100644 index 0000000..e4cb22f --- /dev/null +++ b/platform/examples/include/runtime/moduleDeploymentRunner.hpp @@ -0,0 +1,347 @@ +#pragma once + +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +class ModuleDeploymentRunner +{ + public: + + using instanceId_t = HiCR::Instance::instanceId_t; + using messageType_t = serving::system::channels::Message::messageType_t; + using channelId_t = serving::system::channels::channelId_t; + using processFc_t = serving::modules::roles::replica::Module::processFc_t; + + ModuleDeploymentRunner(Runtime &runtime, serving::configuration::Deployment &deployment, const messageType_t messageType, processFc_t processFc) + : _runtime(runtime), + _deployment(deployment), + _messageType(messageType), + _processFc(std::move(processFc)), + _instanceId(runtime.instanceManager->getCurrentInstance()->getId()), + _deployerId(runtime.instanceManager->getRootInstanceId()), + _partitionCount(deployment.getPartitions().size()), + _requestManagerId(static_cast(_partitionCount * 2)) + {} + + [[nodiscard]] bool isCoordinator() const { return _instanceId < _partitionCount; } + [[nodiscard]] bool isReplica() const { return _instanceId >= _partitionCount && _instanceId < _requestManagerId; } + [[nodiscard]] bool isRequestManager() const { return _instanceId == _requestManagerId; } + [[nodiscard]] bool isDeployer() const { return _instanceId == _deployerId; } + + [[nodiscard]] auto getRequestManager() const { return _requestManagerModule; } + + void addCoreModules() + { + std::vector managerOrder = {_runtime.communicationManager.get()}; + _channelControllerModule = std::make_shared(_instanceId, managerOrder); + _channelDispatcherModule = std::make_shared(1); + _serviceModule = std::make_shared(_runtime.taskr); + + _serviceModule->addService("ChannelController", _channelControllerModule->getService()); + _serviceModule->addService("ChannelDispatcher", _channelDispatcherModule->getService()); + _runtime.serving->addModule("ChannelController", _channelControllerModule); + _runtime.serving->addModule("ChannelDispatcher", _channelDispatcherModule); + _runtime.serving->addModule("Service", _serviceModule); + } + + void wireGraph() + { + const auto partitionInstanceMap = buildPartitionInstanceMap(); + _promptEdgeName = _deployment.getRequestManager()->getInput(); + _resultEdgeName = _deployment.getRequestManager()->getOutput(); + const auto promptPartitionId = partitionInstanceMap.at(getEdgeByName(_deployment, _promptEdgeName)->getConsumer()); + const auto resultPartitionId = partitionInstanceMap.at(getEdgeByName(_deployment, _resultEdgeName)->getProducer()); + + for (serving::configuration::Edge::edgeIndex_t edgeIdx = 0; edgeIdx < _deployment.getEdges().size(); ++edgeIdx) + { + const auto &edge = _deployment.getEdges().at(edgeIdx); + const auto &edgeName = edge->getName(); + instanceId_t sourceId = 0; + instanceId_t targetId = 0; + if (edgeName == _promptEdgeName) + { + sourceId = _requestManagerId; + targetId = promptPartitionId; + } + else if (edgeName == _resultEdgeName) + { + sourceId = resultPartitionId; + targetId = _requestManagerId; + } + else + { + sourceId = partitionInstanceMap.at(edge->getProducer()); + targetId = partitionInstanceMap.at(edge->getConsumer()); + } + _graphChannels[edgeName] = makeGraphChannel(edgeName, sourceId, targetId, edgeIdx); + } + + wireRequestManager(); + wireCoordinator(); + wireReplica(); + } + + void enableRemoteShutdown(const std::string &edgeName, const channelId_t channelId) + { + _doneEdgeName = edgeName; + const auto doneEdgeTemplate = makeInternalEdgeFromTemplate(edgeName, *getEdgeByName(_deployment, _resultEdgeName)); + if (isRequestManager()) _doneOutput = _channelControllerModule->addDesiredProducer(_deployerId, channelId, doneEdgeTemplate, defaultChannelKeyBuilder).lock(); + if (isDeployer()) _doneInput = _channelControllerModule->addDesiredConsumer(_requestManagerId, channelId, doneEdgeTemplate, defaultChannelKeyBuilder).lock(); + } + + void subscribeAfterInitialize() + { + if (isRequestManager()) + { + waitUntilReady(_graphChannels.at(_promptEdgeName).output); + waitUntilReady(_graphChannels.at(_resultEdgeName).input); + if (_doneOutput != nullptr) waitUntilReady(_doneOutput); + for (auto &subscription : _requestManagerModule->buildSubscriptions()) _channelDispatcherModule->subscribe(subscription); + } + + if (_doneInput != nullptr) + { + waitUntilReady(_doneInput); + _channelDispatcherModule->subscribe(serving::modules::Subscription( + _messageType, _doneInput, [this](const std::shared_ptr, const serving::system::channels::Message &) { _runtime.serving->terminate(); })); + } + + if (isCoordinator()) + { + const auto &task = _deployment.getPartitions().at(static_cast(_instanceId))->getTasks().front(); + for (const auto &inputName : task->getInputs()) waitUntilReady(_graphChannels.at(inputName).input); + for (const auto &outputName : task->getOutputs()) waitUntilReady(_graphChannels.at(outputName).output); + for (auto &subscription : _routerModule->buildSubscriptions()) _channelDispatcherModule->subscribe(subscription); + for (auto &subscription : _coordinatorModule->buildSubscriptions()) _channelDispatcherModule->subscribe(subscription); + } + + if (isReplica()) + { + for (auto &subscription : _replicaModule->buildSubscriptions()) _channelDispatcherModule->subscribe(subscription); + } + } + + void signalShutdown() + { + if (_doneOutput == nullptr) HICR_THROW_LOGIC("Remote shutdown is not enabled on this instance."); + serving::system::channels::Message::metadata_t metadata; + metadata.type = _messageType; + metadata.groupId = 0; + metadata.sequenceId = 1; + const uint8_t payload = 1; + _doneOutput->pushMessageLocking(serving::system::channels::Message(&payload, sizeof(payload), metadata)); + } + + void cleanup() + { + if (isRequestManager()) + { + for (const auto &[type, input] : _requestManagerModule->buildUnsubscriptions()) _channelDispatcherModule->unsubscribe(type, input); + if (_doneOutput != nullptr) _channelControllerModule->removeDesiredProducer(_doneEdgeName); + } + + if (_doneInput != nullptr) + { + _channelDispatcherModule->unsubscribe(_messageType, _doneInput); + _channelControllerModule->removeDesiredConsumer(_doneEdgeName); + } + + if (isCoordinator()) + { + for (const auto &[type, input] : _routerModule->buildUnsubscriptions()) _channelDispatcherModule->unsubscribe(type, input); + for (const auto &[type, input] : _coordinatorModule->buildUnsubscriptions()) _channelDispatcherModule->unsubscribe(type, input); + const auto &task = _deployment.getPartitions().at(static_cast(_instanceId))->getTasks().front(); + const auto replicaId = static_cast(_instanceId + _partitionCount); + for (const auto &inputName : task->getInputs()) _channelControllerModule->removeDesiredProducer(makeCoordinatorToReplicaEdgeName(inputName, _instanceId, replicaId)); + for (const auto &outputName : task->getOutputs()) _channelControllerModule->removeDesiredConsumer(makeReplicaToCoordinatorEdgeName(outputName, replicaId, _instanceId)); + } + + if (isReplica()) + { + for (const auto &[type, input] : _replicaModule->buildUnsubscriptions()) _channelDispatcherModule->unsubscribe(type, input); + const auto coordinatorId = static_cast(_instanceId - _partitionCount); + const auto &task = _deployment.getPartitions().at(static_cast(coordinatorId))->getTasks().front(); + for (const auto &inputName : task->getInputs()) _channelControllerModule->removeDesiredConsumer(makeCoordinatorToReplicaEdgeName(inputName, coordinatorId, _instanceId)); + for (const auto &outputName : task->getOutputs()) _channelControllerModule->removeDesiredProducer(makeReplicaToCoordinatorEdgeName(outputName, _instanceId, coordinatorId)); + } + + for (const auto &[edgeName, channel] : _graphChannels) + { + if (channel.output != nullptr) _channelControllerModule->removeDesiredProducer(edgeName); + if (channel.input != nullptr) _channelControllerModule->removeDesiredConsumer(edgeName); + } + } + + private: + + struct GraphChannel + { + std::shared_ptr edge; + channelId_t channelId = 0; + instanceId_t sourceId = 0; + instanceId_t targetId = 0; + std::shared_ptr input; + std::shared_ptr output; + }; + + std::unordered_map buildPartitionInstanceMap() const + { + std::unordered_map out; + for (const auto &partition : _deployment.getPartitions()) out[partition->getName()] = partition->getCoordinatorInstanceId(); + return out; + } + + GraphChannel makeGraphChannel(const std::string &edgeName, const instanceId_t sourceId, const instanceId_t targetId, const channelId_t channelId) + { + GraphChannel channel; + channel.edge = getEdgeByName(_deployment, edgeName); + channel.channelId = channelId; + channel.sourceId = sourceId; + channel.targetId = targetId; + if (_instanceId == sourceId) channel.output = _channelControllerModule->addDesiredProducer(targetId, channelId, *channel.edge, defaultChannelKeyBuilder).lock(); + if (_instanceId == targetId) channel.input = _channelControllerModule->addDesiredConsumer(sourceId, channelId, *channel.edge, defaultChannelKeyBuilder).lock(); + return channel; + } + + void wireRequestManager() + { + if (!isRequestManager()) return; + _requestManagerModule = std::make_shared(); + _requestManagerModule->addMessageType(_messageType); + _requestManagerModule->setPromptOutput(_graphChannels.at(_promptEdgeName).output); + _requestManagerModule->setResultInput(_graphChannels.at(_resultEdgeName).input); + _runtime.serving->addModule("RequestManager", _requestManagerModule); + } + + void wireCoordinator() + { + if (!isCoordinator()) return; + const auto &partition = *_deployment.getPartitions().at(static_cast(_instanceId)); + const auto &task = partition.getTasks().front(); + const auto replicaId = static_cast(_instanceId + _partitionCount); + + _jobFactory = std::make_shared(_instanceId, _deployment); + _coordinatorModule = std::make_shared(1); + _routerModule = std::make_shared(); + _coordinatorModule->addMessageType(_messageType); + _routerModule->addMessageType(_messageType); + _runtime.serving->addModule("Coordinator", _coordinatorModule); + _runtime.serving->addModule("Router", _routerModule); + _serviceModule->addService("Coordinator", _coordinatorModule->getService()); + + serving::modules::roles::coordinator::Module::inputChannelMap_t replicaInputChannels; + serving::modules::roles::coordinator::Module::outputChannelMap_t replicaOutputChannels; + size_t channelOffset = 0; + for (const auto &inputName : task->getInputs()) + { + const auto edge = getEdgeByName(_deployment, inputName); + const auto internalEdgeName = makeCoordinatorToReplicaEdgeName(inputName, _instanceId, replicaId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + replicaInputChannels[inputName] = + _channelControllerModule->addDesiredProducer(replicaId, static_cast(1000 + _instanceId * 100 + channelOffset++), internalEdge, defaultChannelKeyBuilder).lock(); + _routerModule->addInput(inputName, _graphChannels.at(inputName).input); + } + channelOffset = 0; + for (const auto &outputName : task->getOutputs()) + { + const auto edge = getEdgeByName(_deployment, outputName); + const auto internalEdgeName = makeReplicaToCoordinatorEdgeName(outputName, replicaId, _instanceId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + replicaOutputChannels[outputName] = + _channelControllerModule->addDesiredConsumer(replicaId, static_cast(2000 + _instanceId * 100 + channelOffset++), internalEdge, defaultChannelKeyBuilder).lock(); + _routerModule->addOutput(outputName, _graphChannels.at(outputName).output); + } + _coordinatorModule->addReplica(replicaId, replicaInputChannels, replicaOutputChannels); + + auto defaultProcess = serving::modules::router::makeDefaultProcessFunction(*_jobFactory, *_coordinatorModule, partition); + _routerModule->setProcessFunction(defaultProcess); + _coordinatorModule->setCompletionCallback( + [this](const serving::system::channels::Message::metadata_t &metadata, const std::vector &outputs) { + _routerModule->routeOutputs(metadata, outputs); + }); + } + + void wireReplica() + { + if (!isReplica()) return; + const auto coordinatorId = static_cast(_instanceId - _partitionCount); + const auto &task = _deployment.getPartitions().at(static_cast(coordinatorId))->getTasks().front(); + _replicaModule = std::make_shared(task->getFunctionName(), task->getInputs(), task->getOutputs(), _processFc); + _replicaModule->addMessageType(_messageType); + _runtime.serving->addModule("Replica", _replicaModule); + + serving::modules::roles::replica::Module::inputMap_t inputChannels; + serving::modules::roles::replica::Module::outputMap_t outputChannels; + size_t channelOffset = 0; + for (const auto &inputName : task->getInputs()) + { + const auto edge = getEdgeByName(_deployment, inputName); + const auto internalEdgeName = makeCoordinatorToReplicaEdgeName(inputName, coordinatorId, _instanceId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + inputChannels[inputName] = + _channelControllerModule->addDesiredConsumer(coordinatorId, static_cast(1000 + coordinatorId * 100 + channelOffset++), internalEdge, defaultChannelKeyBuilder) + .lock(); + } + channelOffset = 0; + for (const auto &outputName : task->getOutputs()) + { + const auto edge = getEdgeByName(_deployment, outputName); + const auto internalEdgeName = makeReplicaToCoordinatorEdgeName(outputName, _instanceId, coordinatorId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + outputChannels[outputName] = + _channelControllerModule->addDesiredProducer(coordinatorId, static_cast(2000 + coordinatorId * 100 + channelOffset++), internalEdge, defaultChannelKeyBuilder) + .lock(); + } + _replicaModule->setCoordinator(coordinatorId, outputChannels, inputChannels); + } + + Runtime &_runtime; + serving::configuration::Deployment &_deployment; + messageType_t _messageType; + processFc_t _processFc; + instanceId_t _instanceId; + instanceId_t _deployerId; + size_t _partitionCount; + instanceId_t _requestManagerId; + std::string _promptEdgeName; + std::string _resultEdgeName; + std::string _doneEdgeName; + + std::unordered_map _graphChannels; + + std::shared_ptr _channelControllerModule; + std::shared_ptr _channelDispatcherModule; + std::shared_ptr _serviceModule; + std::shared_ptr _requestManagerModule; + std::shared_ptr _coordinatorModule; + std::shared_ptr _routerModule; + std::shared_ptr _jobFactory; + std::shared_ptr _replicaModule; + std::shared_ptr _doneOutput; + std::shared_ptr _doneInput; +}; diff --git a/platform/examples/meson.build b/platform/examples/meson.build index 860f6d7..deaa6c2 100644 --- a/platform/examples/meson.build +++ b/platform/examples/meson.build @@ -1 +1,10 @@ +modulesDependency = declare_dependency(include_directories: include_directories('include')) + +subdir('modules/broadcastDeployment') +subdir('modules/channelController') +subdir('modules/coordinatorReplica') +subdir('modules/singlePartition') +subdir('modules/dynamicScaling') +subdir('modules/service') + subdir('system/engine') diff --git a/platform/examples/modules/broadcastDeployment/broadcastDeployment.cpp b/platform/examples/modules/broadcastDeployment/broadcastDeployment.cpp new file mode 100644 index 0000000..96454fa --- /dev/null +++ b/platform/examples/modules/broadcastDeployment/broadcastDeployment.cpp @@ -0,0 +1,84 @@ +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +int main(int argc, char *argv[]) +{ + auto runtime = makeRuntime(&argc, &argv); + const auto &instance = runtime.instanceManager->getCurrentInstance(); + auto &serving = *runtime.serving; + + // Check whether the instance is root + const auto isRoot = runtime.instanceManager->getCurrentInstance()->isRootInstance(); + const auto instanceId = runtime.instanceManager->getCurrentInstance()->getId(); + const auto deployerInstanceId = runtime.instanceManager->getRootInstanceId(); + + ///// Configuration parsing + serving::configuration::Deployment deployment; + + // If I am root, checking arguments. + // Do not assume other instances will have the correct arguments set (e.g., file that exists only on root instance) + if (isRoot == true) + { + if (argc != 2) + { + fprintf(stderr, "Error: Must provide the config file path.\n"); + runtime.instanceManager->abort(-1); + } + // Read and parse config file + readAndParseConfiguration(argv, deployment, runtime.instanceManager); + } + + std::shared_ptr broadcastDeploymentModule; + if (instanceId == deployerInstanceId) + { + broadcastDeploymentModule = std::make_shared( + runtime.instanceManager, runtime.taskComputeManager, runtime.rpcEngine, deployerInstanceId, instanceId, deployment); + } + else + { + broadcastDeploymentModule = + std::make_shared(runtime.instanceManager, runtime.taskComputeManager, runtime.rpcEngine, deployerInstanceId, instanceId); + } + + const auto &receivedDeployment = broadcastDeploymentModule->getDeployment(); + // Adding broadcast deployment module to serving + serving.addModule("BroadcastDeployment", broadcastDeploymentModule); + + // Initializing serving + serving.initialize(); + + // Running serving + serving.run(); + + // Finalizing serving + serving.terminate(); + + // Awaiting serving termination + serving.await(); + + // Printing deployment information to verify it was correctly received + std::this_thread::sleep_for(std::chrono::seconds(instanceId)); // Sleep a bit to ensure the output is not mixed + printf("[Instance %lu] Received deployment configuration:\n%s\n", instanceId, receivedDeployment.serialize().dump(2).c_str()); + + // Finalize Instance Manager + runtime.instanceManager->finalize(); +} diff --git a/platform/examples/modules/broadcastDeployment/meson.build b/platform/examples/modules/broadcastDeployment/meson.build new file mode 100644 index 0000000..cdab03f --- /dev/null +++ b/platform/examples/modules/broadcastDeployment/meson.build @@ -0,0 +1,16 @@ +testSuite = ['examples', 'modules'] + +e = executable('broadcastDeployment', ['broadcastDeployment.cpp'], dependencies: [servingBuildDep, modulesDependency]) +if get_option('buildTests') + test( + 'broadcastDeployment', + mpirunExecutable, + args: [ + '-np', '5', + '--oversubscribe', e.full_path(), + meson.current_source_dir() + '/policy.json', + ], + timeout: 60, + suite: testSuite, + ) +endif \ No newline at end of file diff --git a/platform/examples/modules/broadcastDeployment/policy.json b/platform/examples/modules/broadcastDeployment/policy.json new file mode 100644 index 0000000..177bebe --- /dev/null +++ b/platform/examples/modules/broadcastDeployment/policy.json @@ -0,0 +1,42 @@ +{ + "Name": "Basic Deployment", + "Settings": { + "Heartbeat": { + "Enabled": true, + "Visible": false, + "Interval": 500, + "Tolerance": 1000 + }, + "Control Buffer": { + "Capacity": 32, + "Size": 16384 + } + }, + "Request Manager": { + "Input": "input", + "Output": "output" + }, + "Partitions": [ + { + "Name": "P0", + "Tasks": [] + }, + { + "Name": "P1", + "Tasks": [] + }, + { + "Name": "P2", + "Tasks": [] + }, + { + "Name": "P3", + "Tasks": [] + }, + { + "Name": "P4", + "Tasks": [] + } + ], + "Edges": [] +} \ No newline at end of file diff --git a/platform/examples/modules/channelController/channelController.cpp b/platform/examples/modules/channelController/channelController.cpp new file mode 100644 index 0000000..0e96473 --- /dev/null +++ b/platform/examples/modules/channelController/channelController.cpp @@ -0,0 +1,75 @@ +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include +#include "telephoneGame.hpp" + +int main(int argc, char *argv[]) +{ + auto runtime = makeRuntime(&argc, &argv); + const auto &instance = runtime.instanceManager->getCurrentInstance(); + const auto instanceId = instance->getId(); + const auto isRoot = instance->isRootInstance(); + auto &serving = *runtime.serving; + + serving::configuration::Deployment deployment; + + if (argc != 2) + { + fprintf(stderr, "Error: Must provide the config file path.\n"); + runtime.instanceManager->abort(-1); + return -1; + } + + readAndParseConfiguration(argv, deployment, runtime.instanceManager); + assignEdgeManagers(deployment, runtime.communicationManager.get(), runtime.memoryManager.get(), runtime.bufferMemorySpace); + + std::vector managerOrder = {runtime.communicationManager.get()}; + auto channelControllerModule = std::make_shared(instanceId, managerOrder); + + auto channels = createDesiredSingleLocalChannels(deployment, instanceId, defaultChannelKeyBuilder, channelControllerModule); + + auto serviceModule = std::make_shared(runtime.taskr); + serviceModule->addService("ChannelController", channelControllerModule->getService()); + + serving.addModule("ChannelController", channelControllerModule); + serving.addModule("Service", serviceModule); + + serving.initialize(); + + serving.run(); + + waitUntilReady(channels.input); + waitUntilReady(channels.output); + + telephoneGame(*channels.input, *channels.output, instanceId, isRoot); + + removeDesiredSingleLocalChannels(channelControllerModule, channels); + + if (isRoot) serving.terminate(); + + serving.await(); + + runtime.instanceManager->finalize(); + + return 0; +} diff --git a/platform/examples/modules/channelController/meson.build b/platform/examples/modules/channelController/meson.build new file mode 100644 index 0000000..76b99ff --- /dev/null +++ b/platform/examples/modules/channelController/meson.build @@ -0,0 +1,16 @@ +testSuite = ['examples', 'modules'] + +e = executable('channelController', ['channelController.cpp'], dependencies: [servingBuildDep, modulesDependency]) +if get_option('buildTests') + test( + 'channelController', + mpirunExecutable, + args: [ + '-np', '5', + '--oversubscribe', e.full_path(), + meson.current_source_dir() + '/policy.json', + ], + timeout: 60, + suite: testSuite, + ) +endif \ No newline at end of file diff --git a/platform/examples/modules/channelController/policy.json b/platform/examples/modules/channelController/policy.json new file mode 100644 index 0000000..7e63bcc --- /dev/null +++ b/platform/examples/modules/channelController/policy.json @@ -0,0 +1,118 @@ +{ + "Name": "Basic Deployment", + "Settings": { + "Heartbeat": { + "Enabled": true, + "Visible": false, + "Interval": 500, + "Tolerance": 1000 + }, + "Control Buffer": { + "Capacity": 32, + "Size": 16384 + } + }, + "Request Manager": { + "Input": "", + "Output": "" + }, + "Partitions": [ + { + "Name": "P0", + "Tasks": [ + { + "Function Name": "T0", + "Inputs": [ + "Channel 0" + ], + "Outputs": [ + "Channel 1" + ], + "Dependencies": [] + } + ] + }, + { + "Name": "P1", + "Tasks": [ + { + "Function Name": "T1", + "Inputs": [ + "Channel 1" + ], + "Outputs": [ + "Channel 2" + ], + "Dependencies": [] + } + ] + }, + { + "Name": "P2", + "Tasks": [ + { + "Function Name": "T2", + "Inputs": [ + "Channel 2" + ], + "Outputs": [ + "Channel 3" + ], + "Dependencies": [] + } + ] + }, + { + "Name": "P3", + "Tasks": [ + { + "Function Name": "T3", + "Inputs": [ + "Channel 3" + ], + "Outputs": [ + "Channel 4" + ], + "Dependencies": [] + } + ] + }, + { + "Name": "P4", + "Tasks": [ + { + "Function Name": "T4", + "Inputs": [ + "Channel 4" + ], + "Outputs": [ + "Channel 0" + ], + "Dependencies": [] + } + ] + } + ], + "Edges": [ + { + "Name": "Channel 0", + "Buffer Size": 4096 + }, + { + "Name": "Channel 1", + "Buffer Size": 4096 + }, + { + "Name": "Channel 2", + "Buffer Size": 4096 + }, + { + "Name": "Channel 3", + "Buffer Size": 4096 + }, + { + "Name": "Channel 4", + "Buffer Size": 4096 + } + ] +} \ No newline at end of file diff --git a/platform/examples/modules/channelController/telephoneGame.hpp b/platform/examples/modules/channelController/telephoneGame.hpp new file mode 100644 index 0000000..e0d3cee --- /dev/null +++ b/platform/examples/modules/channelController/telephoneGame.hpp @@ -0,0 +1,48 @@ +#pragma once + +#include +#include +#include + +#include + +#include +#include +#include + +#define _REPLICAS_PER_PARTITION 1 + +__INLINE__ void telephoneGame(serving::system::channels::Input &inputChannel, + serving::system::channels::Output &outputChannel, + const HiCR::Instance::instanceId_t instanceId, + const bool isRoot) +{ + if (isRoot) + { + // Root instance starts the game by sending a message to the next instance + const std::string text = "Hello from root instance!"; + printf("[Instance %lu][TelephoneGame] Sending message: %s\n", instanceId, text.c_str()); + auto input = serving::system::channels::Message(reinterpret_cast(text.data()), text.size(), serving::system::channels::Message::metadata_t{}); + outputChannel.pushMessageLocking(input); + + // wait for the return message + while (inputChannel.hasMessage() == false) { std::this_thread::sleep_for(std::chrono::milliseconds(500)); } + auto output = inputChannel.getMessage(); + + printf("[Instance %lu][TelephoneGame] Received message: %s\n", instanceId, std::string(reinterpret_cast(output.getData()), output.getSize()).c_str()); + inputChannel.popMessage(); + } + else + { + while (inputChannel.hasMessage() == false) { std::this_thread::sleep_for(std::chrono::milliseconds(500)); } + auto input = inputChannel.getMessage(); + + printf("[Instance %lu][TelephoneGame] Received message: %s\n", instanceId, std::string(reinterpret_cast(input.getData()), input.getSize()).c_str()); + + auto text = std::string(reinterpret_cast(input.getData()), input.getSize()); + printf("[Instance %lu][TelephoneGame] Sending message: %s\n", instanceId, text.c_str()); + auto output = serving::system::channels::Message(reinterpret_cast(text.data()), text.size(), serving::system::channels::Message::metadata_t{}); + outputChannel.pushMessageLocking(output); + inputChannel.popMessage(); + } +} \ No newline at end of file diff --git a/platform/examples/modules/coordinatorReplica/coordinator.hpp b/platform/examples/modules/coordinatorReplica/coordinator.hpp new file mode 100644 index 0000000..3e80368 --- /dev/null +++ b/platform/examples/modules/coordinatorReplica/coordinator.hpp @@ -0,0 +1,159 @@ +#pragma once + +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +using instanceId_t = HiCR::Instance::instanceId_t; + +struct CoordinatorRuntime +{ + using messageType_t = serving::system::channels::Message::messageType_t; + using input_t = std::shared_ptr; + + std::shared_ptr coordinatorModule; + std::shared_ptr jobFactory; + desiredSingleLocalChannels_t graphChannels; + std::pair graphUnsubscription; + std::shared_ptr> completed = std::make_shared>(false); +}; + +__INLINE__ void submitGraphJob(const std::shared_ptr &jobFactory, + const std::shared_ptr &coordinatorModule, + const std::string &jobName, + const std::string &inputEdgeName, + const serving::system::channels::Message &message) +{ + auto job = std::make_shared(jobFactory->createJob(jobName, message.getMetadata())); + auto &dependency = job->getInputDependency(inputEdgeName); + dependency.storeData(message.getData(), message.getSize()); + dependency.setSatisfied(true); + coordinatorModule->submitJob(job); +} + +__INLINE__ void handleGraphMessage(const bool isRoot, + const std::shared_ptr> &completed, + const std::shared_ptr &jobFactory, + const std::shared_ptr &coordinatorModule, + const std::string &jobName, + const std::string &inputEdgeName, + const serving::system::channels::Message &message) +{ + if (isRoot) + { + completed->store(true); + return; + } + submitGraphJob(jobFactory, coordinatorModule, jobName, inputEdgeName, message); +} + +__INLINE__ void routeOutput(const std::shared_ptr &outputChannel, + const serving::system::channels::Message::metadata_t &metadata, + const std::vector &outputs) +{ + for (const auto &output : outputs) + { + waitUntilReady(outputChannel); + const auto message = serving::system::channels::Message( + output.data == nullptr ? nullptr : static_cast(output.data->getPointer()), output.data == nullptr ? 0 : output.data->getSize(), metadata); + outputChannel->pushMessageLocking(message); + } +} + +// partitionIndex is the index of the partition this instance coordinates. +// replicaIds lists all replica instance IDs for this partition (first may be co-located). +__INLINE__ CoordinatorRuntime coordinator(const HiCR::Instance &instance, + const size_t partitionIndex, + const std::vector &replicaIds, + serving::configuration::Deployment &deployment, + const serving::system::channels::keyBuilderFc_t &keyBuilder, + const std::shared_ptr &channelControllerModule, + const std::shared_ptr &channelDispatcherModule, + const std::shared_ptr &serviceModule, + serving::system::Engine &serving, + const serving::system::channels::Message::messageType_t messageType) +{ + const auto instanceId = instance.getId(); + const auto isRoot = instance.isRootInstance(); + + CoordinatorRuntime coordinatorRuntime; + + coordinatorRuntime.jobFactory = std::make_shared(partitionIndex, deployment); + coordinatorRuntime.coordinatorModule = std::make_shared(100); + coordinatorRuntime.coordinatorModule->addMessageType(messageType); + + serving.addModule("Coordinator", coordinatorRuntime.coordinatorModule); + serviceModule->addService("Coordinator", coordinatorRuntime.coordinatorModule->getService()); + + coordinatorRuntime.graphChannels = createDesiredSingleLocalChannels(deployment, instanceId, keyBuilder, channelControllerModule); + + const auto &task = deployment.getPartitions().at(partitionIndex)->getTasks().front(); + + // Wire internal channels to every replica in this partition. + for (const auto replicaId : replicaIds) + { + serving::modules::roles::coordinator::Module::inputChannelMap_t replicaInputChannels; + serving::modules::roles::coordinator::Module::outputChannelMap_t replicaOutputChannels; + size_t channelOffset = 0; + + for (const auto &inputName : task->getInputs()) + { + const auto edge = getEdgeByName(deployment, inputName); + const auto internalEdgeName = makeCoordinatorToReplicaEdgeName(inputName, instanceId, replicaId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + const auto channelId = static_cast(1000 + instanceId * 100 + channelOffset++); + replicaInputChannels[inputName] = channelControllerModule->addDesiredProducer(replicaId, channelId, internalEdge, keyBuilder).lock(); + } + channelOffset = 0; + for (const auto &outputName : task->getOutputs()) + { + const auto edge = getEdgeByName(deployment, outputName); + const auto internalEdgeName = makeReplicaToCoordinatorEdgeName(outputName, replicaId, instanceId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + const auto channelId = static_cast(2000 + instanceId * 100 + channelOffset++); + replicaOutputChannels[outputName] = channelControllerModule->addDesiredConsumer(replicaId, channelId, internalEdge, keyBuilder).lock(); + } + coordinatorRuntime.coordinatorModule->addReplica(replicaId, replicaInputChannels, replicaOutputChannels); + } + + const auto jobName = task->getFunctionName(); + const auto inputEdgeName = coordinatorRuntime.graphChannels.inputInfo.edge->getName(); + + serving::modules::Subscription graphSubscription( + messageType, + coordinatorRuntime.graphChannels.input, + [isRoot, completed = coordinatorRuntime.completed, jobFactory = coordinatorRuntime.jobFactory, coordinatorModule = coordinatorRuntime.coordinatorModule, jobName, inputEdgeName]( + const std::shared_ptr, const serving::system::channels::Message &message) { + handleGraphMessage(isRoot, completed, jobFactory, coordinatorModule, jobName, inputEdgeName, message); + }); + + channelDispatcherModule->subscribe(graphSubscription); + coordinatorRuntime.graphUnsubscription = {messageType, coordinatorRuntime.graphChannels.input}; + + coordinatorRuntime.coordinatorModule->setCompletionCallback( + [outputChannel = coordinatorRuntime.graphChannels.output](const serving::system::channels::Message::metadata_t &metadata, + const std::vector &outputs) { + routeOutput(outputChannel, metadata, outputs); + }); + + for (auto &subscription : coordinatorRuntime.coordinatorModule->buildSubscriptions()) { channelDispatcherModule->subscribe(subscription); } + return coordinatorRuntime; +} diff --git a/platform/examples/modules/coordinatorReplica/coordinatorReplica.cpp b/platform/examples/modules/coordinatorReplica/coordinatorReplica.cpp new file mode 100644 index 0000000..347dc63 --- /dev/null +++ b/platform/examples/modules/coordinatorReplica/coordinatorReplica.cpp @@ -0,0 +1,152 @@ +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "coordinator.hpp" +#include "replica.hpp" + +constexpr size_t replicasPerPartition = 1; + +constexpr serving::system::channels::Message::messageType_t kMessageType = 100; + +int main(int argc, char *argv[]) +{ + auto runtime = makeRuntime(&argc, &argv); + const auto &instance = runtime.instanceManager->getCurrentInstance(); + const auto instanceId = instance->getId(); + const auto isRoot = instance->isRootInstance(); + auto &serving = *runtime.serving; + + if (argc != 2) + { + fprintf(stderr, "Error: Must provide config file path.\n"); + runtime.instanceManager->abort(-1); + return -1; + } + + serving::configuration::Deployment deployment; + readAndParseConfiguration(argv, deployment, runtime.instanceManager, replicasPerPartition); + assignEdgeManagers(deployment, runtime.communicationManager.get(), runtime.memoryManager.get(), runtime.bufferMemorySpace); + + // Determine roles from the deployment config. + // In the co-located topology, the same instance can be BOTH coordinator and replica. + bool isCoordinator = false; + bool isReplica = false; + size_t coordinatorPartitionIndex = 0; + size_t replicaPartitionIndex = 0; + instanceId_t myCoordinatorId = 0; + std::vector myReplicaIds; + + for (size_t i = 0; i < deployment.getPartitions().size(); i++) + { + const auto &partition = deployment.getPartitions()[i]; + if (partition->getCoordinatorInstanceId() == instanceId) + { + isCoordinator = true; + coordinatorPartitionIndex = i; + for (const auto &r : partition->getReplicas()) myReplicaIds.push_back(r->getInstanceId()); + } + for (const auto &replica : partition->getReplicas()) + { + if (replica->getInstanceId() == instanceId) + { + isReplica = true; + replicaPartitionIndex = i; + myCoordinatorId = partition->getCoordinatorInstanceId(); + } + } + } + + if (isCoordinator) printf("[Instance %lu] Coordinator (partition %zu)\n", instanceId, coordinatorPartitionIndex); + if (isReplica) printf("[Instance %lu] Replica (coordinator: %lu)\n", instanceId, myCoordinatorId); + + serving::system::channels::keyBuilderFc_t keyBuilder = defaultChannelKeyBuilder; + std::vector managerOrder = {runtime.communicationManager.get()}; + auto channelControllerModule = std::make_shared(instanceId, managerOrder); + auto channelDispatcherModule = std::make_shared(20); + auto serviceModule = std::make_shared(runtime.taskr); + + serviceModule->addService("ChannelController", channelControllerModule->getService()); + serviceModule->addService("ChannelDispatcher", channelDispatcherModule->getService()); + serving.addModule("ChannelController", channelControllerModule); + serving.addModule("ChannelDispatcher", channelDispatcherModule); + serving.addModule("Service", serviceModule); + + std::optional coordinatorRt; + std::optional replicaRt; + + if (isCoordinator) + { + coordinatorRt = coordinator( + *instance, coordinatorPartitionIndex, myReplicaIds, deployment, keyBuilder, channelControllerModule, channelDispatcherModule, serviceModule, serving, kMessageType); + } + if (isReplica) + { + replicaRt = + replica(instanceId, myCoordinatorId, replicaPartitionIndex, deployment, keyBuilder, channelControllerModule, channelDispatcherModule, serviceModule, serving, kMessageType); + } + + serving.initialize(); + serving.run(); + + if (isRoot && isCoordinator && coordinatorRt.has_value()) + { + waitUntilReady(coordinatorRt->graphChannels.output); + + const std::string payload = "telephone-start"; + serving::system::channels::Message::metadata_t md; + md.type = kMessageType; + md.groupId = 1; + md.sequenceId = 1; + serving::system::channels::Message msg(reinterpret_cast(payload.data()), payload.size(), md); + coordinatorRt->graphChannels.output->pushMessageLocking(msg); + printf("[Instance %lu] Sent: %s\n", instanceId, payload.c_str()); + + while (!coordinatorRt->completed->load()) { std::this_thread::sleep_for(std::chrono::milliseconds(50)); } + + printf("[Instance %lu] Message returned to root. Terminating.\n", instanceId); + serving.terminate(); + } + + serving.await(); + + // Cleanup + if (coordinatorRt.has_value()) + { + channelDispatcherModule->unsubscribe(coordinatorRt->graphUnsubscription.first, coordinatorRt->graphUnsubscription.second); + for (const auto &[type, input] : coordinatorRt->coordinatorModule->buildUnsubscriptions()) { channelDispatcherModule->unsubscribe(type, input); } + removeDesiredSingleLocalChannels(channelControllerModule, coordinatorRt->graphChannels); + const auto &task = deployment.getPartitions().at(coordinatorPartitionIndex)->getTasks().front(); + for (const auto replicaId : myReplicaIds) + { + for (const auto &inputName : task->getInputs()) channelControllerModule->removeDesiredProducer(makeCoordinatorToReplicaEdgeName(inputName, instanceId, replicaId)); + for (const auto &outputName : task->getOutputs()) channelControllerModule->removeDesiredConsumer(makeReplicaToCoordinatorEdgeName(outputName, replicaId, instanceId)); + } + } + + if (replicaRt.has_value()) + { + for (const auto &[type, input] : replicaRt->replicaModule->buildUnsubscriptions()) { channelDispatcherModule->unsubscribe(type, input); } + const auto &task = deployment.getPartitions().at(replicaPartitionIndex)->getTasks().front(); + for (const auto &inputName : task->getInputs()) channelControllerModule->removeDesiredConsumer(makeCoordinatorToReplicaEdgeName(inputName, myCoordinatorId, instanceId)); + for (const auto &outputName : task->getOutputs()) channelControllerModule->removeDesiredProducer(makeReplicaToCoordinatorEdgeName(outputName, instanceId, myCoordinatorId)); + } + + runtime.instanceManager->finalize(); + + return 0; +} diff --git a/platform/examples/modules/coordinatorReplica/meson.build b/platform/examples/modules/coordinatorReplica/meson.build new file mode 100644 index 0000000..66e388b --- /dev/null +++ b/platform/examples/modules/coordinatorReplica/meson.build @@ -0,0 +1,16 @@ +testSuite = ['examples', 'modules'] + +e = executable('coordinatorReplica', ['coordinatorReplica.cpp'], dependencies: [servingBuildDep, modulesDependency]) +if get_option('buildTests') + test( + 'coordinatorReplica', + mpirunExecutable, + args: [ + '-np', '2', + '--oversubscribe', e.full_path(), + meson.current_source_dir() + '/policy.json', + ], + timeout: 60, + suite: testSuite, + ) +endif diff --git a/platform/examples/modules/coordinatorReplica/policy.json b/platform/examples/modules/coordinatorReplica/policy.json new file mode 100644 index 0000000..cfe98f5 --- /dev/null +++ b/platform/examples/modules/coordinatorReplica/policy.json @@ -0,0 +1,53 @@ +{ + "Name": "Coordinator-Replica Ring", + "Settings": { + "Heartbeat": { + "Enabled": false, + "Visible": false, + "Interval": 500, + "Tolerance": 1000 + }, + "Control Buffer": { + "Capacity": 32, + "Size": 16384 + } + }, + "Request Manager": { + "Input": "", + "Output": "" + }, + "Partitions": [ + { + "Name": "P0", + "Tasks": [ + { + "Function Name": "T0", + "Inputs": ["Channel1"], + "Outputs": ["Channel0"], + "Dependencies": [] + } + ] + }, + { + "Name": "P1", + "Tasks": [ + { + "Function Name": "T1", + "Inputs": ["Channel0"], + "Outputs": ["Channel1"], + "Dependencies": [] + } + ] + } + ], + "Edges": [ + { + "Name": "Channel0", + "Buffer Size": 4096 + }, + { + "Name": "Channel1", + "Buffer Size": 4096 + } + ] +} diff --git a/platform/examples/modules/coordinatorReplica/replica.hpp b/platform/examples/modules/coordinatorReplica/replica.hpp new file mode 100644 index 0000000..f8d9d9b --- /dev/null +++ b/platform/examples/modules/coordinatorReplica/replica.hpp @@ -0,0 +1,86 @@ +#pragma once + +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +using instanceId_t = HiCR::Instance::instanceId_t; + +struct ReplicaRuntime +{ + std::shared_ptr replicaModule; + instanceId_t coordinatorId; +}; + +__INLINE__ ReplicaRuntime replica(const instanceId_t instanceId, + const instanceId_t coordinatorId, + const size_t partitionIndex, + serving::configuration::Deployment &deployment, + const serving::system::channels::keyBuilderFc_t &keyBuilder, + const std::shared_ptr &channelControllerModule, + const std::shared_ptr &channelDispatcherModule, + const std::shared_ptr &serviceModule, + serving::system::Engine &serving, + const serving::system::channels::Message::messageType_t messageType) +{ + ReplicaRuntime replicaRuntime; + replicaRuntime.coordinatorId = coordinatorId; + + const auto &task = deployment.getPartitions().at(partitionIndex)->getTasks().front(); + + auto processFc = [instanceId, inputName = task->getInputs().front(), outputName = task->getOutputs().front()](serving::modules::roles::TaskContext &context) { + const auto input = context.getInput(inputName); + std::string text(reinterpret_cast(input->getPointer()), input->getSize()); + text += " -> R"; + text += std::to_string(instanceId); + printf("[Instance %lu] Replica processed: %s\n", instanceId, text.c_str()); + context.setOutput(outputName, text.data(), text.size()); + }; + + replicaRuntime.replicaModule = std::make_shared(task->getFunctionName(), task->getInputs(), task->getOutputs(), processFc); + replicaRuntime.replicaModule->addMessageType(messageType); + + serving.addModule("Replica", replicaRuntime.replicaModule); + + serving::modules::roles::replica::Module::inputMap_t inputChannels; + serving::modules::roles::replica::Module::outputMap_t outputChannels; + size_t channelOffset = 0; + + for (const auto &inputName : task->getInputs()) + { + const auto edge = getEdgeByName(deployment, inputName); + const auto internalEdgeName = makeCoordinatorToReplicaEdgeName(inputName, coordinatorId, instanceId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + const auto channelId = static_cast(1000 + coordinatorId * 100 + channelOffset++); + inputChannels[inputName] = channelControllerModule->addDesiredConsumer(coordinatorId, channelId, internalEdge, keyBuilder).lock(); + } + channelOffset = 0; + for (const auto &outputName : task->getOutputs()) + { + const auto edge = getEdgeByName(deployment, outputName); + const auto internalEdgeName = makeReplicaToCoordinatorEdgeName(outputName, instanceId, coordinatorId); + const auto internalEdge = makeInternalEdgeFromTemplate(internalEdgeName, *edge); + const auto channelId = static_cast(2000 + coordinatorId * 100 + channelOffset++); + outputChannels[outputName] = channelControllerModule->addDesiredProducer(coordinatorId, channelId, internalEdge, keyBuilder).lock(); + } + replicaRuntime.replicaModule->setCoordinator(coordinatorId, outputChannels, inputChannels); + + for (auto &subscription : replicaRuntime.replicaModule->buildSubscriptions()) { channelDispatcherModule->subscribe(subscription); } + + return replicaRuntime; +} diff --git a/platform/examples/modules/dynamicScaling/dynamicScaling.cpp b/platform/examples/modules/dynamicScaling/dynamicScaling.cpp new file mode 100644 index 0000000..d4d7177 --- /dev/null +++ b/platform/examples/modules/dynamicScaling/dynamicScaling.cpp @@ -0,0 +1,285 @@ +// Dynamic replica scaling example. +// +// Topology: single MPI rank, coordinator co-located with all replicas. +// +// Sequence: +// Phase 1 — two replicas (A, B) handle 30 jobs. +// Phase 2 — drain B (fires callback immediately since B is idle after +// Phase 1); 20 jobs flow only through A. +// Phase 3 — hot-add replica C via addReplicaLive; 30 jobs shared by A + C. +// Shutdown — drain A and C (both idle), clean up. +// +// drainReplica() fires the drainCallback synchronously when the replica has +// no in-flight job. If a job is in flight, the callback fires from +// replicaResponseHandler when that job completes. Either way, removeReplica +// is safe to call only after the callback. + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +using coord_t = serving::modules::roles::coordinator::Module; + +static constexpr serving::system::channels::Message::messageType_t kMsgType = 100; +static constexpr size_t kPayloadBytes = 64; +static constexpr size_t kResultBytes = sizeof(uint64_t); + +static void processFc(serving::modules::roles::TaskContext &ctx) +{ + auto slot = ctx.getInput("tokens"); + const auto first = slot == nullptr ? uint8_t{0} : *static_cast(slot->getPointer()); + uint64_t out = first; + ctx.setOutput("logits", &out, sizeof(out)); +} + +// --------------------------------------------------------------------------- + +static std::shared_ptr makeEdge(const std::string &name, + size_t capacity, + size_t bytes, + HiCR::CommunicationManager *commMgr, + HiCR::MemoryManager *memMgr, + const std::shared_ptr &memSpace) +{ + auto e = std::make_shared(name, capacity, bytes); + e->setPayloadCommunicationManager(commMgr); + e->setPayloadMemoryManager(memMgr); + e->setPayloadMemorySpace(memSpace); + e->setCoordinationCommunicationManager(commMgr); + e->setCoordinationMemoryManager(memMgr); + e->setCoordinationMemorySpace(memSpace); + return e; +} + +struct ReplicaChannels +{ + std::string tokensKey; + std::string logitsKey; + std::shared_ptr tokensOut; + std::shared_ptr logitsIn; + std::shared_ptr tokensIn; + std::shared_ptr logitsOut; +}; + +static ReplicaChannels wireReplica(HiCR::Instance::instanceId_t id, + serving::system::channels::channelId_t base, + const serving::configuration::Edge &tokensEdge, + const serving::configuration::Edge &logitsEdge, + serving::modules::channelController::Module &cc) +{ + const auto tk = "tokens-" + std::to_string(base); + const auto lk = "logits-" + std::to_string(base); + const auto te = makeInternalEdgeFromTemplate(tk, tokensEdge); + const auto le = makeInternalEdgeFromTemplate(lk, logitsEdge); + + ReplicaChannels ch; + ch.tokensKey = tk; + ch.logitsKey = lk; + ch.tokensOut = cc.addDesiredProducer(id, base, te, defaultChannelKeyBuilder).lock(); + ch.logitsIn = cc.addDesiredConsumer(id, base + 1, le, defaultChannelKeyBuilder).lock(); + ch.tokensIn = cc.addDesiredConsumer(id, base, te, defaultChannelKeyBuilder).lock(); + ch.logitsOut = cc.addDesiredProducer(id, base + 1, le, defaultChannelKeyBuilder).lock(); + return ch; +} + +static void waitReady(const ReplicaChannels &ch) +{ + waitUntilReady(ch.tokensOut); + waitUntilReady(ch.logitsIn); + waitUntilReady(ch.tokensIn); + waitUntilReady(ch.logitsOut); +} + +static void submitJobs(size_t n, size_t seqBase, serving::modules::roles::JobFactory &factory, coord_t &coord) +{ + serving::system::channels::Message::metadata_t md; + md.type = kMsgType; + md.groupId = 1; + for (size_t i = 0; i < n; ++i) + { + md.sequenceId = static_cast(seqBase + i + 1); + auto job = std::make_shared(factory.createJob("infer", md)); + std::vector payload(kPayloadBytes, static_cast(md.sequenceId & 0xFF)); + job->getInputDependency("tokens").storeData(payload.data(), payload.size()); + job->getInputDependency("tokens").setSatisfied(true); + coord.submitJob(job); + } +} + +static void waitJobs(const std::atomic &completed, size_t target) +{ + while (completed.load(std::memory_order_acquire) < target) std::this_thread::sleep_for(std::chrono::milliseconds(5)); +} + +// --------------------------------------------------------------------------- + +int main(int argc, char *argv[]) +{ + auto rt = makeRuntime(&argc, &argv); + const auto id = rt.instanceManager->getCurrentInstance()->getId(); + auto &serving = *rt.serving; + auto commMgr = rt.communicationManager.get(); + auto memMgr = rt.memoryManager.get(); + const auto &memSpace = rt.bufferMemorySpace; + + auto tokensEdge = makeEdge("tokens", 8, kPayloadBytes, commMgr, memMgr, memSpace); + auto logitsEdge = makeEdge("logits", 8, kResultBytes, commMgr, memMgr, memSpace); + + serving::configuration::Deployment deployment; + deployment.addChannel(tokensEdge); + deployment.addChannel(logitsEdge); + + auto task = std::make_shared(std::string("infer")); + task->addInput("tokens"); + task->addOutput("logits"); + auto partition = std::make_shared("P0", id); + partition->addTask(task); + partition->addReplica(std::make_shared(id)); + deployment.addPartition(partition); + + std::vector mgrs = {commMgr}; + auto cc = std::make_shared(id, mgrs); + auto cd = std::make_shared(1); + auto svc = std::make_shared(rt.taskr); + svc->addService("ChannelController", cc->getService()); + svc->addService("ChannelDispatcher", cd->getService()); + serving.addModule("ChannelController", cc); + serving.addModule("ChannelDispatcher", cd); + serving.addModule("Service", svc); + + auto factory = serving::modules::roles::JobFactory(0, deployment); + auto coord = std::make_shared(1, "P0"); + coord->addMessageType(kMsgType); + serving.addModule("Coordinator", coord); + svc->addService("Coordinator", coord->getService()); + + // Replica A — channel base 1000 + auto chA = wireReplica(id, 1000, *tokensEdge, *logitsEdge, *cc); + auto repA = std::make_shared("infer", task->getInputs(), task->getOutputs(), processFc); + repA->addMessageType(kMsgType); + serving.addModule("ReplicaA", repA); + repA->setCoordinator(id, {{"logits", chA.logitsOut}}, {{"tokens", chA.tokensIn}}); + coord->addReplica(id, {{"tokens", chA.tokensOut}}, {{"logits", chA.logitsIn}}); + + // Replica B — channel base 1002; will be drained after Phase 1 + auto chB = wireReplica(id, 1002, *tokensEdge, *logitsEdge, *cc); + auto repB = std::make_shared("infer", task->getInputs(), task->getOutputs(), processFc); + repB->addMessageType(kMsgType); + serving.addModule("ReplicaB", repB); + repB->setCoordinator(id, {{"logits", chB.logitsOut}}, {{"tokens", chB.tokensIn}}); + coord->addReplica(id + 1, {{"tokens", chB.tokensOut}}, {{"logits", chB.logitsIn}}); + + std::atomic completed{0}; + std::mutex drainMtx; + std::condition_variable drainCv; + std::atomic drainCount{0}; + + coord->setCompletionCallback( + [&](const serving::system::channels::Message::metadata_t &, const std::vector &) { completed.fetch_add(1, std::memory_order_release); }); + + coord->setDrainCallback([&](HiCR::Instance::instanceId_t rid) { + printf("[drain] replica %lu is idle\n", rid); + drainCount.fetch_add(1, std::memory_order_release); + drainCv.notify_all(); + }); + + for (auto &s : coord->buildSubscriptions()) cd->subscribe(s); + for (auto &s : repA->buildSubscriptions()) cd->subscribe(s); + for (auto &s : repB->buildSubscriptions()) cd->subscribe(s); + + serving.initialize(); + serving.run(); + + waitReady(chA); + waitReady(chB); + + // ── Phase 1: A + B, 30 jobs ────────────────────────────────────────────── + printf("\n=== Phase 1: replicas A + B, 30 jobs ===\n"); + submitJobs(30, 0, factory, *coord); + waitJobs(completed, 30); + printf("[phase1] %zu completed\n", completed.load()); + + // ── Phase 2: drain B (fires immediately, B is idle), 20 jobs on A ──────── + printf("\n=== Phase 2: drain B, 20 jobs on A only ===\n"); + coord->drainReplica(id + 1); // callback fires synchronously (B idle) + { + std::unique_lock lk(drainMtx); + drainCv.wait(lk, [&] { return drainCount.load() >= 1; }); + } + coord->removeReplica(id + 1); + submitJobs(20, 30, factory, *coord); + waitJobs(completed, 50); + printf("[phase2] %zu completed, B removed\n", completed.load()); + + // ── Phase 3: hot-add C (base 1004), 30 jobs on A + C ──────────────────── + printf("\n=== Phase 3: hot-add replica C, 30 jobs on A + C ===\n"); + + auto chC = wireReplica(id, 1004, *tokensEdge, *logitsEdge, *cc); + waitReady(chC); // wait before subscribing so dispatcher never polls uninitialized channels + auto repC = std::make_shared("infer", task->getInputs(), task->getOutputs(), processFc); + repC->addMessageType(kMsgType); + serving.addModule("ReplicaC", repC); + repC->setCoordinator(id, {{"logits", chC.logitsOut}}, {{"tokens", chC.tokensIn}}); + for (auto &s : repC->buildSubscriptions()) cd->subscribe(s); + + // addReplicaLive returns subscriptions for coordinator→replica response path + auto newSubs = coord->addReplicaLive(id + 2, {{"tokens", chC.tokensOut}}, {{"logits", chC.logitsIn}}); + for (auto &s : newSubs) cd->subscribe(s); + + submitJobs(30, 50, factory, *coord); + waitJobs(completed, 80); + printf("[phase3] %zu completed\n", completed.load()); + + // ── Shutdown: drain A + C (both idle after Phase 3) ─────────────────────── + printf("\n=== Shutdown: drain A + C ===\n"); + coord->drainReplica(id); // fires synchronously (A idle) + coord->drainReplica(id + 2); // fires synchronously (C idle) + { + std::unique_lock lk(drainMtx); + drainCv.wait(lk, [&] { return drainCount.load() >= 3; }); + } + coord->removeReplica(id); + coord->removeReplica(id + 2); + + serving.terminate(); + serving.await(); + + for (auto &[t, ch] : coord->buildUnsubscriptions()) cd->unsubscribe(t, ch); + for (auto &[t, ch] : repA->buildUnsubscriptions()) cd->unsubscribe(t, ch); + for (auto &[t, ch] : repB->buildUnsubscriptions()) cd->unsubscribe(t, ch); + for (auto &[t, ch] : repC->buildUnsubscriptions()) cd->unsubscribe(t, ch); + + for (const auto &[tk, lk] : + std::initializer_list>{{chA.tokensKey, chA.logitsKey}, {chB.tokensKey, chB.logitsKey}, {chC.tokensKey, chC.logitsKey}}) + { + cc->removeDesiredProducer(tk); + cc->removeDesiredConsumer(lk); + cc->removeDesiredConsumer(tk); + cc->removeDesiredProducer(lk); + } + + rt.instanceManager->finalize(); + printf("\nDone. %zu jobs processed.\n", completed.load()); + return 0; +} diff --git a/platform/examples/modules/dynamicScaling/meson.build b/platform/examples/modules/dynamicScaling/meson.build new file mode 100644 index 0000000..ad83fff --- /dev/null +++ b/platform/examples/modules/dynamicScaling/meson.build @@ -0,0 +1,21 @@ +testSuite = ['examples', 'dynamicScaling'] + +dynamicScaling = executable( + 'dynamicScaling', + ['dynamicScaling.cpp'], + dependencies: [servingBuildDep, modulesDependency], +) + +if get_option('buildTests') + test( + 'dynamicScaling', + mpirunExecutable, + args: [ + '-np', '1', + '--oversubscribe', + dynamicScaling.full_path(), + ], + timeout: 60, + suite: testSuite, + ) +endif diff --git a/platform/examples/modules/service/meson.build b/platform/examples/modules/service/meson.build new file mode 100644 index 0000000..6f9f85c --- /dev/null +++ b/platform/examples/modules/service/meson.build @@ -0,0 +1,15 @@ +testSuite = ['examples', 'modules'] + +e = executable('service', ['service.cpp'], dependencies: [servingBuildDep, modulesDependency]) +if get_option('buildTests') + test( + 'service', + mpirunExecutable, + args: [ + '-np', '5', + '--oversubscribe', e.full_path() + ], + timeout: 60, + suite: testSuite, + ) +endif \ No newline at end of file diff --git a/platform/examples/modules/service/service.cpp b/platform/examples/modules/service/service.cpp new file mode 100644 index 0000000..b6df7aa --- /dev/null +++ b/platform/examples/modules/service/service.cpp @@ -0,0 +1,119 @@ +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +int main(int argc, char *argv[]) +{ + // Creating HWloc topology object + hwloc_topology_t hwlocTopologyObject; + + // Reserving memory for hwloc + hwloc_topology_init(&hwlocTopologyObject); + + // Initializing host (CPU) topology manager + HiCR::backend::hwloc::TopologyManager hwlocTopologyManager(&hwlocTopologyObject); + + // Gathering topology from the topology manager + const auto topology = hwlocTopologyManager.queryTopology(); + + auto d = *topology.getDevices().begin(); + auto memSpaces = d->getMemorySpaceList(); + if (memSpaces.empty()) HICR_THROW_RUNTIME("No memory spaces found on the queried device"); + auto bufferMemorySpace = *memSpaces.begin(); + + const auto &availableComputeResources = d->getComputeResourceList(); + if (availableComputeResources.size() < 2) HICR_THROW_RUNTIME("Fewer than 2 compute resources available"); + auto computeResourcesIt = availableComputeResources.begin(); + + // Use only 2 cores + std::vector> computeResources; + for (int i = 0; i < 2; i++) + { + computeResources.push_back(*computeResourcesIt); + computeResourcesIt++; + } + auto computeResource = *computeResources.begin(); + + // Getting managers + auto instanceManager = std::shared_ptr(HiCR::backend::mpi::InstanceManager::createDefault(&argc, &argv)); + auto communicationManager = std::make_shared(); + auto memoryManager = std::make_shared(); + auto workerComputeManager = std::make_shared(); + auto taskComputeManager = std::make_shared(); + + // Instantiate RPC Engine + auto rpcEngine = std::make_shared(*communicationManager, *instanceManager, *memoryManager, *workerComputeManager, bufferMemorySpace, computeResource); + + // Initialize RPC Engine + rpcEngine->initialize(); + + // Creating taskr object + nlohmann::json taskrConfig; + taskrConfig["Task Worker Inactivity Time (Ms)"] = 100; // Suspend workers if a certain time of inactivity elapses + taskrConfig["Task Suspend Interval Time (Ms)"] = 100; // Workers suspend for this time before checking back + taskrConfig["Minimum Active Task Workers"] = 1; // Have at least one worker active at all times + taskrConfig["Service Worker Count"] = 1; // Have one dedicated service workers at all times to listen for incoming messages + taskrConfig["Make Task Workers Run Services"] = true; // Workers will check for meta messages in between executions + auto taskr = std::make_shared(taskComputeManager.get(), workerComputeManager.get(), computeResources, taskrConfig); + + // Creating serving Engine object + serving::system::Engine serving(instanceManager, taskComputeManager, rpcEngine, instanceManager->getRootInstanceId()); + + // Check whether the instance is root + const auto isRoot = instanceManager->getCurrentInstance()->isRootInstance(); + const auto instanceId = instanceManager->getCurrentInstance()->getId(); + + auto serviceModule = std::make_unique(taskr); + + // Adding a simple service that prints "Hello world" one time. Here we call terminate from within the task + // to keep the application simple + int reps = 5; + int count = 0; + auto helloWorldFc = [&]() { + if (count > reps) { return; } + + printf("[Instance %lu] Hello World %d!\n", instanceId, count); + count++; + }; + auto helloWorldService = taskr::Service(helloWorldFc, 10); + serviceModule->addService("helloWorld", &helloWorldService); + + // Adding service module to serving + serving.addModule("service", std::move(serviceModule)); + + // Initializing serving + serving.initialize(); + + // Running serving + serving.run(); + + if (isRoot) + { + printf("[Instance %lu] issuing termination\n", instanceId); + + std::this_thread::sleep_for(std::chrono::seconds(1)); + // Finalizing serving + serving.terminate(); + } + + // Awaiting serving termination + serving.await(); + + // Finalize Instance Manager + instanceManager->finalize(); +} diff --git a/platform/examples/modules/singlePartition/meson.build b/platform/examples/modules/singlePartition/meson.build new file mode 100644 index 0000000..2739441 --- /dev/null +++ b/platform/examples/modules/singlePartition/meson.build @@ -0,0 +1,21 @@ +testSuite = ['examples', 'singlePartition'] + +singlePartition = executable( + 'singlePartition', + ['singlePartition.cpp'], + dependencies: [servingBuildDep, modulesDependency], +) + +if get_option('buildTests') + test( + 'singlePartition', + mpirunExecutable, + args: [ + '-np', '1', + '--oversubscribe', + singlePartition.full_path(), + ], + timeout: 60, + suite: testSuite, + ) +endif diff --git a/platform/examples/modules/singlePartition/singlePartition.cpp b/platform/examples/modules/singlePartition/singlePartition.cpp new file mode 100644 index 0000000..85063db --- /dev/null +++ b/platform/examples/modules/singlePartition/singlePartition.cpp @@ -0,0 +1,256 @@ +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +// --------------------------------------------------------------------------- +// Runtime parameters (set from argv) +// --------------------------------------------------------------------------- +static size_t gNumRequests = 200; // jobs to submit +static uint32_t gComputeUs = 0; // synthetic compute µs in processFc + +static constexpr size_t kPayloadBytes = 4096; +static constexpr serving::system::channels::Message::messageType_t kMessageType = 100; + +// --------------------------------------------------------------------------- +// processFc: the "model inference" hook. +// Reads input bytes, accumulates a checksum (simulates per-token work), +// writes the result back. Same logic is used for the baseline measurement. +// --------------------------------------------------------------------------- +static uint64_t computeChecksum(const uint8_t *data, size_t size) +{ + uint64_t acc = 0; + for (size_t i = 0; i < size; ++i) acc += data[i]; + return acc; +} + +static void processFc(serving::modules::roles::TaskContext &ctx) +{ + auto slot = ctx.getInput("tokens"); + const auto data = static_cast(slot == nullptr ? nullptr : slot->getPointer()); + const auto size = slot == nullptr ? 0 : slot->getSize(); + uint64_t result = computeChecksum(data, size); + + // Simulate inference compute (configurable via gComputeUs) + if (gComputeUs > 0) + { + auto deadline = std::chrono::steady_clock::now() + std::chrono::microseconds(gComputeUs); + while (std::chrono::steady_clock::now() < deadline) {} + } + + ctx.setOutput("logits", &result, sizeof(result)); +} + +// --------------------------------------------------------------------------- +// Build a minimal single-partition deployment in memory (no JSON). +// One task "infer", input edge "tokens", output edge "logits". +// Coordinator and replica are both on instance 0 (co-located). +// --------------------------------------------------------------------------- +static serving::configuration::Deployment buildDeployment(const HiCR::Instance::instanceId_t instanceId, + HiCR::CommunicationManager *commMgr, + HiCR::MemoryManager *memMgr, + const std::shared_ptr &memSpace) +{ + serving::configuration::Deployment deployment; + + auto tokensEdge = std::make_shared("tokens", /*capacity=*/4, kPayloadBytes); + tokensEdge->setPayloadCommunicationManager(commMgr); + tokensEdge->setPayloadMemoryManager(memMgr); + tokensEdge->setPayloadMemorySpace(memSpace); + tokensEdge->setCoordinationCommunicationManager(commMgr); + tokensEdge->setCoordinationMemoryManager(memMgr); + tokensEdge->setCoordinationMemorySpace(memSpace); + + auto logitsEdge = std::make_shared("logits", /*capacity=*/4, sizeof(uint64_t)); + logitsEdge->setPayloadCommunicationManager(commMgr); + logitsEdge->setPayloadMemoryManager(memMgr); + logitsEdge->setPayloadMemorySpace(memSpace); + logitsEdge->setCoordinationCommunicationManager(commMgr); + logitsEdge->setCoordinationMemoryManager(memMgr); + logitsEdge->setCoordinationMemorySpace(memSpace); + + deployment.addChannel(tokensEdge); + deployment.addChannel(logitsEdge); + + auto task = std::make_shared(std::string("infer")); + task->addInput("tokens"); + task->addOutput("logits"); + auto partition = std::make_shared("P0", instanceId); + partition->addTask(task); + partition->addReplica(std::make_shared(instanceId)); + deployment.addPartition(partition); + + return deployment; +} + +int main(int argc, char *argv[]) +{ + if (argc >= 2) gNumRequests = static_cast(std::stoull(argv[1])); + if (argc >= 3) gComputeUs = static_cast(std::stoull(argv[2])); + + auto runtime = makeRuntime(&argc, &argv); + const auto &instance = runtime.instanceManager->getCurrentInstance(); + const auto instanceId = instance->getId(); + auto &serving = *runtime.serving; + + // ----- Build deployment ----- + auto deployment = buildDeployment(instanceId, runtime.communicationManager.get(), runtime.memoryManager.get(), runtime.bufferMemorySpace); + + // ----- Core modules ----- + std::vector managerOrder = {runtime.communicationManager.get()}; + auto channelController = std::make_shared(instanceId, managerOrder); + auto channelDispatcher = std::make_shared(1); + auto serviceModule = std::make_shared(runtime.taskr); + + serviceModule->addService("ChannelController", channelController->getService()); + serviceModule->addService("ChannelDispatcher", channelDispatcher->getService()); + serving.addModule("ChannelController", channelController); + serving.addModule("ChannelDispatcher", channelDispatcher); + serving.addModule("Service", serviceModule); + + // ----- Coordinator ----- + const auto &partition = deployment.getPartitions().front(); + const auto &task = partition->getTasks().front(); + + auto jobFactory = serving::modules::roles::JobFactory(0, deployment); + auto coordinator = std::make_shared(1); + coordinator->addMessageType(kMessageType); + serving.addModule("Coordinator", coordinator); + serviceModule->addService("Coordinator", coordinator->getService()); + + // Coordinator→Replica internal channels (co-located: same rank, loopback) + const auto internalTokensEdge = makeInternalEdgeFromTemplate("tokens-coord-replica", *deployment.getEdges()[0]); + const auto internalLogitsEdge = makeInternalEdgeFromTemplate("logits-replica-coord", *deployment.getEdges()[1]); + + auto tokensOut = channelController->addDesiredProducer(instanceId, 1000, internalTokensEdge, defaultChannelKeyBuilder).lock(); + auto logitsIn = channelController->addDesiredConsumer(instanceId, 2000, internalLogitsEdge, defaultChannelKeyBuilder).lock(); + + serving::modules::roles::coordinator::Module::inputChannelMap_t replicaInputs = {{"tokens", tokensOut}}; + serving::modules::roles::coordinator::Module::outputChannelMap_t replicaOutputs = {{"logits", logitsIn}}; + coordinator->addReplica(instanceId, replicaInputs, replicaOutputs); + + // ----- Replica ----- + auto replica = std::make_shared("infer", task->getInputs(), task->getOutputs(), processFc); + replica->addMessageType(kMessageType); + serving.addModule("Replica", replica); + + auto tokensIn = channelController->addDesiredConsumer(instanceId, 1000, internalTokensEdge, defaultChannelKeyBuilder).lock(); + auto logitsOut = channelController->addDesiredProducer(instanceId, 2000, internalLogitsEdge, defaultChannelKeyBuilder).lock(); + replica->setCoordinator(instanceId, {{"logits", logitsOut}}, {{"tokens", tokensIn}}); + + // ----- Completion tracking ----- + std::atomic completed{0}; + std::vector results(gNumRequests, 0); + + coordinator->setCompletionCallback( + [&](const serving::system::channels::Message::metadata_t &meta, const std::vector &outputs) { + const size_t idx = meta.sequenceId - 1; + if (!outputs.empty() && outputs[0].data != nullptr) results[idx] = *static_cast(outputs[0].data->getPointer()); + completed.fetch_add(1, std::memory_order_release); + }); + + // ----- Subscribe & start ----- + for (auto &sub : coordinator->buildSubscriptions()) channelDispatcher->subscribe(sub); + for (auto &sub : replica->buildSubscriptions()) channelDispatcher->subscribe(sub); + + serving.initialize(); + serving.run(); + + // Wait for channels to be ready + waitUntilReady(tokensOut); + waitUntilReady(logitsIn); + waitUntilReady(tokensIn); + waitUntilReady(logitsOut); + + // ----------------------------------------------------------------------- + // Baseline: N direct processFc-equivalent calls, no framework overhead. + // ----------------------------------------------------------------------- + double baselineMs = 0.0; + { + std::vector buf(kPayloadBytes, 0xAB); + uint64_t sink = 0; + auto t0 = std::chrono::steady_clock::now(); + for (size_t i = 0; i < gNumRequests; ++i) + { + sink += computeChecksum(buf.data(), buf.size()); + if (gComputeUs > 0) + { + auto dl = std::chrono::steady_clock::now() + std::chrono::microseconds(gComputeUs); + while (std::chrono::steady_clock::now() < dl) {} + } + } + auto t1 = std::chrono::steady_clock::now(); + baselineMs = std::chrono::duration(t1 - t0).count(); + printf("[Baseline] %zu calls | %.3f ms total | %.4f ms/call (sink=%lu)\n", gNumRequests, baselineMs, baselineMs / gNumRequests, sink); + } + + // ----------------------------------------------------------------------- + // Platform path: same N jobs through coordinator→replica→processFc. + // ----------------------------------------------------------------------- + { + std::vector payload(kPayloadBytes, 0xAB); + auto t0 = std::chrono::steady_clock::now(); + + for (size_t i = 0; i < gNumRequests; ++i) + { + serving::system::channels::Message::metadata_t md; + md.type = kMessageType; + md.groupId = 1; + md.sequenceId = static_cast(i + 1); + + auto job = std::make_shared(jobFactory.createJob("infer", md)); + job->getInputDependency("tokens").storeData(payload.data(), payload.size()); + job->getInputDependency("tokens").setSatisfied(true); + coordinator->submitJob(job); + } + + while (completed.load(std::memory_order_acquire) < gNumRequests) std::this_thread::sleep_for(std::chrono::microseconds(10)); + + auto t1 = std::chrono::steady_clock::now(); + double platformMs = std::chrono::duration(t1 - t0).count(); + double overheadMs = platformMs - baselineMs; + double overheadPc = overheadMs / baselineMs * 100.0; + + printf("[Platform] %zu jobs | %.3f ms total | %.4f ms/call\n", gNumRequests, platformMs, platformMs / gNumRequests); + printf("[Overhead] +%.3f ms/call (%.1f%% of compute time)\n", overheadMs / gNumRequests, overheadPc); + printf(" compute=%.0f us/call → framework %s critical path\n", + static_cast(gComputeUs), + overheadPc < 5.0 ? "NOT in" + : overheadPc < 20.0 ? "borderline" + : "IS in"); + } + + serving.terminate(); + serving.await(); + + // Cleanup + for (auto &[type, ch] : coordinator->buildUnsubscriptions()) channelDispatcher->unsubscribe(type, ch); + for (auto &[type, ch] : replica->buildUnsubscriptions()) channelDispatcher->unsubscribe(type, ch); + channelController->removeDesiredProducer("tokens-coord-replica"); + channelController->removeDesiredConsumer("logits-replica-coord"); + channelController->removeDesiredConsumer("tokens-coord-replica"); + channelController->removeDesiredProducer("logits-replica-coord"); + + runtime.instanceManager->finalize(); + return 0; +} diff --git a/platform/include/modules/broadcastDeployment/module.hpp b/platform/include/modules/broadcastDeployment/module.hpp new file mode 100644 index 0000000..6088fae --- /dev/null +++ b/platform/include/modules/broadcastDeployment/module.hpp @@ -0,0 +1,132 @@ +#pragma once + +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +namespace serving::modules::broadcastDeployment +{ + +#define __SERVING_REQUEST_DEPLOYMENT_CONFIGURATION_RPC_NAME "[serving/modules/broadcastDeployment] Request Deployment Configuration RPC" + +class Module final : public modules::Module +{ + public: + + Module(std::shared_ptr instanceManager, + std::shared_ptr computeManager, + std::shared_ptr rpcEngine, + const HiCR::Instance::instanceId_t deployerInstanceId, + const HiCR::Instance::instanceId_t instanceId, + const configuration::Deployment &deployment) + : modules::Module(), + _instanceManager(instanceManager), + _computeManager(computeManager), + _rpcEngine(rpcEngine), + _deployerInstanceId(deployerInstanceId), + _instanceId(instanceId), + _deployment(deployment) + { + _rpcEngine->addRPCTarget(__SERVING_REQUEST_DEPLOYMENT_CONFIGURATION_RPC_NAME, _computeManager->createExecutionUnit([this](void *) { sendDeploymentConfiguration(); })); + } + + Module(std::shared_ptr instanceManager, + std::shared_ptr computeManager, + std::shared_ptr rpcEngine, + const HiCR::Instance::instanceId_t deployerInstanceId, + const HiCR::Instance::instanceId_t instanceId) + : modules::Module(), + _instanceManager(instanceManager), + _computeManager(computeManager), + _rpcEngine(rpcEngine), + _deployerInstanceId(deployerInstanceId), + _instanceId(instanceId) + { + _rpcEngine->addRPCTarget(__SERVING_REQUEST_DEPLOYMENT_CONFIGURATION_RPC_NAME, _computeManager->createExecutionUnit([this](void *) { sendDeploymentConfiguration(); })); + } + + ~Module() override = default; + + __INLINE__ const configuration::Deployment &getDeployment() const { return _deployment; } + + void initialize() override + { + auto _instances = _instanceManager->getInstances(); + // If I am not the deployer instance, simply request the deployment information from the deployer + if (_instanceId == _deployerInstanceId) + { + for (const auto &instance : _instances) + { + if (instance->getId() == _deployerInstanceId) { continue; } + _rpcEngine->listen(); + } + } + else { retrieveDeployment(); } + } + + // Init-only module (no periodic work) + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override {} + + protected: + + void service() override {} + + private: + + std::shared_ptr _instanceManager; + + std::shared_ptr _computeManager; + + std::shared_ptr _rpcEngine; + + const HiCR::Instance::instanceId_t _deployerInstanceId; + + const HiCR::Instance::instanceId_t _instanceId; + + serving::configuration::Deployment _deployment; + + __INLINE__ void sendDeploymentConfiguration() + { + // Serializing + const auto serializedDeployment = _deployment.serialize().dump(); + + // Returning serialized topology + _rpcEngine->submitReturnValue((void *)serializedDeployment.c_str(), serializedDeployment.size() + 1); + } + + /** + * Retrieves the deployment configuration from the deployer instance. + */ + void retrieveDeployment() + { + // Send request RPC + _rpcEngine->requestRPC(_deployerInstanceId, __SERVING_REQUEST_DEPLOYMENT_CONFIGURATION_RPC_NAME); + + // Wait for serialized information + auto returnValue = _rpcEngine->getReturnValue(); + + // Receiving raw serialized topology information from the worker + std::string deploymentString = (char *)returnValue->getPointer(); + + // Parsing serialized raw topology into a json object + auto deploymentJs = nlohmann::json::parse(deploymentString); + + // Creating the deployment object from the json + _deployment = configuration::Deployment(deploymentJs); + + // Freeing return value + _rpcEngine->getMemoryManager()->freeLocalMemorySlot(returnValue); + } +}; +} // namespace serving::modules::broadcastDeployment \ No newline at end of file diff --git a/platform/include/modules/channelController/module.hpp b/platform/include/modules/channelController/module.hpp new file mode 100644 index 0000000..e18627f --- /dev/null +++ b/platform/include/modules/channelController/module.hpp @@ -0,0 +1,222 @@ +#pragma once + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include + +namespace serving::modules::channelController +{ + +#define __SERVING_DEFAULT_CHANNEL_CONTROLLER_EXCHANGE_TAG 0x0000A100 + +/** + * Creates channels using a reconciliation loop. + * TODO: Right now every instance should be running this module as a service for + * a backend like MPI. use channel creator frontend to mask this. + */ +class Module final : public serving::modules::Module +{ + public: + + using edgeName_t = std::string; + using instanceId_t = HiCR::Instance::instanceId_t; + using input_t = serving::system::channels::Input; + using output_t = serving::system::channels::Output; + using channelId_t = serving::system::channels::channelId_t; + using keyBuilderFc_t = serving::system::channels::keyBuilderFc_t; + + Module(const instanceId_t instance, + const std::vector &communicationManagersInOrder, + const HiCR::GlobalMemorySlot::tag_t exchangeTag = __SERVING_DEFAULT_CHANNEL_CONTROLLER_EXCHANGE_TAG, + const size_t intervalMs = 250) + : serving::modules::Module(intervalMs), + _instanceId(instance), + _communicationManagersInOrder(communicationManagersInOrder), + _exchangeTag(exchangeTag) + { + if (_communicationManagersInOrder.empty()) HICR_THROW_LOGIC("Channel controller requires at least one communication manager."); + } + + ~Module() override = default; + + [[nodiscard]] __INLINE__ std::weak_ptr addDesiredProducer(const instanceId_t targetId, + const channelId_t channelId, + const serving::configuration::Edge &edge, + const keyBuilderFc_t &keyBuilder) + { + std::lock_guard lock(_producerMutex); + const auto &edgeName = edge.getName(); + if (_desiredProducers.contains(edgeName)) HICR_THROW_LOGIC("Desired producer for edge %s to target instance %lu already exists.", edgeName.c_str(), targetId); + _desiredProducers[edgeName] = std::make_shared(edge, channelId, _instanceId, targetId, keyBuilder); + return _desiredProducers[edgeName]; + } + + [[nodiscard]] __INLINE__ std::weak_ptr addDesiredConsumer(const instanceId_t sourceId, + const channelId_t channelId, + const serving::configuration::Edge &edge, + const keyBuilderFc_t &keyBuilder) + { + std::lock_guard lock(_consumerMutex); + const auto &edgeName = edge.getName(); + if (_desiredConsumers.contains(edgeName)) HICR_THROW_LOGIC("Desired consumer for edge %s from source instance %lu already exists.", edgeName.c_str(), sourceId); + _desiredConsumers[edgeName] = std::make_shared(edge, channelId, sourceId, _instanceId, keyBuilder); + return _desiredConsumers[edgeName]; + } + + [[nodiscard]] __INLINE__ bool hasProducer(const edgeName_t edgeName) const + { + std::lock_guard lock(_producerMutex); + return _actualProducers.contains(edgeName); + } + [[nodiscard]] __INLINE__ bool hasConsumer(const edgeName_t edgeName) const + { + std::lock_guard lock(_consumerMutex); + return _actualConsumers.contains(edgeName); + } + + [[nodiscard]] __INLINE__ std::weak_ptr getProducer(const edgeName_t edgeName) const + { + std::lock_guard lock(_producerMutex); + if (_actualProducers.contains(edgeName) == false) HICR_THROW_RUNTIME("No actual producer found for edge %s.", edgeName.c_str()); + return _actualProducers.at(edgeName); + } + + [[nodiscard]] __INLINE__ std::weak_ptr getConsumer(const edgeName_t edgeName) const + { + std::lock_guard lock(_consumerMutex); + if (_actualConsumers.contains(edgeName) == false) HICR_THROW_RUNTIME("No actual consumer found for edge %s.", edgeName.c_str()); + return _actualConsumers.at(edgeName); + } + + __INLINE__ void removeDesiredProducer(const edgeName_t edgeName) + { + std::lock_guard lock(_producerMutex); + _desiredProducers.erase(edgeName); + } + + __INLINE__ void removeDesiredConsumer(const edgeName_t edgeName) + { + std::lock_guard lock(_consumerMutex); + _desiredConsumers.erase(edgeName); + } + + void initialize() override { reconcile(); } + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override + { + std::scoped_lock lock(_producerMutex, _consumerMutex); + _desiredProducers.clear(); + _actualProducers.clear(); + _desiredConsumers.clear(); + _actualConsumers.clear(); + } + + protected: + + void service() override { reconcile(); } + + private: + + __INLINE__ void reconcile() + { + std::unordered_map> producersToCreate; + std::unordered_map> consumersToCreate; + std::vector producersToDelete; + std::vector consumersToDelete; + + // Diff desired vs actual + { + std::scoped_lock lock(_producerMutex, _consumerMutex); + for (const auto &[target, output] : _desiredProducers) + if (_actualProducers.contains(target) == false) producersToCreate[target] = output; + for (const auto &[source, input] : _desiredConsumers) + if (_actualConsumers.contains(source) == false) consumersToCreate[source] = input; + for (const auto &[target, _] : _actualProducers) + if (_desiredProducers.contains(target) == false) producersToDelete.push_back(target); + for (const auto &[source, _] : _actualConsumers) + if (_desiredConsumers.contains(source) == false) consumersToDelete.push_back(source); + } + + // Create missing channels + if (producersToCreate.empty() == false || consumersToCreate.empty() == false) + { + std::vector memorySlotsToExchange; + for (const auto &[_, output] : producersToCreate) output->getMemorySlotsToExchange(memorySlotsToExchange); + for (const auto &[_, input] : consumersToCreate) input->getMemorySlotsToExchange(memorySlotsToExchange); + + std::map> exchangeMap; + for (const auto manager : _communicationManagersInOrder) exchangeMap[manager] = {}; + + for (const auto &entry : memorySlotsToExchange) + { + if (exchangeMap.contains(entry.communicationManager) == false) HICR_THROW_LOGIC("Memory slot exchange manager is not present in configured manager order."); + exchangeMap[entry.communicationManager].push_back({entry.globalKey, entry.memorySlot}); + } + + for (const auto manager : _communicationManagersInOrder) manager->exchangeGlobalMemorySlots(_exchangeTag, exchangeMap[manager]); + for (const auto manager : _communicationManagersInOrder) manager->fence(_exchangeTag); + + for (const auto &[_, output] : producersToCreate) output->initialize(_exchangeTag); + for (const auto &[_, input] : consumersToCreate) input->initialize(_exchangeTag); + + { + std::scoped_lock lock(_producerMutex, _consumerMutex); + for (const auto &[target, output] : producersToCreate) { _actualProducers[target] = output; } + for (const auto &[source, input] : consumersToCreate) { _actualConsumers[source] = input; } + } + } + + // Delete stale channels + std::vector> staleProducers; + std::vector> staleConsumers; + { + std::scoped_lock lock(_producerMutex, _consumerMutex); + staleProducers.reserve(producersToDelete.size()); + for (const auto &target : producersToDelete) + { + auto it = _actualProducers.find(target); + if (it == _actualProducers.end()) continue; + staleProducers.push_back(std::move(it->second)); + _actualProducers.erase(it); + } + staleConsumers.reserve(consumersToDelete.size()); + for (const auto &source : consumersToDelete) + { + auto it = _actualConsumers.find(source); + if (it == _actualConsumers.end()) continue; + staleConsumers.push_back(std::move(it->second)); + _actualConsumers.erase(it); + } + } + } + + const instanceId_t _instanceId; + + mutable std::mutex _producerMutex; + mutable std::mutex _consumerMutex; + const std::vector _communicationManagersInOrder; + const HiCR::GlobalMemorySlot::tag_t _exchangeTag; + + // Desired state + std::unordered_map> _desiredProducers; // key: target instance + std::unordered_map> _desiredConsumers; // key: source instance + + // Actual created state + std::unordered_map> _actualProducers; // key: target instance + std::unordered_map> _actualConsumers; // key: source instance +}; +} // namespace serving::modules::channelController diff --git a/platform/include/modules/channelDispatcher/module.hpp b/platform/include/modules/channelDispatcher/module.hpp new file mode 100644 index 0000000..25d73f7 --- /dev/null +++ b/platform/include/modules/channelDispatcher/module.hpp @@ -0,0 +1,127 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +#include +#include + +#include + +namespace serving::modules::channelDispatcher +{ + +class Module final : public modules::Module +{ + public: + + Module(const size_t intervalMs) + : modules::Module(intervalMs) + {} + + ~Module() override = default; + + __INLINE__ void subscribe(const Subscription &subscription) + { + const auto &edge = subscription.getEdge(); + + std::lock_guard guard(_subscriptionMutex); + + _subscribedEdges.insert(edge); + + const auto key = std::make_pair(edge.get(), subscription.getType()); + if (_subscriptionToHandlerMap.contains(key)) HICR_THROW_LOGIC("A handler is already subscribed for this input edge and message type."); + _subscriptionToHandlerMap.insert({key, subscription.getHandler()}); + } + + __INLINE__ void unsubscribe(const channels::Message::messageType_t type, const std::shared_ptr edge) + { + std::lock_guard guard(_subscriptionMutex); + + const auto key = std::make_pair(edge.get(), type); + if (_subscriptionToHandlerMap.contains(key) == false) return; + _subscriptionToHandlerMap.erase(key); + + bool hasRemainingHandlers = false; + for (const auto &[channelMessagePair, _] : _subscriptionToHandlerMap) + if (channelMessagePair.first == edge.get()) + { + hasRemainingHandlers = true; + break; + } + if (hasRemainingHandlers == false) _subscribedEdges.erase(edge); + } + + __INLINE__ void poll() + { + // reduce contention + _pollEdges.clear(); + { + std::lock_guard guard(_subscriptionMutex); + _pollEdges.reserve(_subscribedEdges.size()); + _pollEdges.assign(_subscribedEdges.begin(), _subscribedEdges.end()); + } + + for (const auto &edge : _pollEdges) + { + edge->lock(); + struct EdgeUnlockGuard + { + const std::shared_ptr &edge; + ~EdgeUnlockGuard() { edge->unlock(); } + } unlockGuard{edge}; + + if (edge->hasMessage() == false) { continue; } + + const auto message = edge->getMessage(); + const auto messageType = message.getMetadata().type; + const auto key = std::make_pair(edge.get(), messageType); + messageHandler_t handler; + { + std::lock_guard guard(_subscriptionMutex); + if (_subscriptionToHandlerMap.contains(key) == false) + { + printf("[ChannelDispatcher] No handler found for message type %u. Message will be ignored.\n", messageType); + edge->popMessage(); + continue; + } + handler = _subscriptionToHandlerMap.at(key); + } + handler(edge, message); + edge->popMessage(); + } + } + + void initialize() override {} + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override + { + _subscribedEdges.clear(); + _subscriptionToHandlerMap.clear(); + } + + protected: + + void service() override { poll(); } + + private: + + std::mutex _subscriptionMutex; + std::set> _subscribedEdges; + std::map, messageHandler_t> _subscriptionToHandlerMap; + + // Used to reduce contention in poll() + std::vector> _pollEdges; +}; +} // namespace serving::modules::channelDispatcher diff --git a/platform/include/modules/configuration/deployment.hpp b/platform/include/modules/configuration/deployment.hpp new file mode 100644 index 0000000..68aba09 --- /dev/null +++ b/platform/include/modules/configuration/deployment.hpp @@ -0,0 +1,353 @@ +#pragma once + +#include +#include +#include +#include + +#include + +#include +#include + +#include "partition.hpp" +#include "edge.hpp" +#include "requestManager.hpp" + +namespace serving::configuration +{ + +class Deployment final +{ + public: + + /** + * Heartbeat is a runtime check mechanism to verify neighbors and replicas are still alive. + * Every 'interval' milliseconds, the coordinator sends a heartbeat message to its own replicas and vice-versa + * If the 'tolerance' interval passes without a heartbeat, then the other side is deemed non-responsive + * Recovery or fail tolerance mechanisms can be activated upon this happening + */ + struct heartbeat_t + { + // Whether to enable the heartbeat service or not + bool enabled; + + // Whether to print the hearbeat arriving on screen + bool visible; + + // Heartbeat interval (every how many ms do we send a heartbeat message to replicas and neighboring coordinators) + size_t interval; + + // Tolerance is how long to wait before alerting that we haven't received a heartbeat + size_t tolerance; + }; + + /** + * The control buffer is the edge between coordinators and replicas to exchange control messages. + * Control messages are not related to the application, but instead to managing deployment aspects + * (e.g., load balancing, fail detection, migration) + */ + static constexpr size_t _defaultControlBufferCapacity = 256; + static constexpr size_t _defaultControlBufferSize = _defaultControlBufferCapacity * 1024; + struct controlBuffer_t + { + // Capacity (the maximum number of pending tokens in the buffer) + size_t capacity = _defaultControlBufferCapacity; + + // Size (the size in bytes that the buffer can hold) + size_t size = _defaultControlBufferSize; + + // HiCR-specific objects to create the control buffers. These are to be set at runtime + HiCR::CommunicationManager *communicationManager = nullptr; + HiCR::MemoryManager *memoryManager = nullptr; + std::shared_ptr memorySpace = nullptr; + }; + + struct settings_t + { + heartbeat_t heartbeat; + controlBuffer_t controlBuffer; + }; + + Deployment(const std::string &name) + : _name(name){}; + Deployment(const nlohmann::json &js) { deserialize(js); }; + Deployment() = default; + ~Deployment() = default; + + __INLINE__ void setName(const std::string &name) { _name = name; } + __INLINE__ void addPartition(const std::shared_ptr partition) { _partitions.push_back(partition); } + __INLINE__ void addChannel(const std::shared_ptr channel) { _edges.push_back(channel); } + + [[nodiscard]] __INLINE__ std::string getName() const { return _name; } + [[nodiscard]] __INLINE__ auto &getPartitions() const { return _partitions; } + [[nodiscard]] __INLINE__ auto &getEdges() const { return _edges; } + [[nodiscard]] __INLINE__ auto &getHeartbeat() const { return _settings.heartbeat; } + [[nodiscard]] __INLINE__ const auto &getControlBuffer() const { return _settings.controlBuffer; } + [[nodiscard]] __INLINE__ auto &getControlBuffer() { return _settings.controlBuffer; } + [[nodiscard]] __INLINE__ auto &getControlBufferConst() const { return _settings.controlBuffer; } + [[nodiscard]] __INLINE__ auto &getRequestManager() const { return _requestManager; } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + + js["Name"] = _name; + + std::vector partitionsJs; + for (const auto &p : _partitions) partitionsJs.push_back(p->serialize()); + js["Partitions"] = partitionsJs; + + std::vector edgesJs; + for (const auto &e : _edges) edgesJs.push_back(e->serialize()); + js["Edges"] = edgesJs; + + js["Request Manager"] = _requestManager->serialize(); + + ////////////////////// Parsing settings + auto settings = std::map(); + + // Heartbeat + auto heartbeat = std::map(); + heartbeat["Enabled"] = _settings.heartbeat.enabled; + heartbeat["Visible"] = _settings.heartbeat.visible; + heartbeat["Interval"] = _settings.heartbeat.interval; + heartbeat["Tolerance"] = _settings.heartbeat.tolerance; + settings["Heartbeat"] = heartbeat; + + // Control Buffer + auto controlBuffer = std::map(); + controlBuffer["Capacity"] = _settings.controlBuffer.capacity; + controlBuffer["Size"] = _settings.controlBuffer.size; + settings["Control Buffer"] = controlBuffer; + + js["Settings"] = settings; + + return js; + } + + __INLINE__ void deserialize(const nlohmann::json &js) + { + // Clearing objects + _partitions.clear(); + _edges.clear(); + + _name = hicr::json::getString(js, "Name"); + + const auto &partitions = hicr::json::getArray(js, "Partitions"); + for (const auto &p : partitions) _partitions.push_back(std::make_shared(p)); + + const auto &edges = hicr::json::getArray(js, "Edges"); + for (const auto &e : edges) _edges.push_back(std::make_shared(e)); + + const auto &requestManagerJs = hicr::json::getObject(js, "Request Manager"); + _requestManager = std::make_shared(requestManagerJs); + + // Getting settings + nlohmann::json settingsJs = hicr::json::getObject(js, "Settings"); + + nlohmann::json heartbeatJs = hicr::json::getObject(settingsJs, "Heartbeat"); + _settings.heartbeat.enabled = hicr::json::getBoolean(heartbeatJs, "Enabled"); + _settings.heartbeat.visible = hicr::json::getBoolean(heartbeatJs, "Visible"); + _settings.heartbeat.interval = hicr::json::getNumber(heartbeatJs, "Interval"); + _settings.heartbeat.tolerance = hicr::json::getNumber(heartbeatJs, "Tolerance"); + + nlohmann::json controlBufferJs = hicr::json::getObject(settingsJs, "Control Buffer"); + _settings.controlBuffer.capacity = hicr::json::getNumber(controlBufferJs, "Capacity"); + _settings.controlBuffer.size = hicr::json::getNumber(controlBufferJs, "Size"); + } + + // Includes all kinds of sanity checks relevant to a deployment + __INLINE__ void verify() const + { + // Getting rqeuest manager input and output edges + const auto &requestManagerInputEdge = _requestManager->getInput(); + const auto &requestManagerOutputEdge = _requestManager->getOutput(); + + // Getting all partition's edges in a set + std::set partitionNameSet; + for (const auto &partition : _partitions) partitionNameSet.insert(partition->getName()); + + // Getting a list of all edges to check the tasks specifying inputs/outputs actually refer to one of these, or user interface + std::set edgeNameSet; + + // Getting a list of all the inputs and output names to verify they correspond to each other at least once + std::set inputSet; + std::set outputSet; + + // Storage for those partitions with user interface input and output + std::shared_ptr userInterfaceInputPartition = nullptr; + std::shared_ptr userInterfaceOutputPartition = nullptr; + + // Now checking all dependencies belong in the same partition + std::set tasksWithDependencies; + for (const auto &partition : _partitions) + { + // Creating set of task names of this partition + std::set _taskNames; + for (const auto &task : partition->getTasks()) _taskNames.insert(task->getFunctionName()); + + for (const auto &task : partition->getTasks()) + for (const auto &dependency : task->getDependencies()) + { + // Check the task doesn't depend on itself + if (dependency == task->getFunctionName()) HICR_THROW_LOGIC("Task %s has a dependency on itself\n", task->getFunctionName().c_str()); + + // Check the dependency has a task that is referenced by this dependency + if (_taskNames.contains(dependency) == false) + HICR_THROW_LOGIC("Task %s has a dependency on '%s' which is not defined in this partition\n", task->getFunctionName().c_str(), dependency.c_str()); + + // Adding dependency to tasks with dependencies + tasksWithDependencies.insert(dependency); + } + } + + // First, the user interface input/outputs are counted as edges for the purposes of this check + for (const auto &edge : _edges) + { + const auto &edgeName = edge->getName(); + if (edgeNameSet.contains(edgeName)) HICR_THROW_LOGIC("Deployment specifies repeated edge or user input '%s'\n", edgeName.c_str()); + edgeNameSet.insert(edgeName); + } + + // For each input / output, remember which partition is its consumer / producer + std::map consumerPartitionMap; + std::map producerPartitionMap; + for (const auto &partition : _partitions) + for (const auto &task : partition->getTasks()) + { + // Make sure all tasks have at least one input + if (task->getInputs().size() == 0 && task->getDependencies().empty() == true) + HICR_THROW_LOGIC("Deployment specifies task in partition '%s' with function name '%s' without any inputs or dependencies\n", + partition->getName().c_str(), + task->getFunctionName().c_str()); + + // Check that the edge is not used as input more than once. All edges must be 1-to-1 + for (const auto &input : task->getInputs()) + { + if (inputSet.contains(input)) HICR_THROW_LOGIC("Deployment specifies input '%s' used more than once\n", input.c_str()); + if (edgeNameSet.contains(input) == false) + HICR_THROW_LOGIC("Deployment specifies task '%s' with an undefined input '%s'\n", task->getFunctionName().c_str(), input.c_str()); + consumerPartitionMap[input] = partition->getName(); + inputSet.insert(input); + + // Check the task that contains the user interface input does not receive any other inputs + if (input == requestManagerInputEdge) userInterfaceInputPartition = partition; + if (input == requestManagerOutputEdge) + HICR_THROW_LOGIC("Deployment specifies task '%s' with user interface output '%s' which is being used as input\n", task->getFunctionName().c_str(), input.c_str()); + if (input == requestManagerInputEdge && task->getInputs().size() > 1) + HICR_THROW_LOGIC( + "Deployment specifies task '%s' with user interface input '%s' which is not the only input of that task\n", task->getFunctionName().c_str(), input.c_str()); + } + + // Make sure all tasks have at least one output + if (task->getOutputs().size() == 0 && tasksWithDependencies.contains(task->getFunctionName()) == false) + HICR_THROW_LOGIC("Deployment specifies task in partition '%s' with function name '%s' without any outputs or dependents\n", + partition->getName().c_str(), + task->getFunctionName().c_str()); + + // Check that the output is not used as input more than once. All edges must be 1-to-1 + for (const auto &output : task->getOutputs()) + { + if (outputSet.contains(output)) HICR_THROW_LOGIC("Deployment specifies output '%s' used more than once\n", output.c_str()); + if (edgeNameSet.contains(output) == false) + HICR_THROW_LOGIC("Deployment specifies task '%s' with an undefined output '%s'\n", task->getFunctionName().c_str(), output.c_str()); + producerPartitionMap[output] = partition->getName(); + outputSet.insert(output); + + // Check the task that contains the user interface input does not receive any other inputs + if (output == requestManagerOutputEdge) userInterfaceOutputPartition = partition; + if (output == requestManagerInputEdge) + HICR_THROW_LOGIC("Deployment specifies task '%s' with user interface input '%s' which is being used as output\n", task->getFunctionName().c_str(), output.c_str()); + if (output == requestManagerOutputEdge && task->getOutputs().size() > 1) + HICR_THROW_LOGIC( + "Deployment specifies task '%s' with user interface output '%s' which is not the only output of task\n", task->getFunctionName().c_str(), output.c_str()); + } + } + + // Check whether all edges have consumer+producer partitions that do exist + for (const auto &edge : _edges) + { + const auto &edgeName = edge->getName(); + if (edgeName != requestManagerInputEdge) + if (edgeName != requestManagerOutputEdge) + if (consumerPartitionMap.contains(edgeName) == false || producerPartitionMap.contains(edgeName) == false) + HICR_THROW_LOGIC("Deployment specifies edge '%s' but it is either not used as input or output (or neither)\n", edge->getName().c_str()); + + // Getting producer and consumer partitions + const auto &producerPartition = producerPartitionMap[edge->getName()]; + const auto &consumerPartition = consumerPartitionMap[edge->getName()]; + + // Checking the edge connects two different partitions + if (producerPartition == consumerPartition) + HICR_THROW_LOGIC("Deployment specifies edge '%s' that is both produced and consumed by partition %s\n", edge->getName().c_str(), producerPartition.c_str()); + + // Setting producer and consumer partitions for the edge + edge->setProducer(producerPartition); + edge->setConsumer(consumerPartition); + } + + // Make sure all inputs are also used as outputs, as long as it is not the user interface input + for (const auto &input : inputSet) + if (input != requestManagerInputEdge) + if (outputSet.contains(input) == false) HICR_THROW_LOGIC("Deployment input '%s' is not associated to any output\n", input.c_str()); + for (const auto &output : outputSet) + if (output != requestManagerOutputEdge) + if (inputSet.contains(output) == false) HICR_THROW_LOGIC("Deployment output '%s' is not associated to any input\n", output.c_str()); + + // Check the user interface input/output are being used + if (userInterfaceInputPartition == nullptr) HICR_THROW_LOGIC("User interface input '%s' is not associated to any partition\n", requestManagerInputEdge.c_str()); + if (userInterfaceOutputPartition == nullptr) HICR_THROW_LOGIC("User interface output '%s' is not associated to any partition\n", requestManagerOutputEdge.c_str()); + + // Make sure the partition which contains the user interface input does not contain any cross-partition inputs + for (const auto &task : userInterfaceInputPartition->getTasks()) + for (const auto &input : task->getInputs()) + if (input != requestManagerInputEdge) + if (producerPartitionMap.at(input) != userInterfaceInputPartition->getName()) + HICR_THROW_LOGIC("Partition %s consumes the user interface input '%s' but also has other inter-partition inputs (e.g.,: '%s')\n", + userInterfaceInputPartition->getName().c_str(), + requestManagerInputEdge.c_str(), + input.c_str()); + + // Make sure the partition which contains the user interface output does not contain any cross-partition outputs + for (const auto &task : userInterfaceOutputPartition->getTasks()) + for (const auto &output : task->getOutputs()) + if (output != requestManagerOutputEdge) + if (consumerPartitionMap.at(output) != userInterfaceOutputPartition->getName()) + HICR_THROW_LOGIC("Partition %s produces the user interface output '%s' but also has other inter-partition inputs (e.g.,: '%s')\n", + userInterfaceOutputPartition->getName().c_str(), + requestManagerOutputEdge.c_str(), + output.c_str()); + + // Setting producer partition for user interface input to be the same as the consumer + for (const auto &edge : _edges) + { + const auto &edgeName = edge->getName(); + + if (edgeName == requestManagerInputEdge) + { + edge->setProducer(userInterfaceInputPartition->getName()); + edge->setConsumer(userInterfaceInputPartition->getName()); + edge->setPromptEdge(true); + } + + if (edgeName == requestManagerOutputEdge) + { + edge->setProducer(userInterfaceOutputPartition->getName()); + edge->setConsumer(userInterfaceOutputPartition->getName()); + edge->setResultEdge(true); + } + } + } + + private: + + std::string _name; + std::vector> _partitions; + std::vector> _edges; + std::shared_ptr _requestManager; + settings_t _settings; + +}; // class Deployment + +} // namespace serving::configuration \ No newline at end of file diff --git a/platform/include/modules/configuration/edge.hpp b/platform/include/modules/configuration/edge.hpp new file mode 100644 index 0000000..60da64d --- /dev/null +++ b/platform/include/modules/configuration/edge.hpp @@ -0,0 +1,108 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace serving::configuration +{ + +class Edge +{ + public: + +#define __SERVING_PARTITION_DEFAULT_BUFFER_CAPACITY 1 + + typedef uint64_t edgeIndex_t; + + Edge(const nlohmann::json &js) { deserialize(js); } + Edge(const std::string &name, const size_t bufferCapacity, const size_t bufferSize = 0) + : _name(name), + _bufferCapacity(bufferCapacity), + _bufferSize(bufferSize) + {} + virtual ~Edge() = default; + + __INLINE__ void setName(const std::string &name) { _name = name; } + [[nodiscard]] __INLINE__ std::string getName() const { return _name; } + [[nodiscard]] __INLINE__ std::string getProducer() const { return _producer; } + [[nodiscard]] __INLINE__ std::string getConsumer() const { return _consumer; } + [[nodiscard]] __INLINE__ size_t getBufferCapacity() const { return _bufferCapacity; } + [[nodiscard]] __INLINE__ size_t getBufferSize() const { return _bufferSize; } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + + js["Name"] = _name; + js["Producer"] = _producer; + js["Consumer"] = _consumer; + js["Buffer Capacity"] = _bufferCapacity; + js["Buffer Size"] = _bufferSize; + + return js; + } + + __INLINE__ void deserialize(const nlohmann::json &js) + { + _name = hicr::json::getString(js, "Name"); + if (js.contains("Producer")) _producer = hicr::json::getString(js, "Producer"); + if (js.contains("Consumer")) _consumer = hicr::json::getString(js, "Consumer"); + if (js.contains("Buffer Capacity")) _bufferCapacity = hicr::json::getNumber(js, "Buffer Capacity"); + _bufferSize = hicr::json::getNumber(js, "Buffer Size"); + } + + // Functions to set the HiCR elements required for the creation of edge channels + __INLINE__ void setPayloadCommunicationManager(HiCR::CommunicationManager *const communicationManager) { _payloadCommunicationManager = communicationManager; } + __INLINE__ void setPayloadMemoryManager(HiCR::MemoryManager *const memoryManager) { _payloadMemoryManager = memoryManager; } + __INLINE__ void setPayloadMemorySpace(const std::shared_ptr memorySpace) { _payloadMemorySpace = memorySpace; } + __INLINE__ void setCoordinationCommunicationManager(HiCR::CommunicationManager *const communicationManager) { _coordinationCommunicationManager = communicationManager; } + __INLINE__ void setCoordinationMemoryManager(HiCR::MemoryManager *const memoryManager) { _coordinationMemoryManager = memoryManager; } + __INLINE__ void setCoordinationMemorySpace(const std::shared_ptr memorySpace) { _coordinationMemorySpace = memorySpace; } + + __INLINE__ void setProducer(const std::string &partition) { _producer = partition; } + __INLINE__ void setConsumer(const std::string &partition) { _consumer = partition; } + __INLINE__ void setPromptEdge(const bool isPromptEdge) { _isPromptEdge = isPromptEdge; } + __INLINE__ void setResultEdge(const bool isResultEdge) { _isResultEdge = isResultEdge; } + __INLINE__ void setBufferCapacity(const size_t bufferCapacity) { _bufferCapacity = bufferCapacity; } + __INLINE__ void setBufferSize(const size_t bufferSize) { _bufferSize = bufferSize; } + + // Functions to set the HiCR elements required for the creation of edge channels + [[nodiscard]] __INLINE__ HiCR::CommunicationManager *getPayloadCommunicationManager() const { return _payloadCommunicationManager; } + [[nodiscard]] __INLINE__ HiCR::MemoryManager *getPayloadMemoryManager() const { return _payloadMemoryManager; } + [[nodiscard]] __INLINE__ std::shared_ptr getPayloadMemorySpace() const { return _payloadMemorySpace; } + [[nodiscard]] __INLINE__ HiCR::CommunicationManager *getCoordinationCommunicationManager() const { return _coordinationCommunicationManager; } + [[nodiscard]] __INLINE__ HiCR::MemoryManager *getCoordinationMemoryManager() const { return _coordinationMemoryManager; } + [[nodiscard]] __INLINE__ std::shared_ptr getCoordinationMemorySpace() const { return _coordinationMemorySpace; } + [[nodiscard]] __INLINE__ bool isPromptEdge() const { return _isPromptEdge; } + [[nodiscard]] __INLINE__ bool isResultEdge() const { return _isResultEdge; } + + private: + + std::string _name; + std::string _producer; + std::string _consumer; + size_t _bufferCapacity = __SERVING_PARTITION_DEFAULT_BUFFER_CAPACITY; + size_t _bufferSize; + + // This flag serves to indicate whether these are edges that connect to the request manager + bool _isPromptEdge = false; + bool _isResultEdge = false; + + // HiCR-specific objects to create the payload buffers. These are to be set at runtime + HiCR::CommunicationManager *_payloadCommunicationManager = nullptr; + HiCR::MemoryManager *_payloadMemoryManager = nullptr; + std::shared_ptr _payloadMemorySpace = nullptr; + + // HiCR-specific objects to create the coordination buffers. These are to be set at runtime + HiCR::CommunicationManager *_coordinationCommunicationManager = nullptr; + HiCR::MemoryManager *_coordinationMemoryManager = nullptr; + std::shared_ptr _coordinationMemorySpace = nullptr; + +}; // class Base + +} // namespace serving::configuration \ No newline at end of file diff --git a/platform/include/modules/configuration/partition.hpp b/platform/include/modules/configuration/partition.hpp new file mode 100644 index 0000000..e47f79c --- /dev/null +++ b/platform/include/modules/configuration/partition.hpp @@ -0,0 +1,84 @@ +#pragma once + +#include +#include +#include "task.hpp" +#include "replica.hpp" +#include +#include +#include + +namespace serving::configuration +{ + +class Partition final +{ + public: + + typedef uint64_t partitionIndex_t; + + Partition(const nlohmann::json js) { deserialize(js); }; + Partition(const std::string &name, const HiCR::Instance::instanceId_t instanceId) + : _name(name), + _coordinatorInstanceId(instanceId) + {} + ~Partition() = default; + + __INLINE__ void setName(const std::string &name) { _name = name; } + __INLINE__ void setCoordinatorInstanceId(const HiCR::Instance::instanceId_t instanceId) { _coordinatorInstanceId = instanceId; } + __INLINE__ void addTask(const std::shared_ptr task) { _tasks.push_back(task); } + __INLINE__ void addReplica(const std::shared_ptr replica) { _replicas.push_back(replica); } + + [[nodiscard]] __INLINE__ auto getName() const { return _name; } + [[nodiscard]] __INLINE__ auto getCoordinatorInstanceId() const { return _coordinatorInstanceId; } + [[nodiscard]] __INLINE__ auto &getTasks() const { return _tasks; } + [[nodiscard]] __INLINE__ auto &getReplicas() const { return _replicas; } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + + js["Name"] = _name; + js["Coordinator Instance Id"] = _coordinatorInstanceId; + + std::vector tasksJs; + for (const auto &t : _tasks) tasksJs.push_back(t->serialize()); + js["Tasks"] = tasksJs; + + std::vector replicasJs; + for (const auto &r : _replicas) replicasJs.push_back(r->serialize()); + js["Replicas"] = replicasJs; + + return js; + } + + __INLINE__ void deserialize(const nlohmann::json &js) + { + // Clearing objects + _tasks.clear(); + _replicas.clear(); + + _name = hicr::json::getString(js, "Name"); + if (js.contains("Coordinator Instance Id")) + _coordinatorInstanceId = hicr::json::getNumber(js, "Coordinator Instance Id"); // Optional, as it is determined at runtime + + const auto &tasks = hicr::json::getArray(js, "Tasks"); + for (const auto &t : tasks) _tasks.push_back(std::make_shared(t)); + + // This entry is optional, as it can be decided at runtime + if (js.contains("Replicas")) + { + const auto &replicas = hicr::json::getArray(js, "Replicas"); + for (const auto &r : replicas) _replicas.push_back(std::make_shared(r)); + } + } + + private: + + std::string _name; + HiCR::Instance::instanceId_t _coordinatorInstanceId = 0; + std::vector> _tasks; + std::vector> _replicas; +}; // class Partition + +} // namespace serving::configuration \ No newline at end of file diff --git a/platform/include/modules/configuration/replica.hpp b/platform/include/modules/configuration/replica.hpp new file mode 100644 index 0000000..9d933c5 --- /dev/null +++ b/platform/include/modules/configuration/replica.hpp @@ -0,0 +1,47 @@ +#pragma once + +#include +#include +#include +#include + +namespace serving::configuration +{ + +class Replica final +{ + public: + + typedef uint64_t replicaIndex_t; + + Replica(const nlohmann::json js) { deserialize(js); }; + Replica(const HiCR::Instance::instanceId_t instanceId) + : _instanceId(instanceId) + {} + ~Replica() = default; + + __INLINE__ void setInstanceId(const HiCR::Instance::instanceId_t instanceId) { _instanceId = instanceId; } + + [[nodiscard]] __INLINE__ auto getInstanceId() const { return _instanceId; } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + + js["Instance Id"] = _instanceId; + + return js; + } + + __INLINE__ void deserialize(const nlohmann::json &js) + { + if (js.contains("Instance Id")) _instanceId = hicr::json::getNumber(js, "Instance Id"); // Optional, as it is determined at runtime + } + + private: + + HiCR::Instance::instanceId_t _instanceId; + +}; // class Replica + +} // namespace serving::configuration \ No newline at end of file diff --git a/platform/include/modules/configuration/requestManager.hpp b/platform/include/modules/configuration/requestManager.hpp new file mode 100644 index 0000000..30177d7 --- /dev/null +++ b/platform/include/modules/configuration/requestManager.hpp @@ -0,0 +1,56 @@ +#pragma once + +#include +#include +#include +#include + +namespace serving::configuration +{ + +class RequestManager final +{ + public: + + RequestManager(const nlohmann::json js) { deserialize(js); }; + RequestManager(const std::string &input, const std::string &output) + : _input(input), + _output(output) + {} + ~RequestManager() = default; + + __INLINE__ void setInput(const std::string &input) { _input = input; } + __INLINE__ void setOutput(const std::string &output) { _output = output; } + __INLINE__ void setInstanceId(const HiCR::Instance::instanceId_t instanceId) { _instanceId = instanceId; } + + [[nodiscard]] __INLINE__ auto getInput() const { return _input; } + [[nodiscard]] __INLINE__ auto getOutput() const { return _output; } + [[nodiscard]] __INLINE__ auto getInstanceId() const { return _instanceId; } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + + js["Input"] = _input; + js["Output"] = _output; + js["Instance Id"] = _instanceId; + + return js; + } + + __INLINE__ void deserialize(const nlohmann::json &js) + { + _input = hicr::json::getString(js, "Input"); + _output = hicr::json::getString(js, "Output"); + if (js.contains("Instance Id")) _instanceId = hicr::json::getNumber(js, "Instance Id"); // Optional, as it is determined at runtime + } + + private: + + std::string _input; + std::string _output; + HiCR::Instance::instanceId_t _instanceId; + +}; // class RequestManager + +} // namespace serving::configuration \ No newline at end of file diff --git a/platform/include/modules/configuration/task.hpp b/platform/include/modules/configuration/task.hpp new file mode 100644 index 0000000..c04582b --- /dev/null +++ b/platform/include/modules/configuration/task.hpp @@ -0,0 +1,60 @@ +#pragma once + +#include +#include +#include +#include + +namespace serving::configuration +{ + +class Task final +{ + public: + + Task(const nlohmann::json &js) { deserialize(js); }; + Task(const std::string &functionName) + : _functionName(functionName) + {} + ~Task() = default; + + __INLINE__ void setFunctionName(const std::string &functionName) { _functionName = functionName; } + __INLINE__ void addInput(const std::string &input) { _inputs.push_back(input); } + __INLINE__ void addOutput(const std::string &output) { _outputs.push_back(output); } + __INLINE__ void addDependency(const std::string &functionName) { _dependencies.push_back(functionName); } + + [[nodiscard]] __INLINE__ auto getFunctionName() const { return _functionName; } + [[nodiscard]] __INLINE__ auto &getInputs() const { return _inputs; } + [[nodiscard]] __INLINE__ auto &getOutputs() const { return _outputs; } + [[nodiscard]] __INLINE__ auto &getDependencies() const { return _dependencies; } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + + js["Function Name"] = _functionName; + js["Inputs"] = _inputs; + js["Outputs"] = _outputs; + js["Dependencies"] = _dependencies; + + return js; + } + + __INLINE__ void deserialize(const nlohmann::json &js) + { + _functionName = hicr::json::getString(js, "Function Name"); + _inputs = hicr::json::getArray(js, "Inputs"); + _outputs = hicr::json::getArray(js, "Outputs"); + _dependencies = hicr::json::getArray(js, "Dependencies"); + } + + private: + + std::string _functionName; + std::vector _inputs; + std::vector _outputs; + std::vector _dependencies; + +}; // class Task + +} // namespace serving::configuration \ No newline at end of file diff --git a/platform/include/modules/heartbeat/module.hpp b/platform/include/modules/heartbeat/module.hpp new file mode 100644 index 0000000..4fbe874 --- /dev/null +++ b/platform/include/modules/heartbeat/module.hpp @@ -0,0 +1,209 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace serving::modules::heartbeat +{ + +class Module final : public serving::modules::Module +{ + public: + + using instanceId_t = HiCR::Instance::instanceId_t; + using messageType_t = serving::system::channels::Message::messageType_t; + using input_t = std::shared_ptr; + using output_t = std::shared_ptr; + using message_t = serving::system::channels::Message; + + enum class health_t : uint8_t + { + unknown = 0, + healthy = 1, + unhealthy = 2 + }; + + inline static std::string health_tToString(health_t state) + { + switch (state) + { + case health_t::healthy: return "Healthy"; + case health_t::unhealthy: return "Unhealthy"; + case health_t::unknown: return "Unknown"; + } + + HICR_THROW_LOGIC("Invalid health state %u.", static_cast(state)); + } + + struct healthEvent_t + { + instanceId_t instanceId; + health_t previousHealth; + health_t newHealth; + std::chrono::steady_clock::time_point timestamp; + }; + + using healthChangeCallback_t = std::function; + + Module(const instanceId_t instanceId, + const size_t toleranceMs, + const healthChangeCallback_t &healthChangeCallback, + system::channels::MessageTypeRegistry &messageTypeRegistry, + const size_t intervalMs) + : serving::modules::Module(intervalMs), + _instanceId(instanceId), + _toleranceMs(toleranceMs), + _healthChangeCallback(healthChangeCallback), + _messageType(messageTypeRegistry.registerType("modules.heartbeat.heartbeat")) + { + if (_toleranceMs < intervalMs) HICR_THROW_LOGIC("[Heartbeat] tolerance must be >= interval."); + } + + ~Module() override = default; + + __INLINE__ void addInput(const instanceId_t instanceId, const input_t &input) + { + if (_inputs.contains(instanceId)) HICR_THROW_LOGIC("[Heartbeat] input already exists for instance %lu.", instanceId); + _inputs[instanceId] = input; + _health[instanceId] = health_t::unknown; + _lastSeen[instanceId] = std::chrono::steady_clock::time_point::min(); + } + + __INLINE__ void removeInput(const instanceId_t instanceId) + { + _inputs.erase(instanceId); + _health.erase(instanceId); + _lastSeen.erase(instanceId); + } + + __INLINE__ void addOutput(const instanceId_t instanceId, const output_t &output) + { + if (_outputs.contains(instanceId)) HICR_THROW_LOGIC("[Heartbeat] output already exists for instance %lu.", instanceId); + _outputs[instanceId] = output; + } + + __INLINE__ void removeOutput(const instanceId_t instanceId) { _outputs.erase(instanceId); } + + [[nodiscard]] __INLINE__ health_t getHealth(const instanceId_t instanceId) const + { + if (_health.contains(instanceId) == false) return health_t::unknown; + return _health.at(instanceId); + } + + [[nodiscard]] __INLINE__ const std::unordered_map &getHealthSnapshot() const { return _health; } + + [[nodiscard]] __INLINE__ std::vector buildSubscriptions() + { + std::vector out; + out.reserve(_inputs.size()); + for (const auto &[instanceId, input] : _inputs) + { + out.emplace_back(_messageType, input, [this, instanceId](const input_t, const message_t &message) { this->heartbeatMessageHandler(instanceId, message); }); + } + return out; + } + + [[nodiscard]] __INLINE__ std::vector> buildUnsubscriptions() const + { + std::vector> out; + out.reserve(_inputs.size()); + for (const auto &[_, input] : _inputs) out.push_back({_messageType, input}); + return out; + } + + void initialize() override {} + + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override + { + _inputs.clear(); + _outputs.clear(); + _health.clear(); + _lastSeen.clear(); + } + + protected: + + void service() override + { + checkTimeouts(); + sendHeartbeats(); + } + + private: + + __INLINE__ void emitHealthEvent(const instanceId_t instanceId, const health_t previousHealth, const health_t newHealth, const std::chrono::steady_clock::time_point now) + { + _healthChangeCallback(healthEvent_t{.instanceId = instanceId, .previousHealth = previousHealth, .newHealth = newHealth, .timestamp = now}); + } + + __INLINE__ void heartbeatMessageHandler(const instanceId_t instanceId, const message_t &message) + { + if (message.getMetadata().type != _messageType) HICR_THROW_RUNTIME("[Heartbeat] unexpected message type %lu.", message.getMetadata().type); + const auto now = std::chrono::steady_clock::now(); + const auto previous = _health[instanceId]; + _lastSeen[instanceId] = now; + _health[instanceId] = health_t::healthy; + emitHealthEvent(instanceId, previous, _health[instanceId], now); + } + + __INLINE__ void checkTimeouts() + { + const auto now = std::chrono::steady_clock::now(); + const auto timeout = std::chrono::milliseconds(_toleranceMs); + for (const auto &[instanceId, _] : _inputs) + { + if (_lastSeen[instanceId] == std::chrono::steady_clock::time_point::min()) continue; + if (now - _lastSeen[instanceId] > timeout) + { + const auto previous = _health[instanceId]; + _health[instanceId] = health_t::unhealthy; + emitHealthEvent(instanceId, previous, _health[instanceId], now); + } + } + } + + __INLINE__ void sendHeartbeats() + { + const uint8_t payload = 0; + for (const auto &[_, output] : _outputs) + { + serving::system::channels::Message::metadata_t metadata; + metadata.type = _messageType; + metadata.groupId = static_cast(_instanceId); + metadata.sequenceId = 0; + const message_t heartbeatMessage(&payload, sizeof(payload), metadata); + output->pushMessageLocking(heartbeatMessage); + } + } + + const instanceId_t _instanceId; + const size_t _toleranceMs; + + const healthChangeCallback_t _healthChangeCallback; + + const system::channels::MessageTypeRegistry::messageType_t _messageType; + + std::unordered_map _inputs; + std::unordered_map _outputs; + std::unordered_map _health; + std::unordered_map _lastSeen; +}; +} // namespace serving::modules::heartbeat diff --git a/platform/include/modules/requestManager/module.hpp b/platform/include/modules/requestManager/module.hpp new file mode 100644 index 0000000..f7c824b --- /dev/null +++ b/platform/include/modules/requestManager/module.hpp @@ -0,0 +1,170 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include + +namespace serving::modules::roles::requestManager +{ + +class Module final : public serving::modules::Module +{ + public: + + using input_t = std::shared_ptr; + using output_t = std::shared_ptr; + using message_t = serving::system::channels::Message; + using metadata_t = serving::system::channels::Message::metadata_t; + using messageId_t = serving::system::channels::Message::messageId_t; + using messageType_t = serving::system::channels::Message::messageType_t; + using groupId_t = serving::system::channels::Message::groupId_t; + using sequenceId_t = serving::system::channels::Message::sequenceId_t; + + class Prompt final + { + public: + + Prompt(const metadata_t metadata, const uint8_t *data, const size_t size) + : _metadata(metadata), + _input((size > 0 && data != nullptr) ? std::vector(data, data + size) : std::vector{}) + {} + + [[nodiscard]] const metadata_t &getMetadata() const { return _metadata; } + [[nodiscard]] messageId_t getId() const { return _metadata.getId(); } + [[nodiscard]] const std::vector &getInput() const { return _input; } + [[nodiscard]] const std::vector &getResponse() const { return _response; } + [[nodiscard]] bool hasResponse() const { return _hasResponse.load(); } + [[nodiscard]] message_t getInputMessage() const { return message_t(_input.data(), _input.size(), _metadata); } + + private: + + friend class Module; + + void setResponse(const message_t &message) + { + if (message.getMetadata().getId() != getId()) { HICR_THROW_LOGIC("Response metadata does not match prompt metadata."); } + _response.assign(message.getData(), message.getData() + message.getSize()); + _hasResponse.store(true); + } + metadata_t _metadata; + std::vector _input; + std::vector _response; + std::atomic _hasResponse = false; + }; + + Module() + : serving::modules::Module() + {} + + __INLINE__ void addMessageType(const messageType_t messageType) + { + if (_hasMessageType) [[unlikely]] + HICR_THROW_LOGIC("Request manager message type is already configured."); + _messageType = messageType; + _hasMessageType = true; + } + + __INLINE__ void setPromptOutput(const output_t &output) + { + if (output == nullptr) [[unlikely]] + HICR_THROW_LOGIC("Request manager prompt output cannot be null."); + _promptOutput = output; + } + + __INLINE__ void setResultInput(const input_t &input) + { + if (input == nullptr) [[unlikely]] + HICR_THROW_LOGIC("Request manager result input cannot be null."); + _resultInput = input; + } + + [[nodiscard]] __INLINE__ std::shared_ptr submit(const uint8_t *data, const size_t size) + { + if (_hasMessageType == false) HICR_THROW_LOGIC("Request manager has no message type configured."); + if (_promptOutput == nullptr) HICR_THROW_LOGIC("Request manager prompt output has not been set."); + if (data == nullptr && size > 0) HICR_THROW_LOGIC("Cannot submit a null prompt payload with non-zero size."); + + metadata_t metadata; + metadata.type = _messageType; + metadata.groupId = _defaultSessionId; + metadata.sequenceId = _nextPromptId.fetch_add(1); + + auto prompt = std::make_shared(metadata, data, size); + { + std::lock_guard lock(_activePromptsMutex); + const auto promptKey = metadata.getId(); + if (_activePrompts.contains(promptKey)) HICR_THROW_LOGIC("Prompt %lu is already active.", promptKey); + _activePrompts[promptKey] = prompt; + } + + const auto message = message_t(prompt->getInput().data(), prompt->getInput().size(), metadata); + _promptOutput->pushMessageLocking(message); + return prompt; + } + + [[nodiscard]] __INLINE__ std::vector buildSubscriptions() + { + if (_hasMessageType == false) HICR_THROW_LOGIC("Request manager has no message type configured."); + if (_resultInput == nullptr) HICR_THROW_LOGIC("Request manager result input has not been set."); + std::vector subscriptions; + subscriptions.emplace_back(_messageType, _resultInput, [this](const input_t input, const message_t &message) { this->resultMessageHandler(input, message); }); + return subscriptions; + } + + [[nodiscard]] __INLINE__ std::vector> buildUnsubscriptions() const + { + std::vector> unsubscriptions; + if (_hasMessageType && _resultInput != nullptr) { unsubscriptions.push_back({_messageType, _resultInput}); } + return unsubscriptions; + } + + void initialize() override {} + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override {} + + protected: + + void service() override {} + + private: + + __INLINE__ void resultMessageHandler(const input_t, const message_t &message) + { + if (message.getMetadata().type != _messageType) { HICR_THROW_RUNTIME("Request manager received unexpected message type %u.", message.getMetadata().type); } + + std::shared_ptr prompt; + { + std::lock_guard lock(_activePromptsMutex); + const auto promptKey = message.getMetadata().getId(); + if (_activePrompts.contains(promptKey) == false) { HICR_THROW_RUNTIME("Request manager received result for unknown prompt %lu.", promptKey); } + prompt = _activePrompts.at(promptKey); + _activePrompts.erase(promptKey); + } + prompt->setResponse(message); + } + + static constexpr groupId_t _defaultSessionId = 0; + output_t _promptOutput = nullptr; + input_t _resultInput = nullptr; + messageType_t _messageType = 0; + bool _hasMessageType = false; + std::atomic _nextPromptId = 1; + std::mutex _activePromptsMutex; + std::unordered_map> _activePrompts; +}; +} // namespace serving::modules::roles::requestManager diff --git a/platform/include/modules/roles/coordinator/module.hpp b/platform/include/modules/roles/coordinator/module.hpp new file mode 100644 index 0000000..792aae1 --- /dev/null +++ b/platform/include/modules/roles/coordinator/module.hpp @@ -0,0 +1,424 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include +#include +#include +#include + +#include + +#include "../job.hpp" +#include "../jobFactory.hpp" + +namespace serving::modules::roles::coordinator +{ + +class Module final : public modules::Module +{ + public: + + using input_t = std::shared_ptr; + using output_t = std::shared_ptr; + using message_t = serving::system::channels::Message; + using metadata_t = serving::system::channels::Message::metadata_t; + using messageId_t = serving::system::channels::Message::messageId_t; + using messageType_t = serving::system::channels::Message::messageType_t; + using instanceId_t = HiCR::Instance::instanceId_t; + using jobName_t = serving::modules::roles::JobFactory::jobName_t; + using edgeName_t = std::string; + + struct JobOutput + { + edgeName_t name; + std::shared_ptr data; + }; + + using completionCallback_t = std::function &outputs)>; + using drainCallback_t = std::function; + + // partitionName is recorded in RuntimePlan snapshots. + Module(const size_t intervalMs, const std::string &partitionName = "") + : modules::Module(intervalMs), + _partitionName(partitionName) + {} + + using inputChannelMap_t = std::unordered_map; + using outputChannelMap_t = std::unordered_map; + + __INLINE__ void addReplica(const instanceId_t replicaId, const inputChannelMap_t &inputChannels, const outputChannelMap_t &outputChannels) + { + std::unique_lock lock(_replicasMutex); + if (_replicas.contains(replicaId)) HICR_THROW_LOGIC("Replica %lu already registered.", replicaId); + _replicas[replicaId] = ReplicaEntry{ + .inputChannels = inputChannels, + .outputChannels = outputChannels, + .status = system::RuntimePlan::ReplicaStatus::Active, + }; + _readyReplicas.push(replicaId); + _planVersion.fetch_add(1, std::memory_order_relaxed); + } + + // Hot-add a replica to a running coordinator. Safe to call concurrently + // with job dispatch and completion handling. Returns the new subscriptions + // to register with channelDispatcher before the replica can receive work. + [[nodiscard]] __INLINE__ std::vector addReplicaLive(const instanceId_t replicaId, + const inputChannelMap_t &inputChannels, + const outputChannelMap_t &outputChannels) + { + std::vector subs; + { + std::unique_lock lock(_replicasMutex); + if (_replicas.contains(replicaId)) HICR_THROW_LOGIC("addReplicaLive: replica %lu already registered.", replicaId); + _replicas[replicaId] = ReplicaEntry{ + .inputChannels = inputChannels, + .outputChannels = outputChannels, + .status = system::RuntimePlan::ReplicaStatus::Active, + }; + _planVersion.fetch_add(1, std::memory_order_relaxed); + for (const auto &[outputName, readChannel] : outputChannels) + for (const auto messageType : _messageTypes) + subs.emplace_back( + messageType, readChannel, [this, replicaId, outputName](const input_t, const message_t &message) { this->replicaResponseHandler(replicaId, outputName, message); }); + } + { + std::lock_guard readyLock(_readyReplicasMutex); + _readyReplicas.push(replicaId); + } + return subs; + } + + // Begin draining replica replicaId: no new jobs will be dispatched to it. + // If the replica is idle when drained (no in-flight job), the drainCallback + // fires synchronously before this method returns. If the replica has a job + // in flight, the drainCallback fires from replicaResponseHandler when that + // job completes. removeReplica is safe only after the callback fires. + __INLINE__ void drainReplica(const instanceId_t replicaId) + { + { + std::unique_lock lock(_replicasMutex); + if (_replicas.contains(replicaId) == false) HICR_THROW_LOGIC("drainReplica: unknown replica %lu.", replicaId); + if (_replicas.at(replicaId).status != system::RuntimePlan::ReplicaStatus::Active) HICR_THROW_LOGIC("drainReplica: replica %lu is not Active.", replicaId); + _replicas.at(replicaId).status = system::RuntimePlan::ReplicaStatus::Draining; + _planVersion.fetch_add(1, std::memory_order_relaxed); + } + // Check idle (no in-flight job) outside _replicasMutex to avoid ordering + // inversion with _replicaJobsMutex. A narrow race exists: if dispatch is + // mid-assignment (status checked as Active, not yet in _replicaJobs), the + // callback may fire before the last job completes. Callers who drain only + // after waiting for all pending completions do not encounter this race. + bool idle = false; + { + std::lock_guard jobsLock(_replicaJobsMutex); + idle = !_replicaJobs.contains(replicaId); + } + if (idle) + { + bool shouldNotify = false; + { + std::unique_lock lock(_replicasMutex); + auto &entry = _replicas.at(replicaId); + if (!entry.idleNotified) + { + entry.idleNotified = true; + shouldNotify = true; + } + } + if (shouldNotify && _drainCallback) _drainCallback(replicaId); + } + } + + // Remove a drained replica. The replica must be Draining (not Active). + // After this call the replica's channels may be torn down safely. + __INLINE__ void removeReplica(const instanceId_t replicaId) + { + std::unique_lock lock(_replicasMutex); + if (_replicas.contains(replicaId) == false) HICR_THROW_LOGIC("removeReplica: unknown replica %lu.", replicaId); + if (_replicas.at(replicaId).status != system::RuntimePlan::ReplicaStatus::Draining) HICR_THROW_LOGIC("removeReplica: replica %lu must be Draining before removal.", replicaId); + _replicas.at(replicaId).status = system::RuntimePlan::ReplicaStatus::Removed; + _planVersion.fetch_add(1, std::memory_order_relaxed); + } + + // Return an immutable snapshot of the current deployment runtime state. + [[nodiscard]] __INLINE__ system::RuntimePlan getRuntimePlan() const + { + system::RuntimePlan plan; + plan.version = _planVersion.load(std::memory_order_relaxed); + + system::RuntimePlan::PartitionState part; + part.name = _partitionName; + { + std::shared_lock lock(_replicasMutex); + part.replicas.reserve(_replicas.size()); + for (const auto &[id, entry] : _replicas) part.replicas.push_back({id, entry.status}); + } + plan.partitions.push_back(std::move(part)); + return plan; + } + + __INLINE__ void addMessageType(const messageType_t messageType) { _messageTypes.insert(messageType); } + + __INLINE__ void setCompletionCallback(completionCallback_t callback) { _completionCallback = callback; } + + // Called exactly once per replica when it transitions Draining → idle (no + // more in-flight jobs and no new jobs will be assigned). Safe to call + // removeReplica() and tear down channels from inside the callback. + __INLINE__ void setDrainCallback(drainCallback_t callback) { _drainCallback = std::move(callback); } + + __INLINE__ void submitJob(const std::shared_ptr &job) + { + const auto jobId = job->getMetadata().getId(); + { + std::lock_guard lock(_jobsMutex); + if (_jobs.contains(jobId)) { return; } + _jobs[jobId] = job; + } + { + std::lock_guard lock(_pendingJobsMutex); + _pendingJobs.push(job); + } + } + + [[nodiscard]] __INLINE__ std::vector buildSubscriptions() + { + std::vector subscriptions; + size_t subscriptionCount = 0; + for (const auto &[_, replica] : _replicas) subscriptionCount += replica.outputChannels.size() * _messageTypes.size(); + subscriptions.reserve(subscriptionCount); + for (const auto &[replicaId, replica] : _replicas) + { + for (const auto &[outputName, readChannel] : replica.outputChannels) + { + for (const auto messageType : _messageTypes) + subscriptions.emplace_back( + messageType, readChannel, [this, replicaId, outputName](const input_t, const message_t &message) { this->replicaResponseHandler(replicaId, outputName, message); }); + } + } + return subscriptions; + } + + [[nodiscard]] __INLINE__ std::vector> buildUnsubscriptions() const + { + std::vector> unsubscriptions; + size_t unsubscriptionCount = 0; + for (const auto &[_, replica] : _replicas) unsubscriptionCount += replica.outputChannels.size() * _messageTypes.size(); + unsubscriptions.reserve(unsubscriptionCount); + for (const auto &[_, replica] : _replicas) + { + for (const auto &[_, readChannel] : replica.outputChannels) + { + for (const auto &messageType : _messageTypes) { unsubscriptions.push_back({messageType, readChannel}); } + } + } + return unsubscriptions; + } + + void initialize() override + { + if (_messageTypes.empty()) HICR_THROW_LOGIC("Coordinator has no message types configured."); + if (_replicas.empty()) HICR_THROW_LOGIC("Coordinator has no replicas."); + } + + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override {} + + protected: + + void service() override { dispatchReadyJobs(); } + + private: + + struct ReplicaEntry + { + inputChannelMap_t inputChannels; + outputChannelMap_t outputChannels; + system::RuntimePlan::ReplicaStatus status{system::RuntimePlan::ReplicaStatus::Active}; + bool idleNotified{false}; + }; + + __INLINE__ void dispatchReadyJobs() + { + std::shared_ptr job = nullptr; + { + std::lock_guard lock(_pendingJobsMutex); + if (_pendingJobs.empty()) return; + job = _pendingJobs.front(); + _pendingJobs.pop(); + } + + if (job->isReady() == false) + { + std::lock_guard lock(_pendingJobsMutex); + _pendingJobs.push(job); + return; + } + + // Find the next Active replica, discarding any Draining/Removed entries + // that were enqueued before a status transition. Collect drain + // notifications so the callback fires outside all locks. + instanceId_t replicaId; + bool found = false; + std::vector drainNotifications; + { + std::lock_guard lock(_readyReplicasMutex); + while (!_readyReplicas.empty()) + { + const auto candidate = _readyReplicas.front(); + _readyReplicas.pop(); + std::unique_lock statusLock(_replicasMutex); + if (!_replicas.contains(candidate)) continue; + auto &entry = _replicas.at(candidate); + if (entry.status == system::RuntimePlan::ReplicaStatus::Active) + { + replicaId = candidate; + found = true; + break; + } + if (entry.status == system::RuntimePlan::ReplicaStatus::Draining && !entry.idleNotified) + { + entry.idleNotified = true; + drainNotifications.push_back(candidate); + } + } + } + for (const auto id : drainNotifications) + if (_drainCallback) _drainCallback(id); + if (!found) + { + std::lock_guard pendingLock(_pendingJobsMutex); + _pendingJobs.push(job); + return; + } + + { + std::lock_guard lock(_replicaJobsMutex); + _replicaJobs[replicaId] = job->getMetadata().getId(); + } + std::shared_lock replicasLock(_replicasMutex); + const auto &replica = _replicas.at(replicaId); + for (auto &[inputName, dependency] : job->getInputDependencies()) + { + if (replica.inputChannels.contains(inputName) == false) HICR_THROW_RUNTIME("Replica %lu has no input channel for dependency '%s'.", replicaId, inputName.c_str()); + const auto message = message_t(dependency.getDataPointer(), dependency.getDataSize(), job->getMetadata()); + replica.inputChannels.at(inputName)->pushMessageLocking(message); + dependency.freeDataSlot(); + } + } + + __INLINE__ void replicaResponseHandler(const instanceId_t replicaId, const edgeName_t &outputName, const message_t &message) + { + const auto jobId = message.getMetadata().getId(); + + { + std::lock_guard lock(_replicaJobsMutex); + if (!_replicaJobs.contains(replicaId)) { HICR_THROW_RUNTIME("Received response from replica %lu with no assigned job.", replicaId); } + if (_replicaJobs.at(replicaId) != jobId) + { + HICR_THROW_RUNTIME("Received response from replica %lu for job %lu, but expected job %lu.", replicaId, jobId, _replicaJobs.at(replicaId)); + } + } + + std::shared_ptr job = nullptr; + { + std::lock_guard lock(_jobsMutex); + job = _jobs.at(jobId); + } + + auto &dependency = job->getOutputDependency(outputName); + if (dependency.isSatisfied()) HICR_THROW_RUNTIME("Output dependency '%s' for job %lu is already satisfied.", outputName.c_str(), jobId); + dependency.storeData(message.getData(), message.getSize()); + dependency.setSatisfied(true); + + if (job->isFinished() == false) return; + + std::vector outputs; + outputs.reserve(job->getOutputDependencies().size()); + for (auto &[_, outputDependency] : job->getOutputDependencies()) + { + outputs.push_back(JobOutput{ + .name = outputDependency.getName(), + .data = outputDependency.getData(), + }); + } + + { + std::lock_guard lock(_jobsMutex); + _jobs.erase(jobId); + } + + { + std::lock_guard lock(_replicaJobsMutex); + _replicaJobs.erase(replicaId); + } + + // Re-queue only if Active. If Draining and not yet notified, fire the + // drain callback (exactly once, guarded by idleNotified). Both checks + // happen under _replicasMutex; the callback and enqueue happen + // outside all locks to avoid inversion and allow re-entrant calls. + bool shouldRequeue = false; + bool drainIdle = false; + { + std::unique_lock statusLock(_replicasMutex); + if (_replicas.contains(replicaId)) + { + auto &entry = _replicas.at(replicaId); + if (entry.status == system::RuntimePlan::ReplicaStatus::Active) { shouldRequeue = true; } + else if (entry.status == system::RuntimePlan::ReplicaStatus::Draining && !entry.idleNotified) + { + entry.idleNotified = true; + drainIdle = true; + } + } + } + if (shouldRequeue) + { + std::lock_guard lock(_readyReplicasMutex); + _readyReplicas.push(replicaId); + } + if (drainIdle && _drainCallback) _drainCallback(replicaId); + if (_completionCallback != nullptr) { _completionCallback(job->getMetadata(), outputs); } + + for (auto &[_, outputDependency] : job->getOutputDependencies()) { outputDependency.freeDataSlot(); } + } + + std::string _partitionName; + std::atomic _planVersion{0}; + mutable std::shared_mutex _replicasMutex; + + std::unordered_map _replicas; + + std::unordered_set _messageTypes; + + std::unordered_map> _jobs; + std::mutex _jobsMutex; + + std::queue> _pendingJobs; + std::mutex _pendingJobsMutex; + + std::queue _readyReplicas; + std::mutex _readyReplicasMutex; + + std::unordered_map _replicaJobs; + std::mutex _replicaJobsMutex; + + completionCallback_t _completionCallback; + drainCallback_t _drainCallback; +}; +} // namespace serving::modules::roles::coordinator diff --git a/platform/include/modules/roles/dependency.hpp b/platform/include/modules/roles/dependency.hpp new file mode 100644 index 0000000..d633d8d --- /dev/null +++ b/platform/include/modules/roles/dependency.hpp @@ -0,0 +1,69 @@ +#pragma once + +#include + +#include +#include + +#include + +namespace serving::modules::roles +{ + +class Dependency +{ + public: + + Dependency() = delete; + ~Dependency() = default; + + Dependency(const std::string &name, const configuration::Edge &edgeInfo) + : _name(name), + _edgeInfo(edgeInfo) + {} + + [[nodiscard]] __INLINE__ bool isSatisfied() const { return _isSatisfied; } + __INLINE__ void setSatisfied(const bool satisfied = true) { _isSatisfied = satisfied; } + + [[nodiscard]] __INLINE__ const std::shared_ptr getData() const { return _data; } + [[nodiscard]] __INLINE__ bool hasData() const { return _data != nullptr; } + [[nodiscard]] __INLINE__ size_t getDataSize() const { return hasData() ? _data->getSize() : 0; } + [[nodiscard]] __INLINE__ const uint8_t *getDataPointer() const { return hasData() ? static_cast(_data->getPointer()) : nullptr; } + + __INLINE__ void freeDataSlot() + { + if (_data == nullptr) return; + _edgeInfo.getPayloadMemoryManager()->freeLocalMemorySlot(_data); + _data = nullptr; + } + + [[nodiscard]] __INLINE__ const std::string &getName() const { return _name; } + + __INLINE__ void storeData(const uint8_t *srcPtr, const size_t size) + { + if (srcPtr == nullptr && size > 0) HICR_THROW_LOGIC("Dependency '%s' cannot store null data with non-zero size.", _name.c_str()); + + freeDataSlot(); + if (size == 0) return; + + auto edgeMemoryManager = _edgeInfo.getPayloadMemoryManager(); + auto edgeMemorySpace = _edgeInfo.getPayloadMemorySpace(); + auto edgeCommunicationManager = _edgeInfo.getPayloadCommunicationManager(); + + const auto srcSlot = edgeMemoryManager->registerLocalMemorySlot(edgeMemorySpace, (void *)srcPtr, size); + auto dstSlot = edgeMemoryManager->allocateLocalMemorySlot(edgeMemorySpace, size); + edgeCommunicationManager->memcpy(dstSlot, 0, srcSlot, 0, size); + edgeCommunicationManager->fence(dstSlot, 0, 1); + edgeMemoryManager->deregisterLocalMemorySlot(srcSlot); + + _data = dstSlot; + } + + private: + + const std::string _name; + const configuration::Edge _edgeInfo; + std::shared_ptr _data = nullptr; + bool _isSatisfied = false; +}; +} // namespace serving::modules::roles diff --git a/platform/include/modules/roles/job.hpp b/platform/include/modules/roles/job.hpp new file mode 100644 index 0000000..6faaf7c --- /dev/null +++ b/platform/include/modules/roles/job.hpp @@ -0,0 +1,104 @@ +#pragma once + +#include +#include +#include + +#include + +#include +#include + +#include "./dependency.hpp" + +namespace serving::modules::roles +{ + +using dependency_t = std::string; + +class Job +{ + public: + + Job(const std::string &name, + const system::channels::Message::metadata_t &messageMetadata, + std::unordered_map &inputDependencies, + std::unordered_map &outputDependencies) + : _name(name), + _messageMetadata(messageMetadata), + _inputDependencies(inputDependencies), + _outputDependencies(outputDependencies) + {} + + ~Job() = default; + + [[nodiscard]] __INLINE__ Dependency &getInputDependency(const dependency_t &dependencyName) { return _inputDependencies.at(dependencyName); } + [[nodiscard]] __INLINE__ Dependency &getOutputDependency(const dependency_t &dependencyName) { return _outputDependencies.at(dependencyName); } + [[nodiscard]] __INLINE__ std::unordered_map &getInputDependencies() { return _inputDependencies; } + [[nodiscard]] __INLINE__ std::unordered_map &getOutputDependencies() { return _outputDependencies; } + [[nodiscard]] __INLINE__ const std::unordered_map &getInputDependencies() const { return _inputDependencies; } + [[nodiscard]] __INLINE__ const std::unordered_map &getOutputDependencies() const { return _outputDependencies; } + [[nodiscard]] __INLINE__ const system::channels::Message::metadata_t &getMetadata() const { return _messageMetadata; } + + [[nodiscard]] __INLINE__ bool isReady() const + { + for (const auto &[_, dependency] : _inputDependencies) + { + if (dependency.isSatisfied() == false) return false; + } + return true; + } + + [[nodiscard]] __INLINE__ bool isFinished() const + { + for (const auto &[_, dependency] : _outputDependencies) + { + if (dependency.isSatisfied() == false) return false; + } + return true; + } + + [[nodiscard]] __INLINE__ nlohmann::json serialize() const + { + nlohmann::json js; + js["Name"] = _name; + js["Ready"] = isReady(); + js["Finished"] = isFinished(); + + js["Message metadata"] = nlohmann::json::object(); + js["Message metadata"]["Type"] = _messageMetadata.type; + js["Message metadata"]["Group"] = _messageMetadata.groupId; + js["Message metadata"]["Sequence"] = _messageMetadata.sequenceId; + js["Message metadata"]["ID"] = _messageMetadata.getId(); + + js["Input Dependencies"] = nlohmann::json::array(); + for (const auto &[dependencyName, dependency] : _inputDependencies) + { + nlohmann::json depJs; + depJs["Name"] = dependency.getName(); + depJs["Dependency Name"] = dependencyName; + depJs["Satisfied"] = dependency.isSatisfied(); + depJs["Data Size"] = (dependency.getData() == nullptr) ? 0 : dependency.getData()->getSize(); + js["Input Dependencies"].push_back(depJs); + } + js["Output Dependencies"] = nlohmann::json::array(); + for (const auto &[dependencyName, dependency] : _outputDependencies) + { + nlohmann::json depJs; + depJs["Name"] = dependency.getName(); + depJs["Dependency Name"] = dependencyName; + depJs["Satisfied"] = dependency.isSatisfied(); + depJs["Data Size"] = (dependency.getData() == nullptr) ? 0 : dependency.getData()->getSize(); + js["Output Dependencies"].push_back(depJs); + } + return js; + } + + protected: + + const std::string _name; + const system::channels::Message::metadata_t _messageMetadata; + std::unordered_map _inputDependencies; + std::unordered_map _outputDependencies; +}; +} // namespace serving::modules::roles diff --git a/platform/include/modules/roles/jobFactory.hpp b/platform/include/modules/roles/jobFactory.hpp new file mode 100644 index 0000000..48b6e1b --- /dev/null +++ b/platform/include/modules/roles/jobFactory.hpp @@ -0,0 +1,82 @@ +#pragma once + +#include +#include + +#include +#include + +#include "./dependency.hpp" +#include "./job.hpp" + +namespace serving::modules::roles +{ + +class JobFactory +{ + public: + + using jobName_t = std::string; + using dependency_t = std::string; + + JobFactory(const size_t partitionIndex, const configuration::Deployment &deployment) + : _jobInputDependencies(buildTaskInputDependencies(*deployment.getPartitions().at(partitionIndex))), + _jobOutputDependencies(buildTaskOutputDependencies(*deployment.getPartitions().at(partitionIndex))), + _edgeInfos(buildEdgeInfos(deployment)) + {} + + [[nodiscard]] __INLINE__ Job createJob(const std::string &jobName, const system::channels::Message::metadata_t &metadata) + { + auto inputDependencies = std::unordered_map(); + for (const auto &[job, edges] : _jobInputDependencies) + { + if (job != jobName) { continue; } + for (auto &edge : edges) + { + const auto &edgeInfo = _edgeInfos.at(edge); + inputDependencies.try_emplace(edge, edge, *edgeInfo); + } + } + + auto outputDependencies = std::unordered_map(); + for (const auto &[job, edges] : _jobOutputDependencies) + { + if (job != jobName) { continue; } + for (auto &edge : edges) + { + const auto &edgeInfo = _edgeInfos.at(edge); + outputDependencies.try_emplace(edge, edge, *edgeInfo); + } + } + + return Job(jobName, metadata, inputDependencies, outputDependencies); + } + + private: + + std::unordered_map> buildEdgeInfos(const configuration::Deployment &deployment) + { + auto edgeInfos = std::unordered_map>(); + for (const auto &edge : deployment.getEdges()) { edgeInfos[edge->getName()] = edge; } + return edgeInfos; + } + + std::unordered_map> buildTaskInputDependencies(const configuration::Partition &partition) + { + auto taskInfos = std::unordered_map>(); + for (const auto &task : partition.getTasks()) { taskInfos[task->getFunctionName()] = task->getInputs(); } + return taskInfos; + } + + std::unordered_map> buildTaskOutputDependencies(const configuration::Partition &partition) + { + auto taskInfos = std::unordered_map>(); + for (const auto &task : partition.getTasks()) { taskInfos[task->getFunctionName()] = task->getOutputs(); } + return taskInfos; + } + + const std::unordered_map> _jobInputDependencies; + const std::unordered_map> _jobOutputDependencies; + const std::unordered_map> _edgeInfos; +}; +} // namespace serving::modules::roles diff --git a/platform/include/modules/roles/replica/module.hpp b/platform/include/modules/roles/replica/module.hpp new file mode 100644 index 0000000..ce00c90 --- /dev/null +++ b/platform/include/modules/roles/replica/module.hpp @@ -0,0 +1,241 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include +#include +#include +#include + +#include + +namespace serving::modules::roles::replica +{ + +class Module final : public serving::modules::Module +{ + public: + + using instanceId_t = HiCR::Instance::instanceId_t; + using input_t = std::shared_ptr; + using output_t = std::shared_ptr; + using message_t = serving::system::channels::Message; + using metadata_t = serving::system::channels::Message::metadata_t; + using messageId_t = serving::system::channels::Message::messageId_t; + using messageType_t = serving::system::channels::Message::messageType_t; + using edgeName_t = std::string; + using inputMap_t = std::unordered_map; + using outputMap_t = std::unordered_map; + using processFc_t = std::function; + using memorySlot_t = std::shared_ptr; + + Module(const std::string &taskName, const std::vector &inputNames, const std::vector &outputNames, const processFc_t &processFc) + : serving::modules::Module(), + _taskName(taskName), + _inputNames(inputNames.begin(), inputNames.end()), + _outputNames(outputNames.begin(), outputNames.end()), + _isCoordinatorSet(false), + _processFc(processFc) + {} + + ~Module() override = default; + + __INLINE__ void setCoordinator(const instanceId_t coordinatorId, const outputMap_t &outputChannels, const inputMap_t &inputChannels) + { + if (_isCoordinatorSet) { HICR_THROW_LOGIC("Coordinator %lu already registered.", _coordinatorId); } + _coordinatorId = coordinatorId; + _outputChannels = outputChannels; + _inputChannels = inputChannels; + _isCoordinatorSet = true; + } + + __INLINE__ void addMessageType(const messageType_t messageType) { _messageTypes.insert(messageType); } + + [[nodiscard]] __INLINE__ std::vector buildSubscriptions() + { + std::vector out; + out.reserve(_messageTypes.size() * _inputChannels.size()); + for (const auto &[inputName, inputChannel] : _inputChannels) + { + for (const auto messageType : _messageTypes) + out.emplace_back(messageType, inputChannel, [this, inputName](const input_t, const message_t &message) { this->coordinatorMessageHandler(inputName, message); }); + } + return out; + } + + [[nodiscard]] __INLINE__ std::vector> buildUnsubscriptions() const + { + std::vector> out; + out.reserve(_messageTypes.size() * _inputChannels.size()); + for (const auto &[_, inputChannel] : _inputChannels) + { + for (const auto messageType : _messageTypes) out.push_back({messageType, inputChannel}); + } + return out; + } + + void initialize() override + { + if (_isCoordinatorSet == false) HICR_THROW_LOGIC("Coordinator has not been set"); + if (_messageTypes.empty()) HICR_THROW_LOGIC("Replica has no message types configured."); + for (const auto &inputName : _inputNames) + if (_inputChannels.contains(inputName) == false) HICR_THROW_LOGIC("Replica has no input channel for dependency '%s'.", inputName.c_str()); + for (const auto &outputName : _outputNames) + if (_outputChannels.contains(outputName) == false) HICR_THROW_LOGIC("Replica has no output channel for dependency '%s'.", outputName.c_str()); + } + + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override {} + + protected: + + void service() override {} + + private: + + struct ActiveJob + { + metadata_t metadata; + std::unordered_map inputs; + }; + + __INLINE__ void coordinatorMessageHandler(const edgeName_t &inputName, const message_t &message) + { + if (_inputNames.contains(inputName) == false) HICR_THROW_RUNTIME("Replica received undeclared input dependency '%s'.", inputName.c_str()); + if (message.getData() == nullptr && message.getSize() > 0) HICR_THROW_RUNTIME("Replica received null input data for dependency '%s' with non-zero size.", inputName.c_str()); + + // Copy data before acquiring the lock to avoid serializing memcpy/fence under contention. + const auto copiedData = copyMessageData(inputName, message); + const auto jobId = message.getMetadata().getId(); + ActiveJob activeJob; + { + std::lock_guard lock(_activeJobsMutex); + if (_activeJobs.contains(jobId) == false) _activeJobs[jobId] = ActiveJob{.metadata = message.getMetadata()}; + auto &job = _activeJobs.at(jobId); + if (job.inputs.contains(inputName)) + { + if (copiedData != nullptr) _inputChannels.at(inputName)->getConfig().payloadMemoryManager->freeLocalMemorySlot(copiedData); + HICR_THROW_RUNTIME("Replica received duplicate input dependency '%s' for job %lu.", inputName.c_str(), jobId); + } + job.inputs[inputName] = copiedData; + if (isReady(job) == false) return; + + activeJob = std::move(job); + _activeJobs.erase(jobId); + } + + auto outputConfigs = buildOutputConfigs(); + serving::modules::roles::TaskContext context(_taskName, activeJob.metadata, activeJob.inputs, outputConfigs); + + // Scope guard: if _processFc or any output validation throws, free owned + // outputs and input slots before unwinding to prevent slot leaks. + struct CleanupGuard + { + serving::modules::roles::TaskContext &context; + ActiveJob &activeJob; + std::function freeInputSlotsFn; + bool dismissed = false; + ~CleanupGuard() + { + if (!dismissed) + { + context.freeOwnedOutputs(); + freeInputSlotsFn(activeJob); + } + } + } guard{context, activeJob, [this](ActiveJob &j) { freeInputSlots(j); }}; + + _processFc(context); + + std::unordered_set sentOutputs; + for (const auto &output : context.getOutputs()) + { + if (_outputNames.contains(output.name) == false) HICR_THROW_RUNTIME("Task '%s' set undeclared output dependency '%s'.", _taskName.c_str(), output.name.c_str()); + if (sentOutputs.contains(output.name)) HICR_THROW_RUNTIME("Task '%s' set output dependency '%s' more than once.", _taskName.c_str(), output.name.c_str()); + sentOutputs.insert(output.name); + } + + for (const auto &outputName : _outputNames) + if (sentOutputs.contains(outputName) == false) HICR_THROW_RUNTIME("Task '%s' did not set output dependency '%s'.", _taskName.c_str(), outputName.c_str()); + + // All validation passed. Dismiss the guard so outputs stay alive for the + // coordinator completion callback, which is responsible for freeing them. + guard.dismissed = true; + + for (const auto &output : context.getOutputs()) + { + const auto response = message_t( + output.data == nullptr ? nullptr : static_cast(output.data->getPointer()), output.data == nullptr ? 0 : output.data->getSize(), activeJob.metadata); + _outputChannels.at(output.name)->pushMessageLocking(response); + } + + // Input slots are no longer needed once outputs have been dispatched. + freeInputSlots(activeJob); + } + + [[nodiscard]] __INLINE__ memorySlot_t copyMessageData(const edgeName_t &inputName, const message_t &message) const + { + if (message.getSize() == 0) return nullptr; + + const auto &cfg = _inputChannels.at(inputName)->getConfig(); + const auto srcSlot = cfg.payloadMemoryManager->registerLocalMemorySlot(cfg.payloadMemorySpace, const_cast(message.getData()), message.getSize()); + auto dstSlot = cfg.payloadMemoryManager->allocateLocalMemorySlot(cfg.payloadMemorySpace, message.getSize()); + cfg.payloadCommunicationManager->memcpy(dstSlot, 0, srcSlot, 0, message.getSize()); + cfg.payloadCommunicationManager->fence(dstSlot, 0, 1); + cfg.payloadMemoryManager->deregisterLocalMemorySlot(srcSlot); + return dstSlot; + } + + [[nodiscard]] __INLINE__ serving::modules::roles::TaskContext::outputConfigMap_t buildOutputConfigs() const + { + serving::modules::roles::TaskContext::outputConfigMap_t configs; + for (const auto &[name, output] : _outputChannels) configs.emplace(name, output->getConfig()); + return configs; + } + + __INLINE__ void freeInputSlots(ActiveJob &job) const + { + for (auto &[name, slot] : job.inputs) + { + if (slot == nullptr) continue; + _inputChannels.at(name)->getConfig().payloadMemoryManager->freeLocalMemorySlot(slot); + slot = nullptr; + } + } + + [[nodiscard]] __INLINE__ bool isReady(const ActiveJob &job) const + { + for (const auto &inputName : _inputNames) + if (job.inputs.contains(inputName) == false) return false; + return true; + } + + const std::string _taskName; + const std::unordered_set _inputNames; + const std::unordered_set _outputNames; + + bool _isCoordinatorSet; + processFc_t _processFc; + + HiCR::Instance::instanceId_t _coordinatorId; + inputMap_t _inputChannels; + outputMap_t _outputChannels; + std::unordered_set _messageTypes; + std::unordered_map _activeJobs; + std::mutex _activeJobsMutex; +}; +} // namespace serving::modules::roles::replica diff --git a/platform/include/modules/roles/taskContext.hpp b/platform/include/modules/roles/taskContext.hpp new file mode 100644 index 0000000..6617369 --- /dev/null +++ b/platform/include/modules/roles/taskContext.hpp @@ -0,0 +1,106 @@ +#pragma once + +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +namespace serving::modules::roles +{ + +class TaskContext final +{ + public: + + using metadata_t = serving::system::channels::Message::metadata_t; + using memorySlot_t = std::shared_ptr; + using outputConfigMap_t = std::unordered_map; + + struct Output + { + std::string name; + memorySlot_t data; + bool owned = false; + }; + + TaskContext(const std::string &name, const metadata_t &metadata, const std::unordered_map &inputs, const outputConfigMap_t &outputConfigs) + : _name(name), + _metadata(metadata), + _inputs(inputs), + _outputConfigs(outputConfigs) + {} + + [[nodiscard]] __INLINE__ const std::string &getName() const { return _name; } + [[nodiscard]] __INLINE__ const metadata_t &getMetadata() const { return _metadata; } + + [[nodiscard]] __INLINE__ memorySlot_t getInput(const std::string &name) const + { + if (_inputs.contains(name) == false) HICR_THROW_RUNTIME("Task '%s' has no input '%s'.", _name.c_str(), name.c_str()); + const auto &input = _inputs.at(name); + if (input == nullptr) return _emptySlot; + return input; + } + + __INLINE__ void setOutput(const std::string &name, const memorySlot_t &data) + { + if (_outputIndex.contains(name)) HICR_THROW_RUNTIME("Task '%s' set output '%s' more than once.", _name.c_str(), name.c_str()); + if (_outputConfigs.contains(name) == false) HICR_THROW_RUNTIME("Task '%s' set undeclared output '%s'.", _name.c_str(), name.c_str()); + if (data == nullptr) HICR_THROW_RUNTIME("Task '%s' cannot set output '%s' from a null memory slot.", _name.c_str(), name.c_str()); + + _outputIndex[name] = _outputs.size(); + _outputs.push_back(Output{.name = name, .data = data, .owned = false}); + } + + __INLINE__ void setOutput(const std::string &name, const void *data, const size_t size) + { + if (_outputIndex.contains(name)) HICR_THROW_RUNTIME("Task '%s' set output '%s' more than once.", _name.c_str(), name.c_str()); + if (data == nullptr && size > 0) HICR_THROW_LOGIC("Task '%s' cannot set output '%s' from null data with non-zero size.", _name.c_str(), name.c_str()); + if (_outputConfigs.contains(name) == false) HICR_THROW_RUNTIME("Task '%s' set undeclared output '%s'.", _name.c_str(), name.c_str()); + + memorySlot_t outputSlot = nullptr; + if (size > 0) + { + const auto &cfg = _outputConfigs.at(name); + const auto srcSlot = cfg.payloadMemoryManager->registerLocalMemorySlot(cfg.payloadMemorySpace, const_cast(data), size); + outputSlot = cfg.payloadMemoryManager->allocateLocalMemorySlot(cfg.payloadMemorySpace, size); + cfg.payloadCommunicationManager->memcpy(outputSlot, 0, srcSlot, 0, size); + cfg.payloadCommunicationManager->fence(outputSlot, 0, 1); + cfg.payloadMemoryManager->deregisterLocalMemorySlot(srcSlot); + } + + _outputIndex[name] = _outputs.size(); + _outputs.push_back(Output{.name = name, .data = outputSlot, .owned = true}); + } + + [[nodiscard]] __INLINE__ const std::vector &getOutputs() const { return _outputs; } + + __INLINE__ void freeOwnedOutputs() + { + for (auto &output : _outputs) + { + if (output.owned == false || output.data == nullptr) continue; + _outputConfigs.at(output.name).payloadMemoryManager->freeLocalMemorySlot(output.data); + output.data = nullptr; + } + } + + private: + + inline static const memorySlot_t _emptySlot = std::make_shared(nullptr, 0); + const std::string _name; + const metadata_t _metadata; + const std::unordered_map &_inputs; + const outputConfigMap_t &_outputConfigs; + std::vector _outputs; + std::unordered_map _outputIndex; +}; + +} // namespace serving::modules::roles diff --git a/platform/include/modules/router/defaultProcessFunction.hpp b/platform/include/modules/router/defaultProcessFunction.hpp new file mode 100644 index 0000000..00779c3 --- /dev/null +++ b/platform/include/modules/router/defaultProcessFunction.hpp @@ -0,0 +1,96 @@ +#pragma once + +#include +#include +#include +#include + +#include + +#include +#include +#include +#include + +#include "module.hpp" + +namespace serving::modules::router +{ + +class DefaultProcessFunction final +{ + public: + + using messageId_t = serving::system::channels::Message::messageId_t; + using jobName_t = serving::modules::roles::JobFactory::jobName_t; + using edgeName_t = Module::edgeName_t; + using input_t = Module::input_t; + using message_t = Module::message_t; + + DefaultProcessFunction(serving::modules::roles::JobFactory &jobFactory, + serving::modules::roles::coordinator::Module &coordinatorModule, + const serving::configuration::Partition &partition) + : _jobFactory(jobFactory), + _coordinatorModule(coordinatorModule) + { + for (const auto &task : partition.getTasks()) + { + for (const auto &inputEdgeName : task->getInputs()) + { + if (_inputEdgeToJobName.contains(inputEdgeName)) { HICR_THROW_LOGIC("Input edge '%s' is consumed by multiple tasks.", inputEdgeName.c_str()); } + _inputEdgeToJobName[inputEdgeName] = task->getFunctionName(); + } + } + } + + __INLINE__ void process(const input_t &input, const edgeName_t &edgeName, const message_t &message) + { + if (_inputEdgeToJobName.contains(edgeName) == false) { HICR_THROW_LOGIC("No job registered for input edge '%s'.", edgeName.c_str()); } + const auto &jobName = _inputEdgeToJobName.at(edgeName); + const auto jobId = message.getMetadata().getId(); + auto job = getOrCreateJob(jobId, jobName, message); + satisfyInputDependency(*job, edgeName, message); + if (job->isReady()) + { + _coordinatorModule.submitJob(job); + std::lock_guard lock(_jobsMutex); + _jobs.erase(jobId); + } + } + + private: + + __INLINE__ std::shared_ptr getOrCreateJob(const messageId_t jobId, const jobName_t &jobName, const message_t &message) + { + std::lock_guard lock(_jobsMutex); + if (_jobs.contains(jobId)) { return _jobs.at(jobId); } + auto job = std::make_shared(_jobFactory.createJob(jobName, message.getMetadata())); + _jobs[jobId] = job; + return job; + } + + __INLINE__ void satisfyInputDependency(serving::modules::roles::Job &job, const edgeName_t &edge, const message_t &message) + { + auto &dependency = job.getInputDependency(edge); + if (dependency.isSatisfied()) { HICR_THROW_LOGIC("Input dependency %s of job is already satisfied.", edge.c_str()); } + dependency.storeData(message.getData(), message.getSize()); + dependency.setSatisfied(true); + } + + serving::modules::roles::JobFactory &_jobFactory; + serving::modules::roles::coordinator::Module &_coordinatorModule; + std::unordered_map _inputEdgeToJobName; + std::unordered_map> _jobs; + std::mutex _jobsMutex; +}; + +[[nodiscard]] static __INLINE__ Module::processFc_t makeDefaultProcessFunction(serving::modules::roles::JobFactory &jobFactory, + serving::modules::roles::coordinator::Module &coordinatorModule, + const serving::configuration::Partition &partition) +{ + auto processFunction = std::make_shared(jobFactory, coordinatorModule, partition); + return + [processFunction](const Module::input_t &input, const Module::edgeName_t &edgeName, const Module::message_t &message) { processFunction->process(input, edgeName, message); }; +} + +} // namespace serving::modules::router diff --git a/platform/include/modules/router/module.hpp b/platform/include/modules/router/module.hpp new file mode 100644 index 0000000..1218035 --- /dev/null +++ b/platform/include/modules/router/module.hpp @@ -0,0 +1,118 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace serving::modules::router +{ + +class Module final : public serving::modules::Module +{ + public: + + using input_t = std::shared_ptr; + using output_t = std::shared_ptr; + using message_t = serving::system::channels::Message; + using metadata_t = serving::system::channels::Message::metadata_t; + using messageType_t = serving::system::channels::Message::messageType_t; + using instanceId_t = HiCR::Instance::instanceId_t; + using edgeName_t = std::string; + using jobOutput_t = serving::modules::roles::coordinator::Module::JobOutput; + using processFc_t = std::function; + + Module() + : serving::modules::Module() + {} + + __INLINE__ void addMessageType(const messageType_t messageType) { _messageTypes.insert(messageType); } + + __INLINE__ void addInput(const edgeName_t &edgeName, const input_t &input) + { + if (_inputs.contains(edgeName)) HICR_THROW_LOGIC("Graph input '%s' is already registered.", edgeName.c_str()); + _inputs[edgeName] = input; + } + + __INLINE__ void addOutput(const edgeName_t &edgeName, const output_t &output) + { + if (_outputs.contains(edgeName)) HICR_THROW_LOGIC("Graph output '%s' is already registered.", edgeName.c_str()); + _outputs[edgeName] = output; + } + + __INLINE__ void setProcessFunction(processFc_t processFc) { _processFc = processFc; } + + [[nodiscard]] __INLINE__ std::vector buildSubscriptions() + { + std::vector subscriptions; + subscriptions.reserve(_inputs.size() * _messageTypes.size()); + for (const auto &[edgeName, input] : _inputs) + { + for (const auto messageType : _messageTypes) + { + subscriptions.emplace_back(messageType, input, [this, edgeName](const input_t input, const message_t &message) { _processFc(input, edgeName, message); }); + } + } + return subscriptions; + } + + [[nodiscard]] __INLINE__ std::vector> buildUnsubscriptions() const + { + std::vector> unsubscriptions; + unsubscriptions.reserve(_inputs.size() * _messageTypes.size()); + for (const auto &[_, input] : _inputs) + { + for (const auto messageType : _messageTypes) { unsubscriptions.push_back({messageType, input}); } + } + return unsubscriptions; + } + + __INLINE__ void routeOutputs(const metadata_t &metadata, const std::vector &outputs) + { + for (const auto &output : outputs) + { + if (_outputs.contains(output.name) == false) [[unlikely]] { HICR_THROW_LOGIC("No graph output channel registered for edge '%s'.", output.name.c_str()); } + const auto &channel = _outputs.at(output.name); + const auto message = + message_t(output.data == nullptr ? nullptr : static_cast(output.data->getPointer()), output.data == nullptr ? 0 : output.data->getSize(), metadata); + channel->pushMessageLocking(message); + } + } + + void initialize() override {} + void run() override {} + void terminate() override {} + void await() override {} + void finalize() override + { + _inputs.clear(); + _outputs.clear(); + _messageTypes.clear(); + _processFc = nullptr; + } + + protected: + + void service() override {} + + private: + + std::unordered_map _inputs; + std::unordered_map _outputs; + std::unordered_set _messageTypes; + processFc_t _processFc; +}; +} // namespace serving::modules::router diff --git a/platform/include/modules/service/module.hpp b/platform/include/modules/service/module.hpp new file mode 100644 index 0000000..a890482 --- /dev/null +++ b/platform/include/modules/service/module.hpp @@ -0,0 +1,65 @@ +#pragma once +#include +#include +#include +#include + +#include +#include + +#include + +namespace serving::modules::service +{ + +class Module final : public serving::modules::Module +{ + public: + + using serviceFunction_t = taskr::Service::serviceFc_t; + + Module(std::shared_ptr taskr) + : serving::modules::Module(), + _taskr(taskr) + {} + + ~Module() override = default; + + // Ingest external service owned by another module (non-owning pointer). + __INLINE__ void addService(const std::string &name, taskr::Service *service) + { + if (_services.contains(name)) HICR_THROW_LOGIC("[Service] Service '%s' is already registered.", name.c_str()); + _services[name] = service; + } + + void initialize() override + { + // Required for services-only mode (no tasks): do not auto-finish immediately. + _taskr->setFinishOnLastTask(false); + for (const auto &[_, service] : _services) _taskr->addService(service); + _taskr->initialize(); + } + + void run() override { _taskr->run(); } + + void terminate() override { _taskr->setFinishOnLastTask(true); } + + void await() override { _taskr->await(); } + + void finalize() override + { + _taskr->finalize(); + _services.clear(); + } + + protected: + + void service() override {} + + private: + + std::shared_ptr _taskr; + + std::unordered_map _services; +}; +} // namespace serving::modules::service diff --git a/platform/include/modules/taskScheduler/module.hpp b/platform/include/modules/taskScheduler/module.hpp new file mode 100644 index 0000000..5cb8f8a --- /dev/null +++ b/platform/include/modules/taskScheduler/module.hpp @@ -0,0 +1,67 @@ +#pragma once + +#include +#include +#include +#include + +#include +#include + +#include + +namespace serving::modules::taskScheduler +{ + +class Module final : public serving::modules::Module +{ + public: + + using taskFunction_t = taskr::function_t; + + Module(std::shared_ptr computeManager, std::shared_ptr taskr) + : serving::modules::Module(), + _computeManager(computeManager), + _taskr(taskr) + {} + + ~Module() override = default; + + __INLINE__ void addTask(const std::string &name, const taskFunction_t &function) + { + if (_taskNameToIndex.contains(name)) HICR_THROW_LOGIC("Task '%s' is already registered in taskScheduler module.", name.c_str()); + _functions.push_back(std::make_unique(_computeManager.get(), function)); + _tasks.push_back(std::make_unique(_functions.back().get())); + _taskNameToIndex[name] = _tasks.size() - 1; + } + + void initialize() override + { + _taskr->setTaskCallbackHandler(HiCR::tasking::Task::callback_t::onTaskSuspend, [&](taskr::Task *task) { _taskr->resumeTask(task); }); + _taskr->setFinishOnLastTask(false); + for (auto &task : _tasks) _taskr->addTask(task.get()); + _taskr->initialize(); + } + + void run() override { _taskr->run(); } + + void terminate() override { _taskr->setFinishOnLastTask(true); } + + void await() override { _taskr->await(); } + + void finalize() override { _taskr->finalize(); } + + protected: + + void service() override {} + + private: + + std::shared_ptr _computeManager; + std::shared_ptr _taskr; + + std::vector> _functions; + std::vector> _tasks; + std::unordered_map _taskNameToIndex; +}; +} // namespace serving::modules::taskScheduler diff --git a/platform/include/system/channels/base.hpp b/platform/include/system/channels/base.hpp index 82ac843..aab9c92 100644 --- a/platform/include/system/channels/base.hpp +++ b/platform/include/system/channels/base.hpp @@ -123,7 +123,8 @@ class Base __INLINE__ void lock() { _lock.lock(); } __INLINE__ void unlock() { _lock.unlock(); } - __INLINE__ bool isReady() const { return _isReady; } + __INLINE__ bool isReady() const { return _isReady; } + __INLINE__ const channelConfig_t &getConfig() const { return _config; } protected: diff --git a/platform/include/system/runtimePlan.hpp b/platform/include/system/runtimePlan.hpp new file mode 100644 index 0000000..7cb0253 --- /dev/null +++ b/platform/include/system/runtimePlan.hpp @@ -0,0 +1,68 @@ +#pragma once + +#include +#include +#include + +#include + +namespace serving::system +{ + +// Immutable versioned snapshot of a deployment's runtime state. +// +// Produced by coordinator::Module::getRuntimePlan(). Callers may hold a copy +// safely; the coordinator never mutates an already-returned plan. Version is +// monotonically increasing: two plans with the same version describe the same +// state. +struct RuntimePlan +{ + enum class ReplicaStatus + { + Active, // accepting new jobs + Draining, // finishing current job; no new jobs will be assigned + Removed // fully idle; channels may be torn down by the caller + }; + + struct ReplicaState + { + HiCR::Instance::instanceId_t instanceId; + ReplicaStatus status; + }; + + struct PartitionState + { + std::string name; + HiCR::Instance::instanceId_t coordinatorId; + std::vector replicas; + + [[nodiscard]] size_t activeCount() const noexcept + { + size_t n = 0; + for (const auto &r : replicas) + if (r.status == ReplicaStatus::Active) ++n; + return n; + } + + [[nodiscard]] size_t drainingCount() const noexcept + { + size_t n = 0; + for (const auto &r : replicas) + if (r.status == ReplicaStatus::Draining) ++n; + return n; + } + + [[nodiscard]] size_t removedCount() const noexcept + { + size_t n = 0; + for (const auto &r : replicas) + if (r.status == ReplicaStatus::Removed) ++n; + return n; + } + }; + + uint64_t version{0}; + std::vector partitions; +}; + +} // namespace serving::system diff --git a/platform/meson.build b/platform/meson.build index b0e20e8..cb4fc60 100644 --- a/platform/meson.build +++ b/platform/meson.build @@ -74,5 +74,6 @@ if meson.is_subproject() == false if get_option('buildTests') subdir('examples') endif + subdir('python') endif diff --git a/platform/python/bridge.cpp b/platform/python/bridge.cpp new file mode 100644 index 0000000..90ff9fe --- /dev/null +++ b/platform/python/bridge.cpp @@ -0,0 +1,344 @@ +// Single-rank coordinator+replica bridge exposed to Python via pybind11. +// +// The Python callback (processFc) is invoked by the replica for each job. +// It receives raw input bytes (typically a pickled SchedulerOutput) and must +// return raw output bytes (typically a pickled StepOutput). +// +// Usage from Python: +// import serving_platform +// bridge = serving_platform.Bridge(processFc, payloadCapacity=4, +// payloadBytes=262144, resultBytes=65536) +// seq_id = bridge.submit(pickled_bytes) +// seq_id, result_bytes = bridge.getResult(timeoutMs=5000.0) +// bridge.shutdown() +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +namespace py = pybind11; + +static constexpr serving::system::channels::Message::messageType_t kMsgType = 200; +static constexpr HiCR::Instance::instanceId_t kChanPayload = 3000; +static constexpr HiCR::Instance::instanceId_t kChanResult = 4000; + +// ─── Bridge ────────────────────────────────────────────────────────────────── + +class Bridge +{ + public: + + // processFcPy: Python callable(bytes) -> bytes + // payloadCapacity: max jobs in flight + // payloadBytes: max serialized input size per job + // resultBytes: max serialized output size per job + Bridge(py::object processFcPy, size_t payloadCapacity, size_t payloadBytes, size_t resultBytes); + ~Bridge(); + + // Submit one job; returns a sequence ID that matches the one in getResult. + uint64_t submit(py::bytes payload); + + // Block until one result arrives (releases GIL while waiting). + // Returns (seqId, result_bytes). Throws std::runtime_error on timeout. + std::pair getResult(double timeoutMs = 5000.0); + + void shutdown(); + + private: + + static void processFcImpl(serving::modules::roles::TaskContext &ctx, py::object &pyFn); + + Runtime _rt; + + serving::configuration::Deployment _deployment; + std::shared_ptr _cc; + std::shared_ptr _cd; + std::shared_ptr _svc; + std::shared_ptr _coord; + std::shared_ptr _replica; + std::unique_ptr _jobFactory; + + std::shared_ptr _payloadOut; + std::shared_ptr _resultIn; + std::shared_ptr _payloadIn; + std::shared_ptr _resultOut; + + struct Result + { + uint64_t seqId; + std::string bytes; + }; + std::mutex _mtx; + std::condition_variable _cv; + std::queue _queue; + + std::atomic _nextSeqId{1}; + py::object _processFcPy; +}; + +// ─── processFcImpl ────────────────────────────────────────────────────────── + +void Bridge::processFcImpl(serving::modules::roles::TaskContext &ctx, py::object &pyFn) +{ + auto slot = ctx.getInput("payload"); + if (slot == nullptr) return; + + std::string result; + { + py::gil_scoped_acquire gil; + auto pyInput = py::bytes(static_cast(slot->getPointer()), slot->getSize()); + auto pyOutput = pyFn(pyInput); + result = static_cast(pyOutput.cast()); + } + ctx.setOutput("result", result.data(), result.size()); +} + +// ─── Bridge constructor ────────────────────────────────────────────────────── + +Bridge::Bridge(py::object processFcPy, size_t payloadCapacity, size_t payloadBytes, size_t resultBytes) + : _processFcPy(std::move(processFcPy)) +{ + int fakeArgc = 0; + char *fakeArgv = nullptr; + char **pArgv = &fakeArgv; + // makeRuntime calls MPI_Init internally. Guard against double-init + // if another Bridge was previously created and torn down in this process. + int alreadyInit = 0; + MPI_Initialized(&alreadyInit); + if (alreadyInit) + throw std::runtime_error("serving_platform.Bridge: MPI is already initialized. " + "Only one Bridge may exist per process lifetime."); + _rt = makeRuntime(&fakeArgc, &pArgv); + + const auto instanceId = _rt.instanceManager->getCurrentInstance()->getId(); + auto &engine = *_rt.serving; + + // ── Deployment ──────────────────────────────────────────────────────────── + auto makeEdge = [&](const std::string &name, size_t cap, size_t sz) { + auto e = std::make_shared(name, cap, sz); + e->setPayloadCommunicationManager(_rt.communicationManager.get()); + e->setPayloadMemoryManager(_rt.memoryManager.get()); + e->setPayloadMemorySpace(_rt.bufferMemorySpace); + e->setCoordinationCommunicationManager(_rt.communicationManager.get()); + e->setCoordinationMemoryManager(_rt.memoryManager.get()); + e->setCoordinationMemorySpace(_rt.bufferMemorySpace); + return e; + }; + + _deployment.addChannel(makeEdge("payload", payloadCapacity, payloadBytes)); + _deployment.addChannel(makeEdge("result", payloadCapacity, resultBytes)); + + auto task = std::make_shared(std::string("serve")); + task->addInput("payload"); + task->addOutput("result"); + auto partition = std::make_shared("P0", instanceId); + partition->addTask(task); + partition->addReplica(std::make_shared(instanceId)); + _deployment.addPartition(partition); + + // ── Core modules ────────────────────────────────────────────────────────── + std::vector mgrs = {_rt.communicationManager.get()}; + _cc = std::make_shared(instanceId, mgrs); + _cd = std::make_shared(1); + _svc = std::make_shared(_rt.taskr); + + _svc->addService("ChannelController", _cc->getService()); + _svc->addService("ChannelDispatcher", _cd->getService()); + engine.addModule("ChannelController", _cc); + engine.addModule("ChannelDispatcher", _cd); + engine.addModule("Service", _svc); + + // ── Coordinator ─────────────────────────────────────────────────────────── + _coord = std::make_shared(1); + _coord->addMessageType(kMsgType); + engine.addModule("Coordinator", _coord); + _svc->addService("Coordinator", _coord->getService()); + + const auto intPayloadEdge = makeInternalEdgeFromTemplate("payload-coord-replica", *_deployment.getEdges()[0]); + const auto intResultEdge = makeInternalEdgeFromTemplate("result-replica-coord", *_deployment.getEdges()[1]); + + _payloadOut = _cc->addDesiredProducer(instanceId, kChanPayload, intPayloadEdge, defaultChannelKeyBuilder).lock(); + _resultIn = _cc->addDesiredConsumer(instanceId, kChanResult, intResultEdge, defaultChannelKeyBuilder).lock(); + + serving::modules::roles::coordinator::Module::inputChannelMap_t replicaInputs = {{"payload", _payloadOut}}; + serving::modules::roles::coordinator::Module::outputChannelMap_t replicaOutputs = {{"result", _resultIn}}; + _coord->addReplica(instanceId, replicaInputs, replicaOutputs); + + // ── Replica ─────────────────────────────────────────────────────────────── + auto myProcessFc = [this](serving::modules::roles::TaskContext &ctx) { Bridge::processFcImpl(ctx, _processFcPy); }; + + const auto &savedTask = _deployment.getPartitions().front()->getTasks().front(); + _replica = std::make_shared("serve", savedTask->getInputs(), savedTask->getOutputs(), myProcessFc); + _replica->addMessageType(kMsgType); + engine.addModule("Replica", _replica); + + _payloadIn = _cc->addDesiredConsumer(instanceId, kChanPayload, intPayloadEdge, defaultChannelKeyBuilder).lock(); + _resultOut = _cc->addDesiredProducer(instanceId, kChanResult, intResultEdge, defaultChannelKeyBuilder).lock(); + _replica->setCoordinator(instanceId, {{"result", _resultOut}}, {{"payload", _payloadIn}}); + + // ── JobFactory ──────────────────────────────────────────────────────────── + _jobFactory = std::make_unique(0, _deployment); + + // ── Completion callback ─────────────────────────────────────────────────── + _coord->setCompletionCallback( + [this](const serving::system::channels::Message::metadata_t &meta, const std::vector &outputs) { + std::string bytes; + if (!outputs.empty() && outputs[0].data != nullptr) + { + const auto *ptr = static_cast(outputs[0].data->getPointer()); + bytes.assign(ptr, outputs[0].data->getSize()); + } + { + std::lock_guard lk(_mtx); + _queue.push({meta.sequenceId, std::move(bytes)}); + } + _cv.notify_one(); + }); + + // ── Subscribe & start ───────────────────────────────────────────────────── + for (auto &sub : _coord->buildSubscriptions()) _cd->subscribe(sub); + for (auto &sub : _replica->buildSubscriptions()) _cd->subscribe(sub); + + { + py::gil_scoped_release release; + engine.initialize(); + engine.run(); + waitUntilReady(_payloadOut); + waitUntilReady(_resultIn); + waitUntilReady(_payloadIn); + waitUntilReady(_resultOut); + } +} + +// ─── Bridge::submit ────────────────────────────────────────────────────────── + +uint64_t Bridge::submit(py::bytes payload) +{ + const uint64_t seqId = _nextSeqId.fetch_add(1, std::memory_order_relaxed); + + serving::system::channels::Message::metadata_t md; + md.type = kMsgType; + md.groupId = 1; + md.sequenceId = seqId; + + auto job = std::make_shared(_jobFactory->createJob("serve", md)); + + auto sv = static_cast(payload); + job->getInputDependency("payload").storeData(reinterpret_cast(sv.data()), sv.size()); + job->getInputDependency("payload").setSatisfied(true); + + _coord->submitJob(job); + return seqId; +} + +// ─── Bridge::getResult ─────────────────────────────────────────────────────── + +std::pair Bridge::getResult(double timeoutMs) +{ + using clock = std::chrono::steady_clock; + const auto ddl = clock::now() + std::chrono::duration(timeoutMs); + + std::unique_lock lock(_mtx); + bool gotResult = false; + { + py::gil_scoped_release release; + gotResult = _cv.wait_until(lock, ddl, [this] { return !_queue.empty(); }); + } + if (!gotResult) throw std::runtime_error("Bridge::getResult timed out"); + + auto r = std::move(_queue.front()); + _queue.pop(); + return {r.seqId, py::bytes(r.bytes)}; +} + +// ─── Bridge::shutdown ──────────────────────────────────────────────────────── + +void Bridge::shutdown() +{ + auto &engine = *_rt.serving; + { + py::gil_scoped_release release; + engine.terminate(); + engine.await(); + } + for (auto &[type, ch] : _coord->buildUnsubscriptions()) _cd->unsubscribe(type, ch); + for (auto &[type, ch] : _replica->buildUnsubscriptions()) _cd->unsubscribe(type, ch); + _cc->removeDesiredProducer("payload-coord-replica"); + _cc->removeDesiredConsumer("result-replica-coord"); + _cc->removeDesiredConsumer("payload-coord-replica"); + _cc->removeDesiredProducer("result-replica-coord"); + // MPI_Finalize is called once at process exit via atexit, not here, + // because MPI cannot be re-initialized in the same process after finalization. +} + +Bridge::~Bridge() +{ + try + { + shutdown(); + } + catch (...) + {} +} + +// ─── pybind11 module ───────────────────────────────────────────────────────── + +PYBIND11_MODULE(serving_platform, m) +{ + m.doc() = "PyPTO Serving Platform — Python bindings for coordinator/replica bridge"; + + // MPI_Finalize must be called exactly once per process. Register it at + // module import time so it fires on clean Python exit regardless of how + // many Bridge objects are created and destroyed. + std::atexit([] { + int finalized = 0; + MPI_Finalized(&finalized); + if (!finalized) MPI_Finalize(); + }); + + py::class_(m, "Bridge") + .def(py::init(), + py::arg("processFc"), + py::arg("payloadCapacity") = 4, + py::arg("payloadBytes") = 262144, + py::arg("resultBytes") = 65536, + "Create a co-located coordinator+replica bridge.\n\n" + "processFc(bytes) -> bytes is called by the replica for every submitted job.\n" + "payloadCapacity: max concurrent jobs in flight.\n" + "payloadBytes: max serialized input size.\n" + "resultBytes: max serialized output size.") + .def("submit", &Bridge::submit, py::arg("payload"), "Submit a job. Returns the sequence ID (uint64). Thread-safe.") + .def("getResult", + &Bridge::getResult, + py::arg("timeoutMs") = 5000.0, + "Block until a result arrives. Returns (seqId, result_bytes).\n" + "Raises RuntimeError on timeout. Releases the GIL while waiting.") + .def("shutdown", &Bridge::shutdown, "Terminate the engine and release resources."); +} diff --git a/platform/python/meson.build b/platform/python/meson.build new file mode 100644 index 0000000..017b26e --- /dev/null +++ b/platform/python/meson.build @@ -0,0 +1,29 @@ +py_mod = import('python') +py = py_mod.find_installation('python3', required: false) + +if not py.found() + message('python3 not found — skipping serving_platform bindings') + subdir_done() +endif + +pybind11_inc_dir = run_command( + py, ['-c', 'import pybind11; print(pybind11.get_include())'], + check: false, +) +if pybind11_inc_dir.returncode() != 0 + message('pybind11 not installed — skipping serving_platform bindings') + subdir_done() +endif + +pybind11_inc = include_directories(pybind11_inc_dir.stdout().strip()) +examples_inc = include_directories('../examples/include') + +py.extension_module( + 'serving_platform', + 'bridge.cpp', + dependencies : [servingBuildDep], + include_directories: [pybind11_inc, examples_inc], + cpp_args : ['-fvisibility=hidden'], + subdir : 'platform', + install : false, +) diff --git a/python/core/platform_bridge.py b/python/core/platform_bridge.py new file mode 100644 index 0000000..de86bf0 --- /dev/null +++ b/python/core/platform_bridge.py @@ -0,0 +1,74 @@ +"""Python wrapper around the C++ serving_platform.Bridge. + +Replaces the mp.Queue + subprocess pattern of WorkerProcess with the +platform's coordinator→replica MPI channels. The processFc callback +receives a pickled SchedulerOutput and returns a pickled StepOutput. + +The process must be launched via mpirun -np 1 (or mpirun -np N for +multi-rank — not yet wired here) so that MPI_Init has a valid +communicator to work with. +""" +from __future__ import annotations + +import pickle +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from .scheduler import SchedulerOutput + from .types import StepOutput + + +class PlatformBridge: + """Coordinator+replica bridge backed by the C++ platform. + + Usage:: + + bridge = PlatformBridge(worker) + seq_id = bridge.submit(scheduler_output) + seq_id, step_output = bridge.get_result(timeout_ms=5000.0) + bridge.shutdown() + + The ``worker`` must already have called ``init_device_and_model()`` + before the bridge is created (model loading is the caller's + responsibility). + """ + + def __init__( + self, + worker, + *, + payload_capacity: int = 4, + payload_bytes: int = 262144, + result_bytes: int = 65536, + ) -> None: + import serving_platform # noqa: PLC0415 + + self._worker = worker + + def _processFc(raw: bytes) -> bytes: + scheduler_output = pickle.loads(raw) + step_output = worker._execute_step(scheduler_output) + return pickle.dumps(step_output) + + self._bridge = serving_platform.Bridge( + _processFc, + payload_capacity, + payload_bytes, + result_bytes, + ) + + def submit(self, scheduler_output: SchedulerOutput) -> int: + """Pickle and submit one scheduler batch. Returns the sequence ID.""" + raw = pickle.dumps(scheduler_output) + return self._bridge.submit(raw) + + def get_result(self, timeout_ms: float = 5000.0) -> tuple[int, StepOutput]: + """Wait for one result. Returns (seq_id, StepOutput). + + Raises RuntimeError on timeout. + """ + seq_id, raw = self._bridge.getResult(timeout_ms) + return seq_id, pickle.loads(raw) + + def shutdown(self) -> None: + self._bridge.shutdown() diff --git a/tests/test_platform_bridge.py b/tests/test_platform_bridge.py new file mode 100644 index 0000000..19d2101 --- /dev/null +++ b/tests/test_platform_bridge.py @@ -0,0 +1,151 @@ +"""Integration test for PlatformBridge (no NPU required). + +Must be launched via mpirun so MPI_Init has a valid communicator: + + mpirun -np 1 --oversubscribe python3 -m pytest tests/test_platform_bridge.py -v + +The test uses a dummy processFc that echoes the input back (no real +model), so it runs on any machine with the C++ platform built. +""" +from __future__ import annotations + +import pickle +import time + +from dataclasses import dataclass, field + +import pytest + + +# ── module-level dataclasses (needed for pickle across threads) ─────────────── + +@dataclass +class _FakeRequest: + request_id: str + prompt_token_ids: list[int] = field(default_factory=list) + output_token_ids: list[int] = field(default_factory=list) + temperature: float = 1.0 + top_p: float = 1.0 + top_k: int = 0 + + @property + def num_tokens(self): + return len(self.prompt_token_ids) + len(self.output_token_ids) + + @property + def num_prompt_tokens(self): + return len(self.prompt_token_ids) + + +@dataclass +class _FakeScheduledRequest: + request: _FakeRequest + num_new_tokens: int + is_prefill: bool + num_computed_tokens: int = 0 + block_ids: list[int] = field(default_factory=list) + + +@dataclass +class _FakeSchedulerOutput: + scheduled_requests: list[_FakeScheduledRequest] = field(default_factory=list) + num_prefill_tokens: int = 0 + num_decode_tokens: int = 0 + + +# ── helpers ────────────────────────────────────────────────────────────────── + +# ── fixtures ───────────────────────────────────────────────────────────────── + +@pytest.fixture(scope="session") +def echo_bridge(): + """Session-scoped Bridge whose processFc echoes the payload back unchanged. + + Session-scoped because MPI can only be initialized once per process. + """ + try: + import serving_platform + except ImportError: + pytest.skip("serving_platform C++ extension not built — run `ninja -C platform/build python/serving_platform`") + bridge = serving_platform.Bridge(lambda raw: raw, 4, 4096, 4096) + yield bridge + bridge.shutdown() + + +# ── tests ──────────────────────────────────────────────────────────────────── + +def test_single_roundtrip(echo_bridge): + payload = pickle.dumps({"hello": "world", "n": 42}) + seq_id = echo_bridge.submit(payload) + got_id, result = echo_bridge.getResult(timeoutMs=2000.0) + assert got_id == seq_id + assert pickle.loads(result) == {"hello": "world", "n": 42} + + +def test_sequence_ids_are_unique(echo_bridge): + n = 10 + ids = [echo_bridge.submit(pickle.dumps(i)) for i in range(n)] + assert len(set(ids)) == n, "sequence IDs must be unique" + results = {} + for _ in range(n): + sid, raw = echo_bridge.getResult(timeoutMs=5000.0) + results[sid] = pickle.loads(raw) + assert len(results) == n + + +def test_throughput_overhead(echo_bridge): + """Baseline vs platform latency — prints but does not assert ratio. + + The platform MPI loopback adds ~1 ms per job. This test just + documents the overhead at runtime so CI output captures it. + """ + n = 50 + dummy = pickle.dumps(b"x" * 256) + + # Baseline: direct Python call + def _direct(raw: bytes) -> bytes: + return raw + + t0 = time.perf_counter() + for _ in range(n): + _direct(dummy) + baseline_ms = (time.perf_counter() - t0) * 1000.0 + + # Platform path + t0 = time.perf_counter() + for i in range(n): + echo_bridge.submit(dummy) + for _ in range(n): + echo_bridge.getResult(timeoutMs=5000.0) + platform_ms = (time.perf_counter() - t0) * 1000.0 + + overhead_ms = (platform_ms - baseline_ms) / n + print( + f"\n[Baseline] {n} calls | {baseline_ms:.3f} ms total | {baseline_ms/n:.4f} ms/call" + f"\n[Platform] {n} jobs | {platform_ms:.3f} ms total | {platform_ms/n:.4f} ms/call" + f"\n[Overhead] +{overhead_ms:.3f} ms/call" + ) + + +def test_pickling_scheduler_output(echo_bridge): + """Round-trip a realistic SchedulerOutput-shaped object.""" + # Classes must be at module level for pickle to work across threads. + out = _FakeSchedulerOutput( + scheduled_requests=[ + _FakeScheduledRequest( + request=_FakeRequest("req-1", list(range(128))), + num_new_tokens=128, + is_prefill=True, + block_ids=list(range(8)), + ) + ], + num_prefill_tokens=128, + ) + + raw = pickle.dumps(out) + seq_id = echo_bridge.submit(raw) + _, res = echo_bridge.getResult(timeoutMs=2000.0) + + recovered = pickle.loads(res) + assert recovered.scheduled_requests[0].request.request_id == "req-1" + assert recovered.num_prefill_tokens == 128