feat: Add a cooling_down flag to SubnetTopology

[mraszyk]

A "cooling down" subnet stops being sent new messages and lets the messages already in flight drain.
While a subnet is cooling down:

• It inducts all messages from its incoming stream slices.
• It inducts no ingress messages.
• No subnet routes messages to its (outgoing) stream (incl. the loopback stream) to the subnet that is cooling down, except for messages (responses) in subnet output queues that are always routed into streams.
• It keeps executing messages in its input queues.

There is no registry field backing the flag yet, so it is always false in production and only tests set it. The registry plumbing that can actually set the flag must not ship until this code is deployed everywhere, since it also introduces a new error code that requires a staged roll-out.

[Copilot]

Pull request overview

This PR introduces a “cooling down” mode for subnets by adding a cooling_down flag to SubnetTopology, then using it across message routing, ingress handling, and scheduling to quiesce a subnet (reject new work, drain in-flight work) while preserving correctness and retryability via a new ErrorCode::SubnetCoolingDown.

Changes:

• Add SubnetTopology::cooling_down (incl. protobuf encoding/decoding) and topology helpers (NetworkTopology::is_cooling_down, ReplicatedState::is_own_subnet_cooling_down).
• Enforce cooling-down semantics in stream building (reject requests, retain responses/refunds) and ingress payload selection/validation, and skip canister execution in the scheduler while still draining subnet queues.
• Add ErrorCode::SubnetCoolingDown and update related mappings/tests across protobuf, canonical state, PocketIC, and error-type conversions.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
rs/test_utilities/state/src/lib.rs	Initializes new SubnetTopology.cooling_down field in state test builder.
rs/test_utilities/execution_environment/src/lib.rs	Initializes cooling_down in generated subnet topologies for tests.
rs/state_manager/tests/state_manager.rs	Updates state manager tests to include cooling_down in SubnetTopology literals.
rs/replicated_state/src/replicated_state.rs	Adds ReplicatedState::is_own_subnet_cooling_down() helper.
rs/replicated_state/src/metadata_state/tests.rs	Extends topology roundtrip test and adds NetworkTopology::is_cooling_down() unit test.
rs/replicated_state/src/metadata_state/proto.rs	Adds protobuf (de)serialization for SubnetTopology.cooling_down.
rs/replicated_state/src/metadata_state.rs	Adds NetworkTopology::is_cooling_down() and documents SubnetTopology::cooling_down semantics.
rs/replicated_state/src/canister_state/system_state.rs	Generalizes stop-context filtering (is_expired → should_reject) to support cooling-down rejection.
rs/protobuf/src/state/ingress/mod.rs	Adds mapping for ErrorCodePublic::SubnetCoolingDown ↔ ErrorCode::SubnetCoolingDown.
rs/protobuf/src/gen/types/state.ingress.v1.rs	Adds SubnetCoolingDown = 211 to generated public error-code enum/string conversions.
rs/protobuf/src/gen/state/state.metadata.v1.rs	Adds cooling_down field to generated state metadata protobuf.
rs/protobuf/src/gen/state/state.ingress.v1.rs	Adds SubnetCoolingDown = 211 to generated state ingress error-code enum/string conversions.
rs/protobuf/def/state/metadata/v1/metadata.proto	Adds cooling_down field to SubnetTopology protobuf schema.
rs/protobuf/def/state/ingress/v1/ingress.proto	Adds ERROR_CODE_SUBNET_COOLING_DOWN = 211 to ingress error-code schema.
rs/messaging/src/state_machine/tests.rs	Updates state machine tests’ subnet topology literals with cooling_down.
rs/messaging/src/routing/stream_builder/tests.rs	Adds comprehensive tests for cooling-down routing behavior (reject/retain/precedence/loopback).
rs/messaging/src/routing/stream_builder.rs	Implements cooling-down routing semantics for requests/responses/refunds and exports metrics labels.
rs/messaging/src/message_routing.rs	Sets cooling_down: false when populating topology from registry (no registry backing yet).
rs/interfaces/src/ingress_manager.rs	Adds InvalidIngressPayloadReason::SubnetCoolingDown.
rs/ingress_manager/src/ingress_selector.rs	Rejects ingress payload build/validation while cooling down; adds tests for this behavior.
rs/execution_environment/src/scheduler/tests.rs	Adds scheduler tests verifying cooling-down behavior (no canister execution, stop_canister handling).
rs/execution_environment/src/scheduler/test_utilities.rs	Updates IC00 ingress injection helper to return MessageId and set sender deterministically.
rs/execution_environment/src/scheduler/scheduler_metrics.rs	Adds metric counter for inner-round iterations skipped due to cooling down.
rs/execution_environment/src/scheduler.rs	Skips canister execution in inner rounds when cooling down (after draining subnet queues).
rs/execution_environment/src/history.rs	Adds dashboard label mapping for SubnetCoolingDown.
rs/execution_environment/src/execution_environment/tests.rs	Adds ingress-filter test ensuring cooling-down ingress is rejected with SubnetCoolingDown.
rs/execution_environment/src/execution_environment.rs	Adds ingress filter check and stop_canister rejection semantics while cooling down.
rs/canonical_state/tests/hash_tree.rs	Updates error-code change guard hash due to new error code.
rs/canonical_state/src/traversal.rs	Updates traversal tests’ subnet topology literals with cooling_down.
packages/pocket-ic/src/lib.rs	Adds SubnetCoolingDown = 211 to PocketIC error-code enum conversions.
packages/pocket-ic/CHANGELOG.md	Documents newly added PocketIC error code variant.
packages/ic-error-types/src/lib.rs	Adds SubnetCoolingDown to error-code enum, reject-code mapping, and tests.

Comments suppressed due to low confidence (1)

rs/execution_environment/src/execution_environment.rs:4697

• This reject message also uses a \ line continuation with an indented next line, which will insert indentation spaces into the reject string returned to the calling canister.

                    StopCanisterReply::SubnetCoolingDown => Payload::Reject(RejectContext::new(
                        RejectCode::SysTransient,
                        format!(
                            "Subnet {} is cooling down, so canister {canister_id}'s stop request \
                            was rejected",
                            self.own_subnet_id
                        ),
                    )),

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated no new comments.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to 3a84275.

Security Overview

• 🔎 Scanned files: 26 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/f5e9d2bb-29a4-4b64-902a-391d65fd8b2d?scanId=46c50d25-d849-4472-861d-7f89aed4d142&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► packages/ic-error-types/src/lib.rs     Add SubnetCoolingDown variant to error handling ► packages/pocket-ic/CHANGELOG.md     Document SubnetCoolingDown addition ► packages/pocket-ic/src/lib.rs     Add SubnetCoolingDown to ErrorCode enum ► rs/canonical_state/src/traversal.rs     Add cooling_down field in SubnetTopology tests ► rs/canonical_state/tests/hash_tree.rs     Update expected error code list (include 211)
Enhancement	► rs/execution_environment/src/execution_environment.rs     Return SubnetCoolingDown error for ingress when cooling down ► rs/execution_environment/src/execution_environment/tests.rs     Add ingress cooling down test cases
Enhancement	► rs/execution_environment/src/history.rs     Describe SubnetCoolingDown in dashboard labels
Enhancement	► rs/ingress_manager/src/ingress_selector.rs     Validate and error on SubnetCoolingDown during ingress payload validation ► rs/interfaces/src/ingress_manager.rs     Extend InvalidIngressPayloadReason with SubnetCoolingDown
Enhancement	► rs/messaging/src/message_routing.rs     Initialize cooling_down field in registry reader default ► rs/messaging/src/routing/stream_builder.rs     Introduce logic to retain or route messages based on cooling down state ► rs/messaging/src/routing/stream_builder/tests.rs     Add extensive tests for cooling down behavior and streaming retention
Enhancement	► rs/messaging/src/routing/stream_builder.rs     Add handling for exempting certain outputs when subnet is cooling down ► rs/messaging/src/state_machine/tests.rs     Update test fixtures to include cooling_down flag
Enhancement	► rs/protobuf/def/state/ingress/v1/ingress.proto     Add ERROR_CODE_SUBNET_COOLING_DOWN constant to enum

[alin-at-dfinity]

While it is true that the subnet stops accepting new ingress messages, it will definitely induct canister messages (that were already in streams; or that are still being routed into streams by subnets with an older registry version).

I would even argue that inducting messages from streams is more central to what we're trying to achieve (empty the streams and subnet queues) than rejecting / not inducting ingress messages (which merely avoids generating any extra subnet messages while we're trying to consume them).

Edit: Actually not inducting ingress messages is just as important, as we're also trying to empty the ingress history of everything but Processing statuses.

With a view to their relative importance, I would personally structure the bullet points below like: (i) induct all messages from streams; (ii) don't route messages to streams (maybe except subnet responses?); (iii) consume subnet messages, as usual; and (iv) don't execute canister messages (to avoid creating more subnet messages).

Edit: So either before (i) or before (iii) there should be a "don't induct any more ingress messages".

[mraszyk]

Rephrased: d02f44b

[alin-at-dfinity]

LGTM, modulo some nitpicks and a couple of bigger points:

1. We probably need to always route subnet responses, or they'll prevent us from halting the subnet and moving on with the merge.
2. (Something of a personal preference) I wouldn't bother rejecting requests when routing to a cooling down subnet. It's unnecessary, it requires extra logic and still leaves us with all the requests hidden behind a response.

[alin-at-dfinity]

It may be worth elaborating a bit on what effects this has. E.g. it is not immediately obvious that returning an error will cause the ingress message to not be inducted (as opposed to, e.g., simply rejected).

(It is actually obvious, now that I've thought about it, but it's still worth mentioning, since it took me a while.)

[mraszyk]

returning an error will cause the ingress message to not be inducted (as opposed to, e.g., simply rejected)

What's the difference between "not inducted" and "simply rejected" here?

[alin-at-dfinity]

I guess I meant that if this function returns an error, then the ingress message is not even inducted by the payload builder as opposed to being rejected later on by the DSM.

In my case, I wasn't expecting code in rs/execution_environment/src/execution_environment.rs to have effect on the ingress payload builder.

[alin-at-dfinity]

Idle question, for my own edification: assuming that any such stop contexts exist, wouldn't it make sense to time them out by now? Or were we concerned that the caller would not expect a (or this specific) reject response?

[alin-at-dfinity]

Not entirely accurate: we do induct XNet messages, don't we? We just no longer route them.

[mraszyk]

36d4768

[alin-at-dfinity]

Couldn't we have a race condition where the "cooling down" flag is flipped while the ingress pool is non-empty?

If so, the ingress filter check is technically only an optimization. This is the only check than ensures correctness.

[mraszyk]

Indeed: 800ae16