[Protocol Versioning Epic] Version Execution Stack via Dynamic Protocol State #6999

DACI

This is an epic for bundling all tasks related to version the Execution Stack via Dynamic Protocol State.

Motivation and importance of this work

The Dynamic Protocol State includes a framework that has been developed based on the learnings of the existing HCU mechanism. It provides us with the ability to perform general purpose upgrades of the the flow protocol and is much more generally applicable and robust compared to the initial HCU mechanism (only applicable for upgrading the Execution Stack).

• We are starting to use the Dynamic Protocol State for coordinating upgrades of other parts of the protocol.
  We want to utilize the same framework as for execution stack upgrades, to reduce code and intellectual complexity that would
  otherwise result from using significantly different approaches.
• Midterm goal: upgrades of the execution stack are by far the most frequent. By utilizing the Dynamic Protocol State for those upgrades, we want to generate learnings on where the new framework should be improved. Specifically, we want to avoid extending the old HCU mechanism in a way that is incompatible with the Dynamic Protocol State and needing to be rewritten later.
• Longer-term goal: Access Nodes [ANs] can decide on whose blocks’ execution states they can run scripts for across different version of the execution stack.

For more details on the advantages of the new Protocol-State-mediated upgrade framework, please see FLIP 296.

High-level breakdown of work

Milestone 1:

• [Protocol Versioning] Document versioning mechanics for dynamic protocol state #6887
• [Protocol Versioning] Execution Parameters in the Dynamic Protocol State #7000

Milestone 2:
Once the previous issue #7000 is done, we can store the Execution Stack Version (and other protocol parameters controlling the execution) in the Dynamic protocol state. Now we need to add functionality for updating these parameters in the Protocol State:

• [Protocol Versioning] Rules when to Accept or Reject an update Service Event #7143
• [Protocol Versioning] Implement Service Events on Protocol Layer (flow-go only) #7144
• [Protocol Versioning] Implement State Machine for Rejecting Service Event or Accepting it and Updating Protocol State #7145

Milestone 3:

In this milestone, we will focus on the Governance Transactions and Governance Smart Contracts. The system smart contracts will need to be updated and extended with the functionality to emit Service Events, which direct the Protocol state to update the Execution Stack parameters.

• [Protocol Versioning] Governance Smart Contracts to emit the newly defined Service Events #7134

Milestone 4:

In a nutshell, this milestone is for changing the Execution Node to listen to the new triggers from the Protocol State, directing it to stop execution when it reaches the threshold view for the new behaviour.

• [Protocol Versioning] Execution Node's StopControl uses Protocol State #7135

End-to-End Testing Deployment:
Ideally both, but at least one of the following:

• [Protocol Versioning] Integration Test for new HCU mechanism #7146
• [Protocol Versioning] Benchnet test #7148

Deployment:

• [Protocol Versioning] Deployment plan #7136 (it is possible but non-trivial to deploy via a rolling software update)
• [Protocol Versioning] Deployment #7147

Visualization of resulting versioning flow (implementation milestones highlighted)

Low-hanging fruit to help with sporks

Add an HCU-style stopping logic for the ENs, so they don't execute past a governance-specified view threshold. This is helpful for progressively shutting down the network in preparation of a spork.

Further Reading

• notion doc Execution State Parameters in the Dynamic Protocol State