Problem
The current lease deadline is anchored to last_heartbeat_send_ts, a single
shared field overwritten on every AppendEntries dispatch. When tick() and
drain_client_cmds() both fire in the same Raft loop iteration, a stale ACK
may use a newer send_ts, slightly over-extending the lease window. The error
is microsecond-level and absorbed by the network_rtt_p99_ms safety margin,
but it is not strictly correct per-round.
Additionally, nanosecond-precision timestamps are needed for future sub-ms
lease accuracy requirements.
Proposed Solution
Assign a monotonic u64 token to each AppendEntries round at dispatch time.
Carry the token in the request and echo it back in the response. The leader
maintains HashMap<token, send_ts_ns> (nanosecond precision) keyed by token.
On ACK, look up the exact send_ts for that specific round.
// dispatch
let token = self.next_heartbeat_token.fetch_add(1, Ordering::Relaxed);
self.pending_send_ts.insert(token, now_ns()); // nanosecond precision
request.heartbeat_token = token;
// on ACK
if let Some(send_ts_ns) = self.pending_send_ts.remove(&response.heartbeat_token) {
self.update_lease_timestamp_ns(send_ts_ns, lease_duration_ns);
}
Eviction: tokens older than current_token - N (e.g. N=10) are dropped on insert.
Changes Required
AppendEntriesRequest: add heartbeat_token: u64 (proto change, minor version bump)
AppendEntriesResponse: echo heartbeat_token: u64
LeaderState: replace last_heartbeat_send_ts: u64 with pending_send_ts: HashMap<u64, u64>
ReadLease / update_lease_timestamp: extend to nanosecond precision
- Config:
lease_duration_ns replaces lease_duration_ms
Precision Target
Nanosecond-precision send_ts throughout the lease renewal path.
Notes
- This is a protocol-level change (proto field addition) requiring coordinated rollout
- Method A (
last_heartbeat_send_ts + network_rtt_p99_ms config margin) remains the
safety baseline; this ticket improves precision, not the safety level
- Reference: openraft
HeartbeatEvent.time achieves equivalent correctness via
architectural decoupling (separate HeartbeatWorker task per follower)
Problem
The current lease deadline is anchored to
last_heartbeat_send_ts, a singleshared field overwritten on every AppendEntries dispatch. When tick() and
drain_client_cmds() both fire in the same Raft loop iteration, a stale ACK
may use a newer send_ts, slightly over-extending the lease window. The error
is microsecond-level and absorbed by the
network_rtt_p99_mssafety margin,but it is not strictly correct per-round.
Additionally, nanosecond-precision timestamps are needed for future sub-ms
lease accuracy requirements.
Proposed Solution
Assign a monotonic u64 token to each AppendEntries round at dispatch time.
Carry the token in the request and echo it back in the response. The leader
maintains
HashMap<token, send_ts_ns>(nanosecond precision) keyed by token.On ACK, look up the exact send_ts for that specific round.
Eviction: tokens older than current_token - N (e.g. N=10) are dropped on insert.
Changes Required
AppendEntriesRequest: addheartbeat_token: u64(proto change, minor version bump)AppendEntriesResponse: echoheartbeat_token: u64LeaderState: replacelast_heartbeat_send_ts: u64withpending_send_ts: HashMap<u64, u64>ReadLease/update_lease_timestamp: extend to nanosecond precisionlease_duration_nsreplaceslease_duration_msPrecision Target
Nanosecond-precision send_ts throughout the lease renewal path.
Notes
last_heartbeat_send_ts+network_rtt_p99_msconfig margin) remains thesafety baseline; this ticket improves precision, not the safety level
HeartbeatEvent.timeachieves equivalent correctness viaarchitectural decoupling (separate HeartbeatWorker task per follower)