Skip to content

perf: per-round heartbeat token for precise lease deadline anchoring #405

Description

@JoshuaChi

Problem

The current lease deadline is anchored to last_heartbeat_send_ts, a single
shared field overwritten on every AppendEntries dispatch. When tick() and
drain_client_cmds() both fire in the same Raft loop iteration, a stale ACK
may use a newer send_ts, slightly over-extending the lease window. The error
is microsecond-level and absorbed by the network_rtt_p99_ms safety margin,
but it is not strictly correct per-round.

Additionally, nanosecond-precision timestamps are needed for future sub-ms
lease accuracy requirements.

Proposed Solution

Assign a monotonic u64 token to each AppendEntries round at dispatch time.
Carry the token in the request and echo it back in the response. The leader
maintains HashMap<token, send_ts_ns> (nanosecond precision) keyed by token.
On ACK, look up the exact send_ts for that specific round.

// dispatch
let token = self.next_heartbeat_token.fetch_add(1, Ordering::Relaxed);
self.pending_send_ts.insert(token, now_ns());  // nanosecond precision
request.heartbeat_token = token;

// on ACK
if let Some(send_ts_ns) = self.pending_send_ts.remove(&response.heartbeat_token) {
    self.update_lease_timestamp_ns(send_ts_ns, lease_duration_ns);
}

Eviction: tokens older than current_token - N (e.g. N=10) are dropped on insert.

Changes Required

  • AppendEntriesRequest: add heartbeat_token: u64 (proto change, minor version bump)
  • AppendEntriesResponse: echo heartbeat_token: u64
  • LeaderState: replace last_heartbeat_send_ts: u64 with pending_send_ts: HashMap<u64, u64>
  • ReadLease / update_lease_timestamp: extend to nanosecond precision
  • Config: lease_duration_ns replaces lease_duration_ms

Precision Target

Nanosecond-precision send_ts throughout the lease renewal path.

Notes

  • This is a protocol-level change (proto field addition) requiring coordinated rollout
  • Method A (last_heartbeat_send_ts + network_rtt_p99_ms config margin) remains the
    safety baseline; this ticket improves precision, not the safety level
  • Reference: openraft HeartbeatEvent.time achieves equivalent correctness via
    architectural decoupling (separate HeartbeatWorker task per follower)

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:raft-consensusCore Raft protocol — leader election, log replication, membership changes.concern:correctnessLinearizability, serializability, consistency guarantees across replicas.concern:performanceThroughput, latency, CPU/memory/disk efficiency.concern:reliabilityFault tolerance, correctness under failures, crash recovery, network partitions.

    Fields

    No fields configured for Feature.

    Projects

    Status
    Backlog

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions