v0.2.4 - Async IO Architecture + Jepsen Validated

🎯 Key Features

Async IO Architecture (#295, #313, #329–#346)

• Raft event loop fully non-blocking — WAL writes, state machine apply, and replication
  all run off the hot path
• AppendEntries uses a persistent bidirectional stream per peer; replication is pipelined
  across followers
• Dedicated raft-io and SM apply threads eliminate contention on the event loop

Watch API Improvements (#379, #380, #294)

• prev_kv: watch events optionally include the previous value before the change
• Progress heartbeat: periodic events confirm the stream is alive — enables reliable
  gap detection at reconnect
• Prefix watch: watch_prefix(prefix) streams all events under a path prefix
• Buffer overflow protection: watcher limits prevent unbounded memory growth (#294)

Prefix Scan (#378)

• client.scan_prefix(prefix) — returns all keys under a prefix with a revision anchor
• Designed for reliable watch reconnect: scan → rebuild state → resume watch,
  filtering events by revision

Cluster Membership Streaming (#327, #328)

• EmbeddedEngine::watch_membership() / GrpcClient::watch_membership() — real-time
  membership change stream in both embedded and standalone modes
• Yields current snapshot on connect, then one event per committed ConfChange

Simpler Startup (#303)

• EmbeddedEngine::start(data_dir) and StandaloneEngine::run(data_dir, shutdown_rx)
  — no config file required for common cases

Jepsen Validated (#369)

• 5 workloads + 6-hour soak test under combined kill/partition/pause faults
• See Correctness Guarantees

---

🚀 Performance

Embedded Mode vs v0.2.3 (AWS EC2 c5.2xlarge, 3-node, 5-round average)

Scenario	v0.2.3	v0.2.4	Δ
HC Write	64K ops/s	110K ops/s	+72% ✅
Linearizable Read	181K ops/s	327K ops/s	+81% ✅
Lease Read	379K ops/s	341K ops/s	-10% ⚠️
Eventual Read	395K ops/s	363K ops/s	-8% ⚠️

⚠️ Lease/Eventual Read show a ~8–10% regression from scheduling overhead introduced
by the async IO architecture. Root cause identified — fix targeted for v0.2.5.
Linearizable Read and Write are unaffected.

Embedded Mode vs etcd 3.2.0 (AWS EC2 3-node)

Scenario	d-engine v0.2.4	etcd 3.2.0	Δ
HC Write	110,798 ops/s	44,341 ops/s	+2.5x ✅
Linearizable Read	327,355 ops/s	141,578 ops/s	+2.3x ✅
Eventual Read	363,407 ops/s	185,758 ops/s	+96% ✅

Standalone Mode vs v0.2.3 (AWS EC2 3-node)

Scenario	v0.2.3	v0.2.4	Δ
HC Write	36K ops/s	59K ops/s	+65% ✅
Linearizable Read	51K ops/s	77K ops/s	+53% ✅
Lease Read	62K ops/s	76K ops/s	+22% ✅
Eventual Read	151K ops/s	170K ops/s	+12% ✅

Standalone Linearizable Read (77K ops/s) is below etcd's 141K ops/s. etcd uses a
read-index optimization that avoids a full Raft round-trip per read; d-engine
currently issues a full consensus round per standalone linearizable request.
A read-index optimization is planned.

See benchmark report for full details.

---

🐛 Fixes

• #390: Fixed Raft lease clock (SystemTime → Instant) — lease validity now uses
  monotonic time, preventing false expiry under clock adjustments. With a correct lease
  implementation, linearizable reads are served locally when the leader holds a valid
  lease (327K ops/s embedded, +81% vs v0.2.3)
• #381: Linearizable reads incorrectly bypassed quorum check in multi-voter clusters
  — now correctly requires majority confirmation
• #371: WriteBatchWithIndex used in apply_chunk to prevent CAS stale-read under
  concurrent apply
• #308: Snapshot install success now driven by apply result, not transfer ACK —
  fixes silent install failures
• #315: Snapshot disk I/O isolated from Raft event loop via spawn_blocking —
  prevents event loop stalls under large snapshots
• #253: Snapshot stale-file corruption on retry fixed; configurable chunk timeout added
• #290: Reduced snapshot cooldown and fixed check throttling
• #294: Watch buffer overflow protection

---

⚠️ Breaking Changes

StateMachine::apply_chunk Signature (#388)

// Before (v0.2.3)
fn apply_chunk(&mut self, entries: Vec<Entry>) -> Vec<ApplyResult>;

// After (v0.2.4)
fn apply_chunk(&mut self, entries: &[ApplyEntry]) -> Vec<ApplyResult>;

Update your StateMachine implementation to accept &[ApplyEntry]. ApplyEntry
contains the same data — this change eliminates an unnecessary allocation per apply batch.

See MIGRATION_GUIDE.md.

Wire protocol is compatible with v0.2.3 — rolling upgrades are safe.

---

📦 Installation

Embedded mode:

[dependencies]
d-engine = "0.2.4"

Standalone mode (gRPC):

[dependencies]
d-engine = { version = "0.2.4", features = ["client"], default-features = false }

---

🧪 Testing & Reliability

• Jepsen validated: 5 workloads + 6-hour soak test under combined kill/partition/pause faults
• 1000+ tests, all passing with cargo nextest run --all-features

---

📚 Resources

• Docs: docs.rs/d-engine
• Migration Guide: MIGRATION_GUIDE.md
• Examples: examples/
• Benchmarks: benches/reports/v0.2.4/
• Crates.io: crates.io/crates/d-engine

---

Full Changelog: v0.2.3...v0.2.4

v0.2.3 - CAS Operations, Protocol Compliance & Performance Optimizations

🎯 Key Features

Compare-And-Swap (CAS) Operations

• Atomic CAS primitive for distributed coordination (#258 #263)
• Use cases: Distributed locks, leader election, optimistic updates
• API: client.compare_and_swap(key, expected, new_value)
• Performance: No extra round-trips, comparable to regular writes

Unified Client API

• Single ClientApi trait for all operations (#258)
• Breaking: KvClient → ClientApi, KvError → ClientApiError
• Simpler developer experience, less boilerplate

Client Reliability

• Client::refresh() - Explicit cluster rediscovery after failover (#278)
• Fixed TOCTOU race in connection pool (leader crash during connect)
• cluster_ready_timeout configuration for startup/refresh behavior

---

🚀 Performance Improvements

Drain-based batch architecture (#266):

• Eliminated ~1ms timeout penalty under low load
• Natural batching under high load
• Embedded mode results (vs v0.2.2):
  • Linearizable read: +82% throughput (508K ops/s)
  • Lease read: +73% throughput (852K ops/s)
  • Eventual read: +71% throughput (859K ops/s)

AWS EC2 3-node cluster (vs etcd 3.2):

• Write: 64K ops/s (+45%)
• Linearizable read: 181K ops/s (+28%)
• Eventual read: 395K ops/s (+2.1x)

See benchmark report for details.

---

🐛 Critical Fixes

• #288: Node crash on RPC timeout (Candidate vote handling)
• #288: Leader stepdown buffer leak (propose_buffer not drained)
• #268: Commit index calculation when all peers timeout
• #268: Channel leak from discarded senders in empty batch
• #282: Connection pool deadline overshoot prevention

---

⚠️ Breaking Changes

Wire Protocol Incompatible with v0.2.2

Protobuf enums now include UNSPECIFIED=0 (#279):

• NodeRole: Follower=1, Candidate=2, Leader=3, Learner=4 (shifted from 0-3)
• NodeStatus: Promotable=1, ReadOnly=2, Active=3 (shifted from 0-2)

Migration Required:

1. Update config files: role = 0 → role = 1 (Follower)
2. Upgrade all nodes simultaneously (no rolling upgrade)
3. Upgrade client SDKs to v0.2.3

See MIGRATION_GUIDE.md for step-by-step instructions.

API Changes

• KvClient → ClientApi trait
• KvError → ClientApiError
• Default persistence: MemFirst → DiskFirst (add config to restore)

---

📦 Installation

Embedded mode:

[dependencies]
d-engine = "0.2.3"

Standalone mode (gRPC):

[dependencies]
d-engine = { version = "0.2.3", features = ["client"], default-features = false }

---

🧪 Testing & Reliability

• 17 new integration tests covering role transitions, drain batching, fairness
• All tests passing with cargo nextest run --all-features
• Upgraded Go toolchain to 1.25.0 (security fix)

---

📚 Resources

• Docs: docs.rs/d-engine
• Examples: GitHub examples/
• Benchmarks: benches/reports/v0.2.3/
• Crates.io: crates.io/crates/d-engine

---

Full Changelog: v0.2.2...v0.2.3

v0.2.2 - Performance & Stability Improvements

🎯 Key Improvements

ReadIndex Batching - 440% Performance Boost

• Linearizable reads: 440% throughput improvement (#236)
• Embedded mode: Zero-copy performance validated (#233)
• Benchmark: 279K linearizable reads/sec (vs 12K in v0.2.1)

Cluster State APIs - HA Support

New APIs for production deployments (#234):

• is_leader() - Leader detection for load balancers
• node_id() - Instance identification
• current_term() - Raft term monitoring
• wait_ready() - Startup synchronization

Use cases: Health checks, leader discovery, rolling deployments

---

🐛 Critical Fixes

• #228: Inconsistent reads in single-node mode (returning stale data)
• #212: Learner promotion stuck due to voter count bugs
• #242: Storage layer durability bugs causing data loss
• #235: Single-node snapshot purge causing NoPeersAvailable error
• #209: wait_ready() timeout race condition on startup

---

📦 Installation

Embedded mode (in-process):

[dependencies]
d-engine = "0.2"

Standalone mode (gRPC client):

[dependencies]
d-engine = { version = "0.2", features = ["client"], default-features = false }

---

📊 Performance Highlights

Embedded mode (M2 Mac vs etcd on 3 GCE instances):

• Write: 203K ops/sec (4.6x faster than etcd)
• Linearizable read: 279K ops/sec (23x faster than etcd)
• Latency: <0.1ms (26x lower than etcd)

See benchmarks for details.

---

📚 Documentation

• Quick Start: 5-minute embedded guide
• Examples: GitHub examples/
• API Docs: docs.rs/d-engine

---

⚠️ Breaking Changes

None - All changes are backward compatible with v0.2.1

---

🔗 Links

• Crates.io: https://crates.io/crates/d-engine
• Full Changelog: CHANGELOG.md
• Migration Guide: MIGRATION_GUIDE.md

v0.2.1 - Embedded Mode & Modular Architecture

🎉 Major Milestone: v0.2 Series Launch

This is the first stable release of the v0.2 series, introducing embedded mode and modular workspace architecture. After months of development, d-engine is now ready for developers building distributed Rust applications.

---

🚀 Major Features

1. EmbeddedEngine - True In-Process Integration

Embed d-engine directly into your Rust application with zero network overhead:

use d_engine::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Start embedded engine
    let engine = EmbeddedEngine::start().await?;
    engine.wait_ready(Duration::from_secs(5)).await?;

    // Zero-overhead KV operations (<0.1ms)
    let client = engine.client();
    client.put(b"key".to_vec(), b"value".to_vec()).await?;

    engine.stop().await?;
    Ok(())
}

Benefits:

• <0.1ms latency - Direct memory access, no gRPC serialization
• Single binary - No separate server process needed
• Zero configuration - Auto-creates data directories

Use cases: High-frequency trading, real-time analytics, embedded systems

---

2. Modular Workspace - Lighter Dependencies

Choose what you need with feature flags:

# Client-only (lightweight, for connecting to standalone server)
d-engine = { version = "0.2", features = ["client"], default-features = false }

# Embedded server (full Raft engine in-process)
d-engine = { version = "0.2", features = ["server"] }

# Everything (default)
d-engine = "0.2"

Impact: 40-60% faster compile times for client-only usage

---

3. TTL/Lease - Automatic Key Expiration

Built-in support for distributed locks and session management:

// Auto-expire after 60 seconds (survives crashes)
client.put_with_ttl(
    b"lock/resource".to_vec(), 
    b"owner-id".to_vec(), 
    Duration::from_secs(60)
).await?;

Features:

• Crash-safe - Uses absolute expiration time (persisted)
• Efficient cleanup - Lazy (read-time) + background task
• <1% CPU overhead - Minimal performance impact

---

4. Watch API - Real-Time Change Notifications

Monitor key changes with sub-millisecond latency:

let mut watcher = client.watch(b"config/".to_vec()).await?;

while let Some(event) = watcher.recv().await {
    match event.event_type {
        EventType::Put => println!("Updated: {:?}", event.key),
        EventType::Delete => println!("Deleted: {:?}", event.key),
    }
}

Performance:

• Lock-free - No contention under load
• <0.1ms notification latency - Instant propagation
• Tested with 1000+ concurrent watchers

Use cases: Service discovery, configuration hot-reload, cluster monitoring

---

5. StandaloneServer - Production Deployment

One-line server startup for standalone mode:

use d_engine::StandaloneServer;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (shutdown_tx, shutdown_rx) = tokio::sync::oneshot::channel();
    
    // Blocks until shutdown signal
    StandaloneServer::run(shutdown_rx).await?;
    Ok(())
}

Benefits: Automatic lifecycle management, graceful shutdown, signal handling

---

📚 New Examples

Comprehensive examples for common patterns:

• quick-start-embedded - 5-minute embedded mode setup
• quick-start-standalone - Standalone server mode
• single-node-expansion - Scale from 1→3 nodes with zero downtime
• service-discovery-embedded - Watch API with <0.1ms latency
• service-discovery-standalone - Watch API in standalone mode

---

⚠️ Breaking Changes

File-based State Machine WAL Format

The WAL format changed from relative TTL to absolute expiration timestamps.

Action required if upgrading from v0.1.x:

1. Stop all nodes
2. Follow MIGRATION_GUIDE.md
3. Restart cluster

New users: No action needed

---

📊 Performance & Quality

• 1000+ integration tests - Comprehensive edge case coverage
• Zero clippy warnings - Clean codebase across all crates
• Lock-free Watch API - Tested with 1000+ concurrent watchers
• <1% TTL overhead - Efficient background cleanup

---

📦 Installation

[dependencies]
d-engine = "0.2"

---

📚 Documentation

• Quick Start: 5-minute embedded guide
• Integration Modes: Embedded vs Standalone
• API Docs: docs.rs/d-engine
• Examples: GitHub examples/

---

🔗 Links

• Crates.io: https://crates.io/crates/d-engine
• Full Changelog: CHANGELOG.md
• Migration Guide: MIGRATION_GUIDE.md

---

🙏 Acknowledgments

Special thanks to the Rust community for valuable feedback during development.

If you find d-engine useful, ⭐ star the repo and share your use case!