d-engine Product Roadmap

[JoshuaChi]

d-engine Product Roadmap

Audience: Community contributors, early adopters
Version: v0.2 (2026-05)
Maintainer: Joshua

---

Vision

Make distributed coordination accessible — cheap to run, simple to use.

Core philosophy: Choose simple architectures over complex ones.

---

Status Overview

Version	Goal	Status
v0.1.x	Core Protocol & Basic KV	✅ Released
v0.2.x	Trusted Core	🔄 In Progress
v0.3.x	Enterprise Ready	📋 Planned
v0.4.x	Scalability (Multi-Raft)	📋 Planned

---

v0.1.x — Core Protocol & Basic KV

Released

Raft protocol working, basic cluster operations in place.

• Leader election, log replication, state machine persistence
• gRPC-based cluster communication
• Read consistency: LinearizableRead, LeaseRead, EventualConsistency
• RocksDB + file storage adaptors (RocksDB optional via feature flag)
• Learner join, promotion, auto-discovery, node removal
• Snapshot: creation, compression, log compaction
• Peer connection cache, long-lived connections
• 100K+ QPS validated in high-concurrency benchmark

---

v0.2.x — Trusted Core

Goal: Make d-engine reliable and trustworthy

Released (v0.2.1 – v0.2.4)

• Embedded mode: in-process Raft node, zero gRPC overhead (< 0.1ms latency)
• Standalone mode: one-line deployment, blocks until shutdown
• Watch API (exact key), TTL write (crash-safe), CAS
• Cluster State APIs: is_leader(), node_id(), current_term(), wait_ready()
• Drain-based batch architecture: linearizable read +92%, ReadIndex batching +440%
• Async IO Architecture: non-blocking event loop, pipelined replication, persistent
  bidirectional streams per peer
• Jepsen validated: 5 workloads + 6h soak test (kill / partition / pause) → GUARANTEES.md produced
• Snapshot: transfer correctness + stability (stale-file corruption fixed,
  apply-driven install, spawn_blocking isolation)
• Linearizable read lease: fixed Raft lease clock (SystemTime → Instant);
  leader serves reads locally when lease is valid (327K ops/s embedded)
• Watch improvements: Prefix Watch, prev_kv, progress heartbeat, buffer overflow protection
• scan_prefix: returns all keys under a prefix with revision anchor
• watch_membership: real-time membership change stream (Embedded + gRPC)
• Simpler startup: EmbeddedEngine::start(data_dir), StandaloneEngine::run(data_dir, shutdown_rx)

In Progress

• Lease/Eventual Read throughput regression (~8–10%) — root cause identified, fix in progress
• Ongoing: hot path optimization, lock-free design, architecture cleanup
• Community feedback loop: GitHub Discussions, interface scope driven by real use cases（e.g. Multi-key transactions, distributed lock）

---

v0.3.x — Enterprise Ready

Goal: d-engine is ready for enterprise POC and production workloads

Security

• TLS/mTLS, simple token authentication

Stability

• API freeze, no breaking changes
• GUARANTEES.md published
• Membership change edge cases documented and tested

Observability

• Prometheus metrics + Grafana dashboard + Runbook

Deployment

• Docker, Kubernetes, health check API, CLI

Storage

• MemFirst mode (cache workloads)
• SQLite adaptor (community)

KV Operations

• TTL query / keepalive
• Range operations
• Watch from revision

Examples

• Distributed lock, config center, distributed cache with TTL

---

v0.4.x — Scalability

Goal: Break the single-leader throughput ceiling

Prerequisite: v0.3.x milestone complete

• Multi-Raft / sharding: each shard with its own independent leader
• Shared network layer, group_id-based routing
• Linear throughput scaling: total QPS = N shards × single-shard QPS

---

Community Contribution Entry Points

Level	Areas
Beginner	Examples, docs, custom StorageEngine adaptor (SQLite, sled)
Mid	TTL management, CLI tooling, non-Rust SDKs
Advanced	Jepsen workloads, snapshot stress testing, membership edge cases, hot path optimization

---

Notes

• Expect frequent updates as real production use cases inform priorities
• Community feedback loop: GitHub Discussions drives interface scope