Own-Cache-Database

A lightweight, Redis-inspired in-memory cache database built from scratch with Node.js, implementing the RESP protocol, TCP server communication, data persistence, and production-ready monitoring.

🎯 Project Overview

This project is a functional in-memory cache database that mimics Redis behavior, built to understand distributed systems, protocol design, and production deployment considerations. It implements:

✅ RESP Protocol - Redis Serialization Protocol for wire-compatible communication
✅ TCP Server - Native Node.js TCP server for handling client connections
✅ 13+ Commands - Redis-compatible command set (PING, SET, GET, EXPIRE, etc.)
✅ Data Persistence - Snapshot-based persistence with auto-save
✅ Prometheus Metrics - Production monitoring and observability
✅ Docker & Kubernetes - Cloud-native deployment ready
✅ Interactive CLI - User-friendly REPL client

🏗️ Architecture

System Components

┌─────────────────────────────────────────────────────────────┐
│                         Client Layer                         │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │   CLI REPL  │  │   netcat     │  │  Custom Clients  │   │
│  └──────┬──────┘  └──────┬───────┘  └────────┬─────────┘   │
└─────────┼─────────────────┼──────────────────┼──────────────┘
          │                 │                  │
          └─────────────────┴──────────────────┘
                            │ RESP Protocol over TCP
                            ▼
          ┌─────────────────────────────────────┐
          │         TCP Server (port 6379)      │
          │  ┌──────────────────────────────┐   │
          │  │   Connection Manager         │   │
          │  │   - Buffer Management        │   │
          │  │   - RESP Parser              │   │
          │  │   - Command Router           │   │
          │  └──────────────────────────────┘   │
          └─────────────┬───────────────────────┘
                        │
          ┌─────────────▼───────────────────────┐
          │      Command Handlers Layer         │
          │  ┌────────┬────────┬─────────────┐  │
          │  │ Server │  Key   │   String    │  │
          │  │  Ops   │  Ops   │    Ops      │  │
          │  └────────┴────────┴─────────────┘  │
          └─────────────┬───────────────────────┘
                        │
          ┌─────────────▼───────────────────────┐
          │        Storage Layer                │
          │  ┌────────────┐  ┌──────────────┐   │
          │  │   Store    │  │   TTL Map    │   │
          │  │  (Object)  │  │  (Timestamps)│   │
          │  └─────┬──────┘  └──────┬───────┘   │
          └────────┼─────────────────┼───────────┘
                   │                 │
          ┌────────▼─────────────────▼───────────┐
          │   Persistence (db.json)              │
          │   Auto-save every 5 minutes          │
          └──────────────────────────────────────┘

          ┌──────────────────────────────────────┐
          │   Prometheus Metrics (port 9000)     │
          │   - Active Connections               │
          │   - Commands Processed               │
          │   - Runtime Metrics                  │
          └──────────────────────────────────────┘

Core Technologies

Node.js: Event-driven, non-blocking I/O for concurrent connections
TCP Protocol: Native net module for low-latency communication
RESP: Binary protocol parsing with buffer management
Prometheus: Observability and metrics collection
Docker/K8s: Containerization and orchestration

🚀 Quick Start

Prerequisites

Node.js 18+
npm or yarn
(Optional) Docker & Docker Compose

Installation

Clone the repository:

git clone https://github.com/saitadikonda99/Own-Cache-Database.git
cd Own-Cache-Database

Install dependencies:
```
npm install
```
Start the server:
```
node src/server.js
```
Server starts on 0.0.0.0:6379 (Redis default port)

Connect with the CLI client:

npm start
# or
node src/client/client.js

Environment Variables

Configure server settings via environment variables:

export HOST=127.0.0.1
export PORT=6379
node src/server.js

📦 Commands Reference

Server Commands

Command	Description	Example	Response
`PING`	Health check	`PING`	`PONG`

Key Operations

Command	Description	Example	Response
`DEL key`	Delete a key	`DEL name`	`(integer) 1`
`EXISTS key`	Check key existence	`EXISTS name`	`1` or `0`
`KEYS pattern`	Find keys by pattern	`KEYS user:*`	List of matching keys
`TYPE key`	Get value type	`TYPE counter`	`string` or `number`
`PERSIST key`	Remove expiration	`PERSIST session`	`1` or `0`

Expiration Management

Command	Description	Example	Response
`EXPIRE key sec`	Set TTL in seconds	`EXPIRE session 3600`	`1` or `0`
`TTL key`	Get remaining TTL	`TTL session`	`3599` or `-1` (no expiry) or `-2` (not exists)

String Operations

Command	Description	Example	Response
`SET key value [EX seconds]`	Store value (with optional TTL)	`SET name "Alice" EX 60`	`OK`
`GET key`	Retrieve value	`GET name`	`"Alice"` or `(nil)`
`APPEND key value`	Append to string	`APPEND greeting "World"`	`10` (new length)
`INCR key`	Increment by 1	`INCR counter`	`1`, `2`, `3`...
`DECR key`	Decrement by 1	`DECR counter`	`0`, `-1`, `-2`...

Example Session

Own-Cache-Database:6379> PING
PONG

Own-Cache-Database:6379> SET user:1:name "John Doe" EX 300
OK

Own-Cache-Database:6379> GET user:1:name
"John Doe"

Own-Cache-Database:6379> TTL user:1:name
(integer) 295

Own-Cache-Database:6379> INCR user:1:visits
(integer) 1

Own-Cache-Database:6379> KEYS user:*
1) "user:1:name"
2) "user:1:visits"

Own-Cache-Database:6379> exit

🐳 Docker Deployment

Single Container

# Build image
docker build -t own-cache-db .

# Run server
docker run -p 6379:6379 -p 9000:9000 own-cache-db

Docker Compose (Recommended)

# Start both server and client
docker compose up -d

# Attach to interactive client
docker attach cache-client

# View logs
docker compose logs -f cache-server

# Stop services
docker compose down

The docker-compose.yml includes:

cache-server: TCP server on port 6379
cache-client: Interactive CLI client (attach to use)

☸️ Kubernetes Deployment

Full production-ready Kubernetes setup included:

# Deploy server
kubectl apply -f k8s/cache-server/

# Deploy client
kubectl apply -f k8s/cache-client/

# Deploy monitoring stack
kubectl apply -f k8s/prometheus/
kubectl apply -f k8s/grafana/

# Check status
kubectl get pods
kubectl get services

# Access server
kubectl port-forward service/cache-server 6379:6379

# View metrics
kubectl port-forward service/prometheus 9090:9090

Kubernetes Components

Cache Server: Deployment + ClusterIP Service
Cache Client: Deployment for testing
Prometheus: Metrics collection with RBAC
Grafana: Visualization dashboards

📊 Monitoring & Observability

Prometheus Metrics

Exposed on http://localhost:9000/metrics:

tcp_active_connections (Gauge) - Current connected clients
tcp_commands_processed_total (Counter) - Total commands executed
Node.js runtime metrics (memory, CPU, event loop)

Example Queries

# Request rate
rate(tcp_commands_processed_total[1m])

# Active connections
tcp_active_connections

# Memory usage
process_resident_memory_bytes

🧪 Testing

Using netcat

Send raw RESP protocol commands:

# PING command
echo -e "*1\r\n\$4\r\nPING\r\n" | nc localhost 6379

# SET command
echo -e "*3\r\n\$3\r\nSET\r\n\$4\r\nname\r\n\$5\r\nAlice\r\n" | nc localhost 6379

# GET command
echo -e "*2\r\n\$3\r\nGET\r\n\$4\r\nname\r\n" | nc localhost 6379

Using the CLI Client

npm start

🏛️ Project Structure

Own-Cache-Database/
├── src/
│   ├── server.js                    # TCP server & main entry point
│   ├── client/
│   │   └── client.js                # Interactive CLI client
│   ├── command/
│   │   └── commandHandlers.js       # Command implementations
│   └── lib/
│       ├── parser/
│       │   ├── respParser.js        # RESP protocol encoder/decoder
│       │   └── commandParser.js     # User input parser
│       ├── storage/
│       │   └── store.js             # In-memory storage
│       └── display/
│           └── responseFormatter.js # Response formatting
├── k8s/                             # Kubernetes manifests
│   ├── cache-server/                # Server deployment
│   ├── cache-client/                # Client deployment
│   ├── prometheus/                  # Metrics collection
│   └── grafana/                     # Visualization
├── docker-compose.yml               # Multi-container setup
├── Dockerfile                       # Container image
├── db.json                          # Persistence file (auto-generated)
├── package.json                     # Dependencies
└── README.md                        # Project documentation

🔬 Technical Deep Dive

RESP Protocol Implementation

The Redis Serialization Protocol (RESP) is the backbone of client-server communication:

Encoding (`encodeRESP`)

Converts JavaScript data to RESP wire format:

Strings → Bulk String: $5\r\nhello\r\n
Numbers → Integer: :42\r\n
Arrays → Array: *2\r\n$4\r\nPING\r\n
Null → Null Bulk String: $-1\r\n
Errors → Error: -ERR Unknown command\r\n

Decoding (`parseRESP`)

Parses binary buffers with state management:

// Handles partial data streams
let buffer = Buffer.alloc(0);
socket.on('data', (data) => {
    buffer = Buffer.concat([buffer, data]);
    const [command, newOffset] = parseRESP(buffer);
    if (command === null) return; // Incomplete data
    buffer = buffer.subarray(newOffset);
    handleCommand(socket, command);
});

Key Challenge: Commands may arrive in chunks (e.g., network fragmentation), requiring offset tracking and buffer management.

Data Storage Architecture

In-Memory Store

store = {
    "user:1:name": "Alice",
    "user:1:visits": 42,
    "session:abc123": "..."
}

ttl = {
    "session:abc123": 1699999999999  // Unix timestamp (ms)
}

TTL Management

Lazy Expiration: Checked on access (GET, EXISTS, etc.)
Precision: Millisecond-level timestamps
Trade-off: Expired keys remain in memory until accessed
Future Enhancement: Background cleanup task

Persistence Strategy

// Auto-save every 5 minutes
setInterval(saveToDisk, 300000);

// Snapshot format (db.json)
{
  "store": {
    "key1": "value1",
    "key2": "value2"
  }
}

Limitations:

Potential data loss window (0-5 minutes)
TTL not persisted (expires on restart)
No WAL (Write-Ahead Logging)

Concurrency Model

Event-Driven Architecture via Node.js:

┌──────────────┐
│  Client 1    │─────┐
└──────────────┘     │
┌──────────────┐     ├──► ┌─────────────────┐
│  Client 2    │─────┤    │  Event Loop     │
└──────────────┘     │    │  (Single Thread)│
┌──────────────┐     ├──► └─────────────────┘
│  Client 3    │─────┘              │
└──────────────┘                    ▼
                           ┌─────────────────┐
                           │  Store (Shared) │
                           └─────────────────┘

Benefits:

✅ Non-blocking I/O
✅ Low memory overhead
✅ Natural fit for I/O-bound workload

Limitations:

❌ CPU-bound operations block all clients
❌ No multi-core utilization (without clustering)

Error Handling Strategy

Protocol Errors: Invalid RESP format → -ERR Invalid command\r\n
Command Errors: Wrong arguments → -ERR wrong number of arguments\r\n
Type Errors: Non-integer INCR → -ERR value is not an integer\r\n
Socket Errors: Connection drops → Clean up metrics, log event

💡 Design Decisions & Trade-offs

Why Node.js?

✅ Pros:

Event-driven model perfect for I/O-bound cache operations
Native TCP support with net module
Fast prototyping with JavaScript

❌ Cons:

Single-threaded limits CPU utilization
No native support for data structures (lists, sets)

Why RESP Protocol?

✅ Pros:

Industry-standard (Redis compatibility)
Simple to parse
Human-readable for debugging

❌ Cons:

More verbose than binary protocols (e.g., MessagePack)
No built-in compression

Why Snapshot Persistence?

✅ Pros:

Simple implementation
Low I/O overhead
Predictable disk usage

❌ Cons:

Data loss window on crash
TTL not preserved across restarts
Large datasets = slow startup

Alternative Considered: Append-Only File (AOF) like Redis

Would provide better durability
Higher I/O overhead
More complex implementation

Why In-Memory Only?

✅ Pros:

O(1) lookups
Low latency
Simple code

❌ Cons:

Limited by RAM
Data lost on crash (mitigated by snapshots)
No eviction policy (LRU, LFU)

🎓 What I Learned

Technical Skills

Protocol Design
- Binary data parsing with buffers
- Handling streaming data and partial messages
- Importance of explicit length prefixes
Distributed Systems Concepts
- Cache invalidation strategies (TTL)
- Trade-offs: consistency vs. performance
- Persistence vs. durability
Network Programming
- TCP socket management
- Connection lifecycle (connect, data, error, close)
- Backpressure and flow control
Production Considerations
- Observability (metrics, logging)
- Containerization best practices
- Health checks and graceful shutdown

System Design Insights

Caching isn't just "storing data" - it's about expiration, eviction, and consistency
Protocol choice matters - RESP's simplicity enables wide adoption
Monitoring is critical - Can't optimize what you don't measure
Persistence is hard - Balancing durability, performance, and complexity

🚀 Future Enhancements

Phase 1: Core Improvements

Active key expiration (background cleanup)
Persistent TTL storage
Append-Only File (AOF) persistence
Eviction policies (LRU, LFU)

Phase 2: Advanced Features

Data structures (Lists, Sets, Sorted Sets, Hashes)
Pub/Sub messaging
Transactions (MULTI/EXEC)
Pipelining support

Phase 3: Scaling & Security

Master-slave replication
Clustering with consistent hashing
Authentication (AUTH command)
SSL/TLS support

Phase 4: Performance

Connection pooling
Rate limiting
Batch operations
Compression (LZ4/Snappy)

🎤 Interview Talking Points

Elevator Pitch

"I built a Redis-inspired cache system from scratch to deeply understand distributed systems. It implements the RESP protocol, supports 13+ commands, includes TTL-based expiration, data persistence, and is production-ready with Docker, Kubernetes, and Prometheus monitoring."

Key Achievements

Protocol Implementation: Binary RESP parser handling streaming data
Production-Ready: Metrics, containerization, K8s manifests
Full-Stack: TCP server, CLI client, persistence, monitoring
Standards-Compliant: Redis protocol compatibility

Technical Challenges Solved

Challenge 1: Partial Message Handling

// Commands can arrive in multiple chunks
// Solution: Buffer accumulation + offset tracking
let buffer = Buffer.alloc(0);
socket.on('data', (data) => {
    buffer = Buffer.concat([buffer, data]);
    // Parse with offset, keep unparsed data
});

Challenge 2: TTL Precision

// Problem: setTimeout unreliable for many keys
// Solution: Lazy expiration with timestamp comparison
if (key in ttl && ttl[key] < Date.now()) {
    delete store[key];
    delete ttl[key];
}

Challenge 3: Type Coercion

// INCR must handle string -> number conversion
const currentValue = parseInt(store[key]);
if (isNaN(currentValue)) {
    return error('value is not an integer');
}

Questions I Can Answer

"Why not just use Redis?" → Learning project to understand internals; not for production replacement
"How does it handle concurrent connections?" → Node.js event loop; non-blocking I/O; shared memory state
"What about data durability?" → Snapshot every 5 min; trade-off for simplicity; could add AOF
"How would you scale this?" → Master-slave replication, consistent hashing, sharding by key prefix
"Biggest challenge?" → RESP parsing with partial buffers; required careful offset management

📈 Performance Characteristics

Complexity

GET/SET/DEL: O(1)
**KEYS ***: O(n) - scans all keys
EXPIRE/TTL: O(1)

Limitations

Single-threaded: No parallel command execution
Memory-bound: Dataset limited by RAM
No eviction: Memory grows unbounded

Benchmarking (Recommended Tools)

# Using redis-benchmark (if RESP-compatible)
redis-benchmark -h localhost -p 6379 -t set,get -n 100000

# Custom benchmark script
for i in {1..10000}; do
    echo "SET key$i value$i" | nc localhost 6379
done

🤝 Contributing

Contributions are welcome! Here's how you can help:

Areas for Contribution

Additional Redis commands (LPUSH, SADD, HSET, etc.)
Improved persistence (AOF, RDB snapshots)
Benchmark suite and performance tests
Documentation improvements
Bug fixes and error handling

Development Setup

# Fork & clone
git clone https://github.com/YOUR_USERNAME/Own-Cache-Database.git

# Install dependencies
npm install

# Run in dev mode with auto-reload
npm run dev

# Test changes
node src/client/client.js

📄 License

This project is licensed under the ISC License - see the LICENSE file for details.

🙏 Acknowledgments

Redis: Inspiration for protocol and command design
Node.js: Powerful event-driven runtime
RESP Protocol: Simple yet effective wire protocol
Prometheus: Industry-standard monitoring

📚 Resources

Built with ❤️ to understand distributed systems

Star ⭐ this repo if you found it helpful!

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
k8s		k8s
src		src
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json

Latest commit

History

Folders and files

Repository files navigation

Own-Cache-Database

🎯 Project Overview

🏗️ Architecture

System Components

Core Technologies

🚀 Quick Start

Prerequisites

Installation

Environment Variables

📦 Commands Reference

Server Commands

Key Operations

Expiration Management

String Operations

Example Session

🐳 Docker Deployment

Single Container

Docker Compose (Recommended)

☸️ Kubernetes Deployment

Kubernetes Components

📊 Monitoring & Observability

Prometheus Metrics

Example Queries

🧪 Testing

Using netcat

Using the CLI Client

🏛️ Project Structure

🔬 Technical Deep Dive

RESP Protocol Implementation

Encoding (encodeRESP)

Decoding (parseRESP)

Data Storage Architecture

In-Memory Store

TTL Management

Persistence Strategy

Concurrency Model

Error Handling Strategy

💡 Design Decisions & Trade-offs

Why Node.js?

Why RESP Protocol?

Why Snapshot Persistence?

Why In-Memory Only?

🎓 What I Learned

Technical Skills

System Design Insights

🚀 Future Enhancements

Phase 1: Core Improvements

Phase 2: Advanced Features

Phase 3: Scaling & Security

Phase 4: Performance

🎤 Interview Talking Points

Elevator Pitch

Key Achievements

Technical Challenges Solved

Questions I Can Answer

📈 Performance Characteristics

Complexity

Limitations

Benchmarking (Recommended Tools)

🤝 Contributing

Areas for Contribution

Development Setup

📄 License

🙏 Acknowledgments

📚 Resources

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Encoding (`encodeRESP`)

Decoding (`parseRESP`)