Extend Cinderstore

.github/dependabot.yml

@@ -0,0 +1,6 @@
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: monthly

.github/workflows/ci.yml

@@ -21,7 +21,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Set up Crystal
-        uses: crystal-lang/setup-crystal@v2
+        uses: crystal-lang/install-crystal@v1
         with:
           crystal: 1.21.0
 

README.md

@@ -1,7 +1,6 @@
 # Cinderstore
 
-[![CI](https://github.com/<owner>/<repo>/actions/workflows/ci.yml/badge.svg)](https://github.com/<owner>/<repo>/actions/workflows/ci.yml)
-<!-- Replace <owner>/<repo> with your repository path. -->
+[![CI](https://github.com/DanielCuevas1208/cinderstore/actions/workflows/ci.yml/badge.svg)](https://github.com/DanielCuevas1208/cinderstore/actions/workflows/ci.yml)
 
 Cinderstore is an embeddable key and value store. It is built on a log
 structured merge tree (LSM tree). It is written in Crystal and uses only the
@@ -11,15 +10,16 @@ The store keeps a write ahead log for durability. It flushes memory to sorted
 files. It merges those files during compaction. It recovers all data after a
 restart. It serves `get`, `put`, and `delete` over a local socket.
 
-This is release 0.1.0. It is the first coherent release.
+This is release 0.2.0. Compaction now works by level. It rewrites only the
+tables it must touch.
 
 ## Features
 
 - Memory table with ordered writes
 - Durable write ahead log with CRC32 framing
 - Sorted tables with a block index and a bloom filter
 - Block cache for fast repeated reads
-- Background flush and compaction
+- Leveled compaction that rewrites only overlapping tables
 - Range scans with an iterator API
 - Crash recovery from the write ahead log
 - Local TCP server with a line protocol
@@ -55,7 +55,7 @@ scans, flushes, compacts, deletes, and reopens a database. The output is
 deterministic.
 
 ```text
-== Cinderstore 0.1.0 demo ==
+== Cinderstore 0.2.0 demo ==
 
 Loaded 24 products from ...\fixtures\catalog.csv
 Database directory: ...\cinderstore-demo
@@ -72,15 +72,15 @@ Database directory: ...\cinderstore-demo
    ... (24 rows in the store)
 
 3. Flush memtable to a sorted table
-   tables: 1 (l0: 1, l1: 0), entries: 24
+   tables: 1 (l0: 1), entries: 24
    disk bytes: 1649, memtable bytes: 0
 
 4. Delete 4 products, update 2 products, then flush again
-   tables: 2 (l0: 2, l1: 0), entries: 30
+   tables: 2 (l0: 2), entries: 30
    disk bytes: 1902, memtable bytes: 0
 
 5. Compact merges the tables and drops the deleted keys
-   tables: 1 (l0: 0, l1: 1), entries: 20
+   tables: 1 (l1: 1), entries: 20
    disk bytes: 1384, memtable bytes: 0
 
 6. Verify deletes and updates after compaction
@@ -91,9 +91,21 @@ Database directory: ...\cinderstore-demo
 7. Reopen the database and verify recovery
    rows after restart: 20
 
+8. Flush two disjoint ranges, then compact incrementally
+   tables: 3 (l0: 2, l1: 1), entries: 30
+   disk bytes: 2196, memtable bytes: 0
+
+9. Compact cascades the merged ranges into level 2
+   tables: 1 (l2: 1), entries: 30
+   disk bytes: 2057, memtable bytes: 0
+
 Demo complete.
 ```
 
+Step 8 shows incremental compaction in action. Two new ranges wait in level 0.
+The level-1 table stays in place. Its bytes are untouched. Step 9 cascades the
+level-1 tables into a single level-2 table.
+
 ## Use the library
 
 Require the library.
@@ -127,7 +139,7 @@ Flush and compact explicitly.
 
 ```crystal
 db.flush    # Move the memtable into a table.
-db.compact  # Merge tables and drop deleted keys.
+db.compact  # Merge tables and drop stale data.
 ```
 
 ## Command line tool
@@ -211,6 +223,10 @@ printf "PUT forge-hammer steel\nGET forge-hammer\n" | nc 127.0.0.1 7654
 The database stores data in a single directory. The directory contains a
 manifest, one write ahead log, and sorted tables.
 
+Tables live in levels. Level 0 receives every flush. Its tables may overlap.
+Deeper levels hold non-overlapping tables. Reads scan every level and pick the
+newest entry for each key.
+
 ### Write path
 
 A write goes to two places at once.
@@ -229,10 +245,17 @@ file. The old log is deleted only after the file is durable.
 
 ### Compaction
 
-Level-0 tables may overlap. Compaction merges every table into a fresh,
-non-overlapping level-1 set. The merge keeps the newest entry for each key.
-It drops tombstones, because it includes all data. New writes continue into
-the memory table during the merge.
+Level-0 tables may overlap. Compaction merges a level into the next level. It
+rewrites only the tables that overlap. Disjoint tables stay in place.
+
+Each merge writes fresh, non-overlapping tables. The merge keeps the newest
+entry for each key. Tombstones drop only when the merge covers every older
+copy. Deeper levels hold older data. New writes continue into the memory table
+during the merge.
+
+A level starts a merge when it reaches a size threshold. Level 0 uses
+`l0_compact_threshold`. Deeper levels use `l1_compact_threshold`. `max_level`
+bounds how deep the cascade may go.
 
 ### Read path
 
@@ -244,8 +267,8 @@ blocks so repeated reads avoid disk.
 ### Recovery
 
 On open, the database replays the write ahead log into the memory table.
-Recovery is idempotent. A torn tail is detected by its CRC32 and skipped.
-The manifest lists every table. Orphan files from a crash are removed.
+Recovery is idempotent. The CRC32 detects a torn tail and skips it. The
+manifest lists every table. The database removes orphan files from a crash.
 
 ## On-disk format
 
@@ -271,10 +294,15 @@ config.bloom_fpp = 0.01
 config.cache_blocks = 512
 config.sync_writes = true
 config.l0_compact_threshold = 4
+config.l1_compact_threshold = 4
+config.max_level = 6
 config.compact_on_flush = true
 db = Cinderstore::DB.new("data", config)
 ```
 
+`stats` reports a table count per level. The server returns these counts in
+the `STATS` response as the `levels` array.
+
 ## Project layout
 
 ```text
@@ -289,10 +317,11 @@ spec/                    Test suite
 
 ## Test status
 
-The suite runs with `crystal spec`. It has 82 examples. All pass on Windows
+The suite runs with `crystal spec`. It has 87 examples. All pass on Windows
 and Linux. It covers the skip list, the memory table, the write ahead log,
 the bloom filter, and the block cache. It covers the tables, the iterators,
-and the database. It covers compaction, durability, and the server protocol.
+and the database. It covers compaction, leveled cascades, durability, and the
+server protocol.
 
 The CI workflow runs on GitHub Actions for Windows and Ubuntu. It checks
 formatting, runs the suite, and builds the binary.
@@ -303,12 +332,12 @@ formatting, runs the suite, and builds the binary.
 - Values are limited to 4 MB.
 - Keys are limited to 4 KB.
 - The server protocol is unencrypted. Use it on localhost only.
-- Compaction is a full merge. It is correct and simple, not incremental.
-- No multi-threaded runtime is required. The server uses fibers.
+- Compaction merges one level at a time.
+- The server uses fibers. It needs no multi-threaded runtime.
 
 ## Roadmap
 
-- Release 0.2: incremental compaction by level
+- Release 0.2: leveled compaction — complete
 - Release 0.3: snapshot iterators and consistent reads
 - Release 0.4: optional checksum-free fast mode
 - Release 0.5: batch writes and group commit

shard.yml

@@ -1,5 +1,5 @@
 name: cinderstore
-version: 0.1.0
+version: 0.2.0
 description: An embeddable key/value store built on a log structured merge tree.
 crystal: ">= 1.10.0"
 license: Apache-2.0

spec/compaction_spec.cr

@@ -136,4 +136,148 @@ describe "Cinderstore compaction" do
       db.scan.size.should eq(25_000)
     end
   end
+
+  it "leaves disjoint level-1 tables in place during a level-0 merge" do
+    config = Cinderstore::SpecHelpers.fast_config
+    Cinderstore::SpecHelpers.with_db("compact-incremental", config) do |db, _path|
+      5.times { |i| db.put("r1-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("r1-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+      db.stats.l1.should eq(1)
+
+      5.times { |i| db.put("r2-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("r2-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+
+      db.stats.tables.should eq(2)
+      db.stats.l0.should eq(0)
+      db.stats.l1.should eq(2)
+      db.scan.size.should eq(20)
+      db.get("r1-03").should eq("v")
+      db.get("r2-07").should eq("v")
+    end
+  end
+
+  it "cascades a full level into the next level" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.l1_compact_threshold = 2
+    Cinderstore::SpecHelpers.with_db("compact-cascade", config) do |db, _path|
+      5.times { |i| db.put("s1-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("s1-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+      db.stats.l1.should eq(1)
+
+      5.times { |i| db.put("s2-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("s2-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+
+      db.stats.tables.should eq(1)
+      db.stats.l0.should eq(0)
+      db.stats.l1.should eq(0)
+      db.stats.l2.should eq(1)
+      db.scan.size.should eq(20)
+      db.get("s1-04").should eq("v")
+      db.get("s2-09").should eq("v")
+    end
+  end
+
+  it "keeps tombstones while a deeper level may hold older data" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.l1_compact_threshold = 2
+    Cinderstore::SpecHelpers.with_db("compact-tombstone-level", config) do |db, _path|
+      2.times do |round|
+        prefix = "s#{round + 1}"
+        5.times { |i| db.put("#{prefix}-%02d" % i, "v") }
+        db.flush
+        5.times { |i| db.put("#{prefix}-%02d" % (i + 5), "v") }
+        db.flush
+        db.compact
+      end
+      db.stats.l2.should eq(1)
+      db.stats.l1.should eq(0)
+
+      db.put("gone", "old")
+      db.flush
+      db.delete("gone")
+      db.flush
+      db.compact
+
+      db.get("gone").should be_nil
+      db.stats.l1.should eq(1)
+      db.stats.l2.should eq(1)
+      db.scan.map(&.[0]).should_not contain("gone")
+    end
+  end
+
+  it "drops tombstones when compaction reaches the deepest level" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.l1_compact_threshold = 2
+    Cinderstore::SpecHelpers.with_db("compact-tombstone-drop", config) do |db, _path|
+      db.put("gone", "old")
+      5.times { |i| db.put("s1-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("s1-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+
+      5.times { |i| db.put("s2-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("s2-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+      db.stats.l2.should eq(1)
+
+      db.delete("gone")
+      db.flush
+      5.times { |i| db.put("j1-%02d" % i, "v") }
+      db.flush
+      db.compact
+
+      5.times { |i| db.put("j2-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("j2-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+
+      db.get("gone").should be_nil
+      db.stats.l1.should eq(0)
+      db.stats.l2.should eq(1)
+      db.scan.map(&.[0]).should_not contain("gone")
+    end
+  end
+
+  it "persists tables on several levels across a restart" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.l1_compact_threshold = 2
+    Cinderstore::SpecHelpers.with_db_path("compact-levels-restart") do |path|
+      db = Cinderstore::DB.new(path, config)
+      5.times { |i| db.put("s1-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("s1-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+      5.times { |i| db.put("s2-%02d" % i, "v") }
+      db.flush
+      5.times { |i| db.put("s2-%02d" % (i + 5), "v") }
+      db.flush
+      db.compact
+      db.stats.l2.should eq(1)
+      db.close
+
+      reopened = Cinderstore::DB.new(path, config)
+      reopened.stats.levels.should eq([0, 0, 1])
+      reopened.scan.size.should eq(20)
+      reopened.get("s1-07").should eq("v")
+      reopened.get("s2-03").should eq("v")
+      reopened.close
+    end
+  end
 end