Extend Cinderstore

.github/workflows/ci.yml

@@ -4,10 +4,16 @@ on:
   push:
     branches: [main]
   pull_request:
+  schedule:
+    - cron: "17 3 * * *"
 
 permissions:
   contents: read
 
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   test:
     name: Test on ${{ matrix.os }}

README.md

@@ -13,7 +13,8 @@ restart. It serves `get`, `put`, and `delete` over a local socket.
 Snapshots give you a consistent view of the store at one instant. A snapshot
 never blocks new writes. Release it when you are done.
 
-This is release 0.3.0. It adds point-in-time snapshots and consistent reads.
+This is release 0.4.0. It adds leveled compaction. Compaction now moves one
+level at a time instead of rewriting every table.
 
 ## Features
 
@@ -22,6 +23,8 @@ This is release 0.3.0. It adds point-in-time snapshots and consistent reads.
 - Sorted tables with a block index and a bloom filter
 - Block cache for fast repeated reads
 - Background flush and compaction
+- Leveled compaction that moves one level at a time
+- Tombstones that drop only at the deepest level
 - Point-in-time snapshots with consistent reads
 - Snapshot iterators that stay valid during writes
 - Range scans with an iterator API
@@ -59,7 +62,7 @@ scans, flushes, compacts, deletes, snapshots, and reopens a database. The
 output is deterministic.
 
 ```text
-== Cinderstore 0.3.0 demo ==
+== Cinderstore 0.4.0 demo ==
 
 Loaded 24 products from ...\fixtures\catalog.csv
 Database directory: ...\cinderstore-demo
@@ -103,15 +106,27 @@ Database directory: ...\cinderstore-demo
    snapshot iterator SKU-0010 to SKU-0016
    SKU-0010, SKU-0011, SKU-0013, SKU-0014, SKU-0015
 
-8. Reopen the database and verify recovery
-   rows after restart: 21
+8. Leveled compaction grows deeper levels
+   Write 90 products across 3 flushes
+   tables: 5 (l0: 4, l1: 1), entries: 113
+   compact moved level 0 into level 1
+   tables: 1 (l0: 0, l1: 1), entries: 111
+   level 1 passed its target, so compact moved it to level 2
+   tables: 1 (l0: 0, l1: 0, l2: 1), entries: 111
+
+9. Reopen the database and verify recovery
+   rows after restart: 111
 
 Demo complete.
 ```
 
 Step 7 shows the value of a snapshot. The live store drops SKU-0001 and adds
 two products. The snapshot still sees the state before those writes.
 
+Step 8 shows leveled compaction. Three flushes pile up level 0. One compact
+merges them into level 1. The level passes its target. The next compact moves
+it to level 2.
+
 ## Use the library
 
 Require the library.
@@ -181,7 +196,7 @@ Flush and compact explicitly.
 
 ```crystal
 db.flush    # Move the memtable into a table.
-db.compact  # Merge tables and drop deleted keys.
+db.compact  # Merge one level into the level below.
 ```
 
 ## Command line tool
@@ -283,13 +298,24 @@ file. The old log is deleted only after the file is durable.
 
 ### Compaction
 
-Level-0 tables may overlap. Compaction merges every table into a fresh,
-non-overlapping level-1 set. The merge keeps the newest entry for each key.
-It drops tombstones, because it includes all data. New writes continue into
-the memory table during the merge.
+Compaction works one level at a time. Level zero holds recent flushes. Its
+tables may overlap. Every deeper level holds tables that do not overlap.
+
+A compact moves one level into the level below. It merges the source tables
+with the tables below that overlap them. The merge keeps the newest entry
+for each key. It writes the result to the level below.
 
-Compaction keeps a table file on disk while a snapshot references it. The
-file is deleted only after the last snapshot releases it.
+Level zero compacts when it holds several tables. A deeper level compacts
+when its size passes the level target. The level target grows by a fixed
+ratio at each level.
+
+A tombstone flows down through the levels. It is dropped only when it
+reaches the deepest level. Deeper tables can hold older values. An early
+drop could resurrect a deleted key.
+
+New writes continue into the memory table during the merge. Compaction keeps
+a table file on disk while a snapshot references it. The file is deleted
+only after the last snapshot releases it.
 
 ### Snapshots
 
@@ -341,14 +367,22 @@ config.cache_blocks = 512
 config.sync_writes = true
 config.l0_compact_threshold = 4
 config.compact_on_flush = true
+config.level_target_bytes = 8_i64 * 1024 * 1024
+config.level_ratio = 10
+config.max_levels = 7
 db = Cinderstore::DB.new("data", config)
 ```
 
+`level_target_bytes` sets the size that triggers a level-one compact. Every
+deeper level has a target that is `level_ratio` times larger. `max_levels`
+caps the total number of levels.
+
 ## Project layout
 
 ```text
 src/cinderstore.cr       Library entry point
 src/cinderstore/         Core components
+src/cinderstore/db.cr    Database and leveled compaction
 src/cinderstore/snapshot.cr  Point-in-time snapshot support
 src/cli.cr               Command line tool
 examples/demo.cr         Library walkthrough
@@ -359,11 +393,11 @@ spec/                    Test suite
 
 ## Test status
 
-The suite runs with `crystal spec`. It has 97 examples. All pass on Windows
+The suite runs with `crystal spec`. It has 106 examples. All pass on Windows
 and Linux. It covers the skip list, the memory table, the write ahead log,
 the bloom filter, and the block cache. It covers the tables, the iterators,
-and the database. It covers compaction, durability, snapshots, and the
-server protocol.
+and the database. It covers durability, snapshots, and the server protocol.
+It covers leveled compaction across several levels.
 
 The CI workflow runs on GitHub Actions for Windows and Ubuntu. It checks
 formatting, runs the suite, runs the demo, and builds the binary.
@@ -374,23 +408,25 @@ formatting, runs the suite, runs the demo, and builds the binary.
 - Values are limited to 4 MB.
 - Keys are limited to 4 KB.
 - The server protocol is unencrypted. Use it on localhost only.
-- Compaction is a full merge. It is correct and simple, not incremental.
+- Compaction rewrites whole levels, not single tables.
 - No multi-threaded runtime is required. The server uses fibers.
 - Release snapshots before you close the database.
 
 ## Roadmap
 
 Planned:
 
-- Release 0.2: incremental compaction by level
-- Release 0.4: optional checksum-free fast mode
-- Release 0.5: batch writes and group commit
-- Release 0.6: secondary indexes
+- Release 0.5: optional checksum-free fast mode
+- Release 0.6: batch writes and group commit
+- Release 0.7: secondary indexes
 
 Delivered:
 
 - Release 0.3: snapshot iterators and consistent reads. Snapshots give a
   stable view of the store. Compaction keeps referenced files alive.
+- Release 0.4: leveled compaction. Compaction moves one level at a time.
+  A level compacts when it passes its size target. Tombstones drop at the
+  deepest level.
 
 ## License
 

shard.lock

@@ -1 +1 @@
-version: 0.3.0
+version: 0.4.0

shard.yml

@@ -1,5 +1,5 @@
 name: cinderstore
-version: 0.3.0
+version: 0.4.0
 description: An embeddable key/value store built on a log structured merge tree.
 crystal: ">= 1.10.0"
 license: Apache-2.0

spec/compaction_spec.cr

@@ -103,7 +103,7 @@ describe "Cinderstore compaction" do
     end
   end
 
-  it "handles many small flushes and a full merge" do
+  it "handles many small flushes and compacts them into one table" do
     config = Cinderstore::SpecHelpers.fast_config
     Cinderstore::SpecHelpers.with_db("compact-many", config) do |db, _path|
       6.times do |round|

spec/levels_spec.cr

@@ -0,0 +1,172 @@
+require "./spec_helper"
+
+# A config with a tiny level-one target so tests can grow levels fast.
+private def tiny_level_config(level_target_bytes : Int64 = 4_096)
+  config = Cinderstore::SpecHelpers.fast_config
+  config.level_target_bytes = level_target_bytes
+  config
+end
+
+# Writes `count` keys with enough bytes to exceed a 4 KB level target.
+private def put_batch(db : Cinderstore::DB, prefix : String, count : Int32) : Nil
+  count.times do |i|
+    db.put("#{prefix}-%03d" % i, "v#{i}-" + "x" * 40)
+  end
+end
+
+describe "Cinderstore leveled compaction" do
+  it "moves a lone flushed table down one level" do
+    config = Cinderstore::SpecHelpers.fast_config
+    Cinderstore::SpecHelpers.with_db("levels-move", config) do |db, _path|
+      db.put("a", "1")
+      db.put("b", "2")
+      db.flush
+      db.stats.levels.should eq([1])
+      db.compact
+      db.stats.levels.should eq([0, 1])
+      db.get("a").should eq("1")
+      db.get("b").should eq("2")
+    end
+  end
+
+  it "keeps tables in a level disjoint" do
+    config = Cinderstore::SpecHelpers.fast_config
+    Cinderstore::SpecHelpers.with_db("levels-disjoint", config) do |db, _path|
+      db.put("a", "1")
+      db.put("m", "2")
+      db.flush
+      db.compact
+      db.put("n", "3")
+      db.put("z", "4")
+      db.flush
+      db.compact
+      db.stats.levels.should eq([0, 2])
+      db.scan.map(&.[0]).should eq(%w[a m n z])
+    end
+  end
+
+  it "preserves level-one tables that do not overlap a compact" do
+    config = Cinderstore::SpecHelpers.fast_config
+    Cinderstore::SpecHelpers.with_db("levels-preserve", config) do |db, _path|
+      db.put("a", "1")
+      db.put("m", "2")
+      db.flush
+      db.compact
+      db.put("n", "3")
+      db.put("z", "4")
+      db.flush
+      db.compact
+      db.put("a", "updated")
+      db.flush
+      db.compact
+      db.stats.levels.should eq([0, 2])
+      db.get("a").should eq("updated")
+      db.scan.map(&.[1]).should eq(%w[updated 2 3 4])
+    end
+  end
+
+  it "compacts an overflowing level one into level two" do
+    config = tiny_level_config
+    Cinderstore::SpecHelpers.with_db("levels-grow", config) do |db, _path|
+      put_batch(db, "key", 120)
+      db.flush
+      db.compact
+      db.stats.levels.should eq([0, 1])
+      db.compact
+      db.stats.levels.should eq([0, 0, 1])
+      db.get("key-040").should eq("v40-" + "x" * 40)
+    end
+  end
+
+  it "keeps a tombstone while older data lives below" do
+    config = tiny_level_config
+    Cinderstore::SpecHelpers.with_db("levels-keep", config) do |db, _path|
+      put_batch(db, "k", 120)
+      db.flush
+      db.compact
+      db.compact
+      db.stats.levels.should eq([0, 0, 1])
+
+      db.delete("k-040")
+      db.flush
+      put_batch(db, "j", 120)
+      db.flush
+      db.compact
+      db.stats.levels.should eq([0, 1, 1])
+      db.get("k-040").should be_nil
+    end
+  end
+
+  it "drops a tombstone once it reaches the deepest level" do
+    config = tiny_level_config
+    Cinderstore::SpecHelpers.with_db("levels-drop", config) do |db, _path|
+      put_batch(db, "k", 120)
+      db.flush
+      db.compact
+      db.compact
+      db.delete("k-040")
+      db.flush
+      put_batch(db, "j", 120)
+      db.flush
+      db.compact
+      db.stats.levels.should eq([0, 1, 1])
+
+      db.compact
+      db.stats.levels.should eq([0, 0, 1])
+      db.get("k-040").should be_nil
+      db.scan.size.should eq(239)
+    end
+  end
+
+  it "reads the newest value across three levels" do
+    config = tiny_level_config
+    Cinderstore::SpecHelpers.with_db("levels-newest", config) do |db, _path|
+      put_batch(db, "k", 120)
+      db.flush
+      db.compact
+      db.compact
+      db.put("k-040", "newest")
+      db.flush
+      db.compact
+      db.get("k-040").should eq("newest")
+      db.scan.size.should eq(120)
+    end
+  end
+
+  it "recovers a multi-level store after restart" do
+    config = tiny_level_config
+    Cinderstore::SpecHelpers.with_db_path("levels-restart") do |path|
+      db = Cinderstore::DB.new(path, config)
+      put_batch(db, "k", 120)
+      db.flush
+      db.compact
+      db.compact
+      db.put("j-100", "later")
+      db.flush
+      db.compact
+      db.stats.levels.should eq([0, 1, 1])
+      db.close
+
+      reopened = Cinderstore::DB.new(path, config)
+      reopened.get("k-040").should eq("v40-" + "x" * 40)
+      reopened.get("j-100").should eq("later")
+      reopened.scan.size.should eq(121)
+      reopened.close
+    end
+  end
+
+  it "reports per level table counts in stats" do
+    config = tiny_level_config
+    Cinderstore::SpecHelpers.with_db("levels-stats", config) do |db, _path|
+      put_batch(db, "k", 120)
+      db.flush
+      db.compact
+      db.compact
+      stats = db.stats
+      stats.levels.should eq([0, 0, 1])
+      stats.tables.should eq(1)
+      stats.l0.should eq(0)
+      stats.to_json.should contain("levels")
+    end
+  end
+end