From de25abdc1a45259e8a26b96a4e604da32613fca6 Mon Sep 17 00:00:00 2001 From: DanieCuevas <43822444+DanielCuevas1208@users.noreply.github.com> Date: Mon, 3 Aug 2026 03:27:19 -0700 Subject: [PATCH] feat: extend cinderstore --- .github/dependabot.yml | 6 ++ .github/workflows/ci.yml | 2 +- README.md | 69 ++++++++++++------ shard.yml | 2 +- spec/compaction_spec.cr | 144 +++++++++++++++++++++++++++++++++++++ src/cinderstore/db.cr | 108 ++++++++++++++++++++++------ src/cinderstore/demo.cr | 33 ++++++++- src/cinderstore/version.cr | 2 +- 8 files changed, 319 insertions(+), 47 deletions(-) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..3a626c3 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,6 @@ +version: 2 +updates: + - package-ecosystem: github-actions + directory: / + schedule: + interval: monthly diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0fdcddd..e8ca4f4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,7 +21,7 @@ jobs: uses: actions/checkout@v4 - name: Set up Crystal - uses: crystal-lang/setup-crystal@v2 + uses: crystal-lang/install-crystal@v1 with: crystal: 1.21.0 diff --git a/README.md b/README.md index 2c2aff3..8de4956 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # Cinderstore -[![CI](https://github.com///actions/workflows/ci.yml/badge.svg)](https://github.com///actions/workflows/ci.yml) - +[![CI](https://github.com/DanielCuevas1208/cinderstore/actions/workflows/ci.yml/badge.svg)](https://github.com/DanielCuevas1208/cinderstore/actions/workflows/ci.yml) Cinderstore is an embeddable key and value store. It is built on a log structured merge tree (LSM tree). It is written in Crystal and uses only the @@ -11,7 +10,8 @@ The store keeps a write ahead log for durability. It flushes memory to sorted files. It merges those files during compaction. It recovers all data after a restart. It serves `get`, `put`, and `delete` over a local socket. -This is release 0.1.0. It is the first coherent release. +This is release 0.2.0. Compaction now works by level. It rewrites only the +tables it must touch. ## Features @@ -19,7 +19,7 @@ This is release 0.1.0. It is the first coherent release. - Durable write ahead log with CRC32 framing - Sorted tables with a block index and a bloom filter - Block cache for fast repeated reads -- Background flush and compaction +- Leveled compaction that rewrites only overlapping tables - Range scans with an iterator API - Crash recovery from the write ahead log - Local TCP server with a line protocol @@ -55,7 +55,7 @@ scans, flushes, compacts, deletes, and reopens a database. The output is deterministic. ```text -== Cinderstore 0.1.0 demo == +== Cinderstore 0.2.0 demo == Loaded 24 products from ...\fixtures\catalog.csv Database directory: ...\cinderstore-demo @@ -72,15 +72,15 @@ Database directory: ...\cinderstore-demo ... (24 rows in the store) 3. Flush memtable to a sorted table - tables: 1 (l0: 1, l1: 0), entries: 24 + tables: 1 (l0: 1), entries: 24 disk bytes: 1649, memtable bytes: 0 4. Delete 4 products, update 2 products, then flush again - tables: 2 (l0: 2, l1: 0), entries: 30 + tables: 2 (l0: 2), entries: 30 disk bytes: 1902, memtable bytes: 0 5. Compact merges the tables and drops the deleted keys - tables: 1 (l0: 0, l1: 1), entries: 20 + tables: 1 (l1: 1), entries: 20 disk bytes: 1384, memtable bytes: 0 6. Verify deletes and updates after compaction @@ -91,9 +91,21 @@ Database directory: ...\cinderstore-demo 7. Reopen the database and verify recovery rows after restart: 20 +8. Flush two disjoint ranges, then compact incrementally + tables: 3 (l0: 2, l1: 1), entries: 30 + disk bytes: 2196, memtable bytes: 0 + +9. Compact cascades the merged ranges into level 2 + tables: 1 (l2: 1), entries: 30 + disk bytes: 2057, memtable bytes: 0 + Demo complete. ``` +Step 8 shows incremental compaction in action. Two new ranges wait in level 0. +The level-1 table stays in place. Its bytes are untouched. Step 9 cascades the +level-1 tables into a single level-2 table. + ## Use the library Require the library. @@ -127,7 +139,7 @@ Flush and compact explicitly. ```crystal db.flush # Move the memtable into a table. -db.compact # Merge tables and drop deleted keys. +db.compact # Merge tables and drop stale data. ``` ## Command line tool @@ -211,6 +223,10 @@ printf "PUT forge-hammer steel\nGET forge-hammer\n" | nc 127.0.0.1 7654 The database stores data in a single directory. The directory contains a manifest, one write ahead log, and sorted tables. +Tables live in levels. Level 0 receives every flush. Its tables may overlap. +Deeper levels hold non-overlapping tables. Reads scan every level and pick the +newest entry for each key. + ### Write path A write goes to two places at once. @@ -229,10 +245,17 @@ file. The old log is deleted only after the file is durable. ### Compaction -Level-0 tables may overlap. Compaction merges every table into a fresh, -non-overlapping level-1 set. The merge keeps the newest entry for each key. -It drops tombstones, because it includes all data. New writes continue into -the memory table during the merge. +Level-0 tables may overlap. Compaction merges a level into the next level. It +rewrites only the tables that overlap. Disjoint tables stay in place. + +Each merge writes fresh, non-overlapping tables. The merge keeps the newest +entry for each key. Tombstones drop only when the merge covers every older +copy. Deeper levels hold older data. New writes continue into the memory table +during the merge. + +A level starts a merge when it reaches a size threshold. Level 0 uses +`l0_compact_threshold`. Deeper levels use `l1_compact_threshold`. `max_level` +bounds how deep the cascade may go. ### Read path @@ -244,8 +267,8 @@ blocks so repeated reads avoid disk. ### Recovery On open, the database replays the write ahead log into the memory table. -Recovery is idempotent. A torn tail is detected by its CRC32 and skipped. -The manifest lists every table. Orphan files from a crash are removed. +Recovery is idempotent. The CRC32 detects a torn tail and skips it. The +manifest lists every table. The database removes orphan files from a crash. ## On-disk format @@ -271,10 +294,15 @@ config.bloom_fpp = 0.01 config.cache_blocks = 512 config.sync_writes = true config.l0_compact_threshold = 4 +config.l1_compact_threshold = 4 +config.max_level = 6 config.compact_on_flush = true db = Cinderstore::DB.new("data", config) ``` +`stats` reports a table count per level. The server returns these counts in +the `STATS` response as the `levels` array. + ## Project layout ```text @@ -289,10 +317,11 @@ spec/ Test suite ## Test status -The suite runs with `crystal spec`. It has 82 examples. All pass on Windows +The suite runs with `crystal spec`. It has 87 examples. All pass on Windows and Linux. It covers the skip list, the memory table, the write ahead log, the bloom filter, and the block cache. It covers the tables, the iterators, -and the database. It covers compaction, durability, and the server protocol. +and the database. It covers compaction, leveled cascades, durability, and the +server protocol. The CI workflow runs on GitHub Actions for Windows and Ubuntu. It checks formatting, runs the suite, and builds the binary. @@ -303,12 +332,12 @@ formatting, runs the suite, and builds the binary. - Values are limited to 4 MB. - Keys are limited to 4 KB. - The server protocol is unencrypted. Use it on localhost only. -- Compaction is a full merge. It is correct and simple, not incremental. -- No multi-threaded runtime is required. The server uses fibers. +- Compaction merges one level at a time. +- The server uses fibers. It needs no multi-threaded runtime. ## Roadmap -- Release 0.2: incremental compaction by level +- Release 0.2: leveled compaction — complete - Release 0.3: snapshot iterators and consistent reads - Release 0.4: optional checksum-free fast mode - Release 0.5: batch writes and group commit diff --git a/shard.yml b/shard.yml index c8e269b..4fc81c1 100644 --- a/shard.yml +++ b/shard.yml @@ -1,5 +1,5 @@ name: cinderstore -version: 0.1.0 +version: 0.2.0 description: An embeddable key/value store built on a log structured merge tree. crystal: ">= 1.10.0" license: Apache-2.0 diff --git a/spec/compaction_spec.cr b/spec/compaction_spec.cr index f0c3a09..f208cc1 100644 --- a/spec/compaction_spec.cr +++ b/spec/compaction_spec.cr @@ -136,4 +136,148 @@ describe "Cinderstore compaction" do db.scan.size.should eq(25_000) end end + + it "leaves disjoint level-1 tables in place during a level-0 merge" do + config = Cinderstore::SpecHelpers.fast_config + Cinderstore::SpecHelpers.with_db("compact-incremental", config) do |db, _path| + 5.times { |i| db.put("r1-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("r1-%02d" % (i + 5), "v") } + db.flush + db.compact + db.stats.l1.should eq(1) + + 5.times { |i| db.put("r2-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("r2-%02d" % (i + 5), "v") } + db.flush + db.compact + + db.stats.tables.should eq(2) + db.stats.l0.should eq(0) + db.stats.l1.should eq(2) + db.scan.size.should eq(20) + db.get("r1-03").should eq("v") + db.get("r2-07").should eq("v") + end + end + + it "cascades a full level into the next level" do + config = Cinderstore::SpecHelpers.fast_config + config.l1_compact_threshold = 2 + Cinderstore::SpecHelpers.with_db("compact-cascade", config) do |db, _path| + 5.times { |i| db.put("s1-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("s1-%02d" % (i + 5), "v") } + db.flush + db.compact + db.stats.l1.should eq(1) + + 5.times { |i| db.put("s2-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("s2-%02d" % (i + 5), "v") } + db.flush + db.compact + + db.stats.tables.should eq(1) + db.stats.l0.should eq(0) + db.stats.l1.should eq(0) + db.stats.l2.should eq(1) + db.scan.size.should eq(20) + db.get("s1-04").should eq("v") + db.get("s2-09").should eq("v") + end + end + + it "keeps tombstones while a deeper level may hold older data" do + config = Cinderstore::SpecHelpers.fast_config + config.l1_compact_threshold = 2 + Cinderstore::SpecHelpers.with_db("compact-tombstone-level", config) do |db, _path| + 2.times do |round| + prefix = "s#{round + 1}" + 5.times { |i| db.put("#{prefix}-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("#{prefix}-%02d" % (i + 5), "v") } + db.flush + db.compact + end + db.stats.l2.should eq(1) + db.stats.l1.should eq(0) + + db.put("gone", "old") + db.flush + db.delete("gone") + db.flush + db.compact + + db.get("gone").should be_nil + db.stats.l1.should eq(1) + db.stats.l2.should eq(1) + db.scan.map(&.[0]).should_not contain("gone") + end + end + + it "drops tombstones when compaction reaches the deepest level" do + config = Cinderstore::SpecHelpers.fast_config + config.l1_compact_threshold = 2 + Cinderstore::SpecHelpers.with_db("compact-tombstone-drop", config) do |db, _path| + db.put("gone", "old") + 5.times { |i| db.put("s1-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("s1-%02d" % (i + 5), "v") } + db.flush + db.compact + + 5.times { |i| db.put("s2-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("s2-%02d" % (i + 5), "v") } + db.flush + db.compact + db.stats.l2.should eq(1) + + db.delete("gone") + db.flush + 5.times { |i| db.put("j1-%02d" % i, "v") } + db.flush + db.compact + + 5.times { |i| db.put("j2-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("j2-%02d" % (i + 5), "v") } + db.flush + db.compact + + db.get("gone").should be_nil + db.stats.l1.should eq(0) + db.stats.l2.should eq(1) + db.scan.map(&.[0]).should_not contain("gone") + end + end + + it "persists tables on several levels across a restart" do + config = Cinderstore::SpecHelpers.fast_config + config.l1_compact_threshold = 2 + Cinderstore::SpecHelpers.with_db_path("compact-levels-restart") do |path| + db = Cinderstore::DB.new(path, config) + 5.times { |i| db.put("s1-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("s1-%02d" % (i + 5), "v") } + db.flush + db.compact + 5.times { |i| db.put("s2-%02d" % i, "v") } + db.flush + 5.times { |i| db.put("s2-%02d" % (i + 5), "v") } + db.flush + db.compact + db.stats.l2.should eq(1) + db.close + + reopened = Cinderstore::DB.new(path, config) + reopened.stats.levels.should eq([0, 0, 1]) + reopened.scan.size.should eq(20) + reopened.get("s1-07").should eq("v") + reopened.get("s2-03").should eq("v") + reopened.close + end + end end diff --git a/src/cinderstore/db.cr b/src/cinderstore/db.cr index 9ca6401..8b619ee 100644 --- a/src/cinderstore/db.cr +++ b/src/cinderstore/db.cr @@ -30,6 +30,10 @@ module Cinderstore property sync_writes : Bool = true # Number of level-0 tables that trigger compaction. property l0_compact_threshold : Int32 = 4 + # Number of tables in a level that trigger a merge into the next level. + property l1_compact_threshold : Int32 = 4 + # Deepest level that compaction may write into. + property max_level : Int32 = 6 # Start background compaction after a flush when true. property compact_on_flush : Bool = true end @@ -37,8 +41,12 @@ module Cinderstore # A snapshot of database counters for reporting. class Stats getter tables : Int32 + # Count of tables per level. Index zero is level 0. + getter levels : Array(Int32) getter l0 : Int32 getter l1 : Int32 + getter l2 : Int32 + getter l3 : Int32 getter entries : Int64 getter disk_bytes : Int64 getter memtable_bytes : Int64 @@ -47,16 +55,23 @@ module Cinderstore getter cache_hits : Int64 getter cache_misses : Int64 - def initialize(@tables : Int32, @l0 : Int32, @l1 : Int32, @entries : Int64, + def initialize(@tables : Int32, @levels : Array(Int32), @entries : Int64, @disk_bytes : Int64, @memtable_bytes : Int64, @wal_bytes : Int64, @seq : Int64, @cache_hits : Int64, @cache_misses : Int64) + @l0 = @levels.size > 0 ? @levels[0] : 0 + @l1 = @levels.size > 1 ? @levels[1] : 0 + @l2 = @levels.size > 2 ? @levels[2] : 0 + @l3 = @levels.size > 3 ? @levels[3] : 0 end - def to_h : Hash(String, Int32 | Int64) + def to_h : Hash(String, Int32 | Int64 | Array(Int32)) { "tables" => @tables, + "levels" => @levels, "l0" => @l0, "l1" => @l1, + "l2" => @l2, + "l3" => @l3, "entries" => @entries, "disk_bytes" => @disk_bytes, "memtable_bytes" => @memtable_bytes, @@ -72,7 +87,7 @@ module Cinderstore end def to_s(io : IO) : Nil - io << "tables: #{@tables} (l0: #{@l0}, l1: #{@l1})\n" + io << "tables: #{@tables} (levels: #{@levels})\n" io << "entries: #{@entries}\n" io << "disk bytes: #{@disk_bytes}\n" io << "memtable bytes: #{@memtable_bytes}\n" @@ -232,8 +247,7 @@ module Cinderstore end Stats.new( tables: table_refs.size, - l0: @levels[0].size, - l1: @levels.size > 1 ? @levels[1].size : 0, + levels: @levels.map(&.size), entries: entries, disk_bytes: disk_bytes, memtable_bytes: @mem.approximate_bytes, @@ -446,28 +460,80 @@ module Cinderstore # ------------------------------------------------------------------ private def do_compact : Nil - tables = nil - @lock.synchronize do - return if @closed - tables = @levels.flatten - return if tables.size < 2 + loop do + level = @lock.synchronize do + return if @closed + select_compaction_level + end + break unless level + break unless compact_level(level) + end + end + + # Returns the topmost level that needs compaction, or nil. + # + # Level 0 compacts once it holds two tables. An explicit compact merges + # the newest level-0 pair even below the background threshold. Deeper + # levels compact when they grow past `l1_compact_threshold`. + private def select_compaction_level : Int32? + l0 = @levels[0].size + return 0 if l0 >= @config.l0_compact_threshold || l0 >= 2 + (1...@levels.size).each do |i| + next if i + 1 >= @config.max_level + return i if @levels[i].size >= @config.l1_compact_threshold end + nil + end - outputs = merge_tables(tables.not_nil!) + # Merges every table in `level` with the overlapping tables of the + # next level. Outputs go to the next level. Tables that do not overlap + # the source ranges stay in place, so compaction only rewrites the part + # of the store it must touch. Returns false when there is nothing to + # merge. + private def compact_level(level : Int32) : Bool + sources, overlaps = @lock.synchronize do + return false if @closed + src = @levels[level] + tgt = @levels[level + 1]? || ([] of TableRef) + ovl = tgt.select { |ref| overlaps_any?(ref, src) } + {src, ovl} + end + + to_merge = sources + overlaps + return false if to_merge.size < 2 + + # A tombstone is safe to drop only when the merge covers every table + # that could hold an older value for its key. That holds only when + # this level feeds the deepest existing level. + drop_tombstones = (level + 1) >= @levels.size - 1 + outputs = merge_tables(to_merge, drop_tombstones) @lock.synchronize do - @levels.flatten.each(&.close) - @levels = [[] of TableRef, outputs.map { |m| TableRef.new(m.id, m.first, m.last, m.count, table_path(m.id)) }] - save_manifest - tables.not_nil!.each do |ref| - path = ref.path - File.delete(path) if File.exists?(path) + return false if @closed + while @levels.size <= level + 1 + @levels << [] of TableRef + end + to_merge.each(&.close) + @levels[level] = [] of TableRef + kept = @levels[level + 1].reject { |ref| overlaps_any?(ref, sources) } + @levels[level + 1] = kept + outputs.map { |m| TableRef.new(m.id, m.first, m.last, m.count, table_path(m.id)) } + to_merge.each do |ref| + File.delete(ref.path) if File.exists?(ref.path) end + save_manifest + true end end - # Merges every table into fresh tables and drops tombstones. - private def merge_tables(tables : Array(TableRef)) : Array(TableMeta) + # Returns true when the key ranges of `ref` and any table in `others` + # overlap. Ranges are inclusive, so touching boundaries count as an + # overlap. A conservative answer is safe: it rewrites more, never less. + private def overlaps_any?(ref : TableRef, others : Array(TableRef)) : Bool + others.any? { |other| ref.first <= other.last && other.first <= ref.last } + end + + # Merges tables into fresh tables and drops tombstones when allowed. + private def merge_tables(tables : Array(TableRef), drop_tombstones : Bool) : Array(TableMeta) readers = @lock.synchronize { tables.map { |ref| ref.reader(@block_cache) } } sources = [] of Store::Iter readers.each { |reader| sources << TableIter.new(reader) } @@ -492,7 +558,7 @@ module Cinderstore } while entry = iter.next? - next unless entry.alive + next if drop_tombstones && !entry.alive if writer.nil? current_id = @lock.synchronize do id = @next_id @@ -502,7 +568,7 @@ module Cinderstore io = File.open(File.join(@path, "#{Util.file_stem(current_id)}#{TMP_SUFFIX}"), "w") writer = SstableWriter.new(io.not_nil!, current_id, @config.block_size, @config.bloom_fpp) end - writer.not_nil!.add(entry.key, entry.value, entry.seq, true) + writer.not_nil!.add(entry.key, entry.value, entry.seq, drop_tombstones ? true : entry.alive) written += 1 finish_output.call if written >= MAX_TABLE_ENTRIES end diff --git a/src/cinderstore/demo.cr b/src/cinderstore/demo.cr index fd224fc..c08369e 100644 --- a/src/cinderstore/demo.cr +++ b/src/cinderstore/demo.cr @@ -25,6 +25,7 @@ module Cinderstore config = DB::Config.new config.sync_writes = false config.compact_on_flush = false + config.l1_compact_threshold = 2 db = DB.new(path, config) puts "== Cinderstore #{VERSION} demo ==" @@ -69,8 +70,8 @@ module Cinderstore puts "" puts "6. Verify deletes and updates after compaction" - puts " get SKU-0003 => #{db.get("SKU-0003").inspect}" - puts " get SKU-0001 => #{db.get("SKU-0001").inspect}" + puts " get SKU-0003 => #{db.get("SKU-0003") || "nil"}" + puts " get SKU-0001 => #{db.get("SKU-0001") || "nil"}" puts " scan count => #{db.scan.size}" puts "" @@ -80,6 +81,27 @@ module Cinderstore reopened = DB.new(path, config) count = reopened.scan.size puts " rows after restart: #{count}" + puts "" + + puts "8. Flush two disjoint ranges, then compact incrementally" + %w[TOOL-0001 TOOL-0002 TOOL-0003 TOOL-0004 TOOL-0005].each do |sku| + reopened.put(sku, %({"name":"Tool Rack #{sku}","price":12.50,"stock":40})) + end + reopened.flush + %w[GADG-0001 GADG-0002 GADG-0003 GADG-0004 GADG-0005].each do |sku| + reopened.put(sku, %({"name":"Gadget #{sku}","price":8.75,"stock":55})) + end + reopened.flush + print_stats(reopened) + puts " The new ranges wait in level 0. The level-1 table stays in place." + puts "" + + puts "9. Compact cascades the merged ranges into level 2" + reopened.compact + print_stats(reopened) + puts " Compaction merged every level-1 table into a single level-2 table." + puts "" + reopened.close puts "" puts "Demo complete." @@ -88,7 +110,12 @@ module Cinderstore private def print_stats(db : DB) : Nil stats = db.stats - puts " tables: #{stats.tables} (l0: #{stats.l0}, l1: #{stats.l1}), entries: #{stats.entries}" + labels = [] of String + stats.levels.each_with_index do |count, i| + labels << "l#{i}: #{count}" if count > 0 + end + label = labels.empty? ? "empty" : labels.join(", ") + puts " tables: #{stats.tables} (#{label}), entries: #{stats.entries}" puts " disk bytes: #{stats.disk_bytes}, memtable bytes: #{stats.memtable_bytes}" end diff --git a/src/cinderstore/version.cr b/src/cinderstore/version.cr index 16eaad2..f883acc 100644 --- a/src/cinderstore/version.cr +++ b/src/cinderstore/version.cr @@ -1,3 +1,3 @@ module Cinderstore - VERSION = "0.1.0" + VERSION = "0.2.0" end