Extend Cinderstore

.github/workflows/ci.yml

@@ -37,5 +37,8 @@ jobs:
       - name: Run the demo
         run: crystal run examples/demo.cr
 
+      - name: Run the server demo
+        run: crystal run examples/server_demo.cr
+
       - name: Build the binary
         run: shards build --production

README.md

@@ -13,12 +13,13 @@ restart. It serves `get`, `put`, and `delete` over a local socket.
 Snapshots give you a consistent view of the store at one instant. A snapshot
 never blocks new writes. Release it when you are done.
 
-This is release 0.3.0. It adds point-in-time snapshots and consistent reads.
+This is release 0.4.0. It adds an optional checksum-free fast mode.
 
 ## Features
 
 - Memory table with ordered writes
 - Durable write ahead log with CRC32 framing
+- Optional checksum-free fast mode
 - Sorted tables with a block index and a bloom filter
 - Block cache for fast repeated reads
 - Background flush and compaction
@@ -59,7 +60,7 @@ scans, flushes, compacts, deletes, snapshots, and reopens a database. The
 output is deterministic.
 
 ```text
-== Cinderstore 0.3.0 demo ==
+== Cinderstore 0.4.0 demo ==
 
 Loaded 24 products from ...\fixtures\catalog.csv
 Database directory: ...\cinderstore-demo
@@ -106,12 +107,24 @@ Database directory: ...\cinderstore-demo
 8. Reopen the database and verify recovery
    rows after restart: 21
 
+9. Fast mode skips checksums
+   checksummed disk bytes: 1649
+   fast disk bytes:        1645
+   fast mode saves 4 bytes on disk
+   fast mode scan count: 24
+   fast mode snapshot rows: 24
+   fast mode rows after restart: 24
+
 Demo complete.
 ```
 
 Step 7 shows the value of a snapshot. The live store drops SKU-0001 and adds
 two products. The snapshot still sees the state before those writes.
 
+Step 9 shows the fast mode. Both databases hold the same data. The fast
+table drops the CRC32 guards, so it is smaller. Reads, snapshots, and
+recovery work the same.
+
 ## Use the library
 
 Require the library.
@@ -133,6 +146,20 @@ end
 db.close
 ```
 
+### Use fast mode
+
+Set `checksums` to `false` to skip CRC32 checks.
+
+```crystal
+config = Cinderstore::DB::Config.new
+config.checksums = false
+db = Cinderstore::DB.new("data/my-store", config)
+```
+
+Fast mode writes smaller files and does less CPU work. It cannot detect torn
+or corrupt data. Use it when a little risk is acceptable. Keep the default
+when data integrity matters most.
+
 ### Read a snapshot
 
 A snapshot is a consistent view at one instant. Take a snapshot, read it,
@@ -225,6 +252,15 @@ Start the local server.
 bin/cinderstore server --db data --port 7654
 ```
 
+Run the demo without checksums.
+
+```console
+bin/cinderstore demo --no-checksums
+```
+
+The `server`, `put`, `del`, `flush`, and `compact` commands also accept
+`--no-checksums`.
+
 Run `bin/cinderstore help` for the full list of commands.
 
 ## Wire protocol
@@ -275,6 +311,19 @@ A write goes to two places at once.
 The default mode fsyncs after every write. Set `sync_writes` to `false` for
 faster, less durable writes.
 
+Every file records its own format. The log keeps a header. The table keeps a
+version in the footer. The reader detects the format, so a database can mix
+modes.
+
+### Fast mode
+
+Checksums find corruption. Each log record and each table block carries a
+CRC32. The footer carries one too. Verifying them costs CPU time.
+
+Fast mode skips every CRC32. Files are smaller and reads do less work.
+Corruption may go unnoticed. A fast file and a checked file can live in one
+database. Toggle `checksums` at any time.
+
 ### Flush
 
 When the memory table grows past its limit, the database freezes it. A new
@@ -313,8 +362,9 @@ blocks so repeated reads avoid disk.
 ### Recovery
 
 On open, the database replays the write ahead log into the memory table.
-Recovery is idempotent. A torn tail is detected by its CRC32 and skipped.
-The manifest lists every table. Orphan files from a crash are removed.
+Recovery is idempotent. A torn tail is skipped. With checksums on, the CRC32
+catches a torn tail. Without them, a short record stops the replay. The
+manifest lists every table. Orphan files from a crash are removed.
 
 ## On-disk format
 
@@ -324,7 +374,8 @@ Tables use a compact binary format.
 - A block index maps the first key of each block to its offset.
 - A bloom filter covers every key in the table.
 - A footer stores offsets, a version, and a CRC32.
-- Each block and each log record carries a CRC32.
+- Each block and each log record carries a CRC32 in the default format.
+- The fast format writes no CRC32 guards.
 
 Sequence numbers make versions unique. They are per-write and never reused.
 
@@ -339,6 +390,7 @@ config.memtable_limit = 4_i64 * 1024 * 1024
 config.bloom_fpp = 0.01
 config.cache_blocks = 512
 config.sync_writes = true
+config.checksums = true
 config.l0_compact_threshold = 4
 config.compact_on_flush = true
 db = Cinderstore::DB.new("data", config)
@@ -359,14 +411,14 @@ spec/                    Test suite
 
 ## Test status
 
-The suite runs with `crystal spec`. It has 97 examples. All pass on Windows
+The suite runs with `crystal spec`. It has 108 examples. All pass on Windows
 and Linux. It covers the skip list, the memory table, the write ahead log,
 the bloom filter, and the block cache. It covers the tables, the iterators,
-and the database. It covers compaction, durability, snapshots, and the
-server protocol.
+and the database. It covers compaction, durability, snapshots, the server
+protocol, and the fast mode.
 
 The CI workflow runs on GitHub Actions for Windows and Ubuntu. It checks
-formatting, runs the suite, runs the demo, and builds the binary.
+formatting, runs the suite, runs both demos, and builds the binary.
 
 ## Limitations
 
@@ -377,20 +429,22 @@ formatting, runs the suite, runs the demo, and builds the binary.
 - Compaction is a full merge. It is correct and simple, not incremental.
 - No multi-threaded runtime is required. The server uses fibers.
 - Release snapshots before you close the database.
+- Fast mode cannot detect corruption. It skips every CRC32 guard.
 
 ## Roadmap
 
 Planned:
 
-- Release 0.2: incremental compaction by level
-- Release 0.4: optional checksum-free fast mode
 - Release 0.5: batch writes and group commit
-- Release 0.6: secondary indexes
+- Release 0.6: incremental compaction by level
+- Release 0.7: secondary indexes
 
 Delivered:
 
 - Release 0.3: snapshot iterators and consistent reads. Snapshots give a
   stable view of the store. Compaction keeps referenced files alive.
+- Release 0.4: optional checksum-free fast mode. Every file records its
+  format. Readers detect it, so modes can mix.
 
 ## License
 

shard.lock

@@ -1 +1 @@
-version: 0.3.0
+version: 0.4.0

shard.yml

@@ -1,5 +1,5 @@
 name: cinderstore
-version: 0.3.0
+version: 0.4.0
 description: An embeddable key/value store built on a log structured merge tree.
 crystal: ">= 1.10.0"
 license: Apache-2.0

spec/fast_mode_spec.cr

@@ -0,0 +1,159 @@
+require "./spec_helper"
+
+describe "Cinderstore fast mode" do
+  it "stores and reads values without checksums" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-basic", config) do |db, _path|
+      db.put("a", "1")
+      db.put("b", "2")
+      db.get("a").should eq("1")
+      db.get("b").should eq("2")
+      db.get("c").should be_nil
+    end
+  end
+
+  it "recovers writes after a restart" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db_path("fast-restart") do |path|
+      db = Cinderstore::DB.new(path, config)
+      db.put("a", "1")
+      db.put("b", "2")
+      db.close
+
+      reopened = Cinderstore::DB.new(path, config)
+      reopened.get("a").should eq("1")
+      reopened.get("b").should eq("2")
+      reopened.close
+    end
+  end
+
+  it "flushes into a table and reads it back" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-flush", config) do |db, _path|
+      20.times { |i| db.put("k%02d" % i, "v#{i}") }
+      db.flush
+      db.stats.tables.should eq(1)
+      db.scan.size.should eq(20)
+      db.get("k10").should eq("v10")
+    end
+  end
+
+  it "compacts fast tables into a valid level-1 set" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-compact", config) do |db, _path|
+      db.put("a", "1")
+      db.flush
+      db.put("b", "2")
+      db.flush
+      db.compact
+      db.stats.l1.should eq(1)
+      db.get("a").should eq("1")
+      db.get("b").should eq("2")
+      db.scan.size.should eq(2)
+    end
+  end
+
+  it "keeps snapshots consistent in fast mode" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-snapshot", config) do |db, _path|
+      db.put("a", "old")
+      snap = db.snapshot
+      db.put("b", "new")
+      db.flush
+      snap.get("a").should eq("old")
+      snap.get("b").should be_nil
+      snap.count.should eq(1_i64)
+      db.scan.size.should eq(2)
+    end
+  end
+
+  it "writes smaller tables than checksummed mode" do
+    entries = 300.times.map { |i| Cinderstore::Entry.new("key-%04d" % i, i.to_i64, true, "value-#{i}") }.to_a
+    dir = Cinderstore::SpecHelpers.tmp_db_path("fast-size")
+    Dir.mkdir_p(dir)
+    checksummed_path = File.join(dir, "000001.sst")
+    fast_path = File.join(dir, "000002.sst")
+
+    File.open(checksummed_path, "w") do |io|
+      writer = Cinderstore::SstableWriter.new(io, 1_i64, 256, 0.01, true)
+      entries.each { |entry| writer.add(entry.key, entry.value, entry.seq, entry.alive) }
+      writer.finish
+    end
+    File.open(fast_path, "w") do |io|
+      writer = Cinderstore::SstableWriter.new(io, 2_i64, 256, 0.01, false)
+      entries.each { |entry| writer.add(entry.key, entry.value, entry.seq, entry.alive) }
+      writer.finish
+    end
+
+    File.size(fast_path).should be < File.size(checksummed_path)
+  end
+
+  it "round trips a fast table through a reader" do
+    dir = Cinderstore::SpecHelpers.tmp_db_path("fast-table")
+    Dir.mkdir_p(dir)
+    path = File.join(dir, "000001.sst")
+    entries = 300.times.map { |i| Cinderstore::Entry.new("key-%04d" % i, i.to_i64, true, "value-#{i}") }.to_a
+    File.open(path, "w") do |io|
+      writer = Cinderstore::SstableWriter.new(io, 1_i64, 128, 0.01, false)
+      entries.each { |entry| writer.add(entry.key, entry.value, entry.seq, entry.alive) }
+      writer.finish
+    end
+
+    reader = Cinderstore::SstableReader.new(path, 1_i64, nil)
+    all = [] of Cinderstore::Entry
+    reader.block_count.times { |i| all.concat(reader.load_block_entries(i)) }
+    all.should eq(entries)
+    reader.close
+  end
+
+  it "keeps the intact prefix of a torn fast log" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db_path("fast-torn") do |path|
+      db = Cinderstore::DB.new(path, config)
+      db.put("alpha", "1")
+      db.put("beta", "2")
+      db.put("gamma", "3")
+      db.close
+
+      wal = Dir.children(path).find { |n| n.ends_with?(".wal") }.not_nil!
+      Cinderstore::SpecHelpers.truncate(File.join(path, wal), File.size(File.join(path, wal)) - 3)
+
+      reopened = Cinderstore::DB.new(path, config)
+      reopened.get("alpha").should eq("1")
+      reopened.get("beta").should eq("2")
+      reopened.get("gamma").should be_nil
+      reopened.close
+    end
+  end
+
+  it "mixes fast and checksummed tables in one database" do
+    Cinderstore::SpecHelpers.with_db_path("fast-mix") do |path|
+      db = Cinderstore::DB.new(path, Cinderstore::SpecHelpers.fast_config)
+      db.put("a", "1")
+      db.put("b", "2")
+      db.flush
+      db.close
+
+      fast_config = Cinderstore::SpecHelpers.fast_config
+      fast_config.checksums = false
+      reopened = Cinderstore::DB.new(path, fast_config)
+      reopened.put("c", "3")
+      reopened.put("d", "4")
+      reopened.flush
+      reopened.scan.size.should eq(4)
+      reopened.get("a").should eq("1")
+      reopened.get("d").should eq("4")
+      reopened.close
+
+      again = Cinderstore::DB.new(path, Cinderstore::SpecHelpers.fast_config)
+      again.scan.size.should eq(4)
+      again.close
+    end
+  end
+end