Extend Cinderstore

README.md

@@ -6,14 +6,14 @@ Cinderstore is an embeddable key and value store. It is built on a log
 structured merge tree (LSM tree). It is written in Crystal and uses only the
 Crystal standard library.
 
-The store keeps a write ahead log for durability. It flushes memory to sorted
-files. It merges those files during compaction. It recovers all data after a
-restart. It serves `get`, `put`, and `delete` over a local socket.
+The store keeps a write ahead log for durability. It flushes memory to
+sorted files. It merges those files during compaction. It recovers all data
+after a restart. It serves `get`, `put`, and `delete` over a local socket.
 
 Snapshots give you a consistent view of the store at one instant. A snapshot
 never blocks new writes. Release it when you are done.
 
-This is release 0.3.0. It adds point-in-time snapshots and consistent reads.
+This is release 0.4.0. It adds an optional checksum-free fast mode.
 
 ## Features
 
@@ -27,11 +27,12 @@ This is release 0.3.0. It adds point-in-time snapshots and consistent reads.
 - Range scans with an iterator API
 - Crash recovery from the write ahead log
 - Local TCP server with a line protocol
+- Optional checksum-free fast mode
 - Zero runtime dependencies
 
 ## Quick start
 
-You need Crystal 1.10 or newer.
+Install Crystal 1.10 or newer.
 
 ```console
 crystal spec
@@ -59,7 +60,7 @@ scans, flushes, compacts, deletes, snapshots, and reopens a database. The
 output is deterministic.
 
 ```text
-== Cinderstore 0.3.0 demo ==
+== Cinderstore 0.4.0 demo ==
 
 Loaded 24 products from ...\fixtures\catalog.csv
 Database directory: ...\cinderstore-demo
@@ -106,12 +107,23 @@ Database directory: ...\cinderstore-demo
 8. Reopen the database and verify recovery
    rows after restart: 21
 
+9. Compare the checksum modes on 400 synthetic products
+   checksummed rows: 400, disk bytes: 53573
+   checksum-free rows: 400, disk bytes: 53521
+   fast mode saves 52 bytes and skips CRC32 work
+
+10. Reopen the checksum-free database and verify recovery
+   rows after restart: 400
+
 Demo complete.
 ```
 
 Step 7 shows the value of a snapshot. The live store drops SKU-0001 and adds
 two products. The snapshot still sees the state before those writes.
 
+Steps 9 and 10 show the checksum-free fast mode. The fast store holds the
+same rows in fewer bytes. It reads and recovers correctly.
+
 ## Use the library
 
 Require the library.
@@ -184,6 +196,19 @@ db.flush    # Move the memtable into a table.
 db.compact  # Merge tables and drop deleted keys.
 ```
 
+### Fast mode
+
+Disable checksums in the configuration.
+
+```crystal
+config = Cinderstore::DB::Config.new
+config.checksums = false
+db = Cinderstore::DB.new("data/my-store", config)
+```
+
+Fast mode skips the CRC32 work. It trades integrity checking for speed. Each
+file records its own mode. Recovery always reads files correctly.
+
 ## Command line tool
 
 The tool uses a database directory. The default directory is
@@ -225,6 +250,12 @@ Start the local server.
 bin/cinderstore server --db data --port 7654
 ```
 
+Disable checksums for faster writes.
+
+```console
+bin/cinderstore put --key forge-hammer --value steel --no-checksums
+```
+
 Run `bin/cinderstore help` for the full list of commands.
 
 ## Wire protocol
@@ -291,14 +322,23 @@ the memory table during the merge.
 Compaction keeps a table file on disk while a snapshot references it. The
 file is deleted only after the last snapshot releases it.
 
+### Checksum modes
+
+Checksums detect corruption in the write ahead log and in table blocks. The
+default mode writes a CRC32 after every log record and table block.
+
+Fast mode skips the CRC32 work. It writes fewer bytes. Each file records its
+own mode in its header. Recovery reads that header first. This keeps mixed
+configurations safe across restarts.
+
 ### Snapshots
 
 A snapshot is an immutable view of the store at one instant. Creation copies
 the active memory table and takes a reference to each table file.
 
 Writes, flushes, and compactions after the snapshot do not change what the
-snapshot sees. The snapshot reads the table files it holds, so compaction
-can replace those files without breaking the snapshot.
+snapshot sees. The snapshot reads the table files it holds. Compaction can
+replace those files without breaking the snapshot.
 
 Snapshots never block writes. Reads over a snapshot use the same merge path
 as normal reads. Release a snapshot when you are done with it.
@@ -326,7 +366,9 @@ Tables use a compact binary format.
 - A footer stores offsets, a version, and a CRC32.
 - Each block and each log record carries a CRC32.
 
-Sequence numbers make versions unique. They are per-write and never reused.
+In fast mode, the per-block and per-record checksums are omitted. The footer
+and the WAL header keep their integrity markers. Sequence numbers make
+versions unique. They are per-write and never reused.
 
 ## Configuration
 
@@ -341,6 +383,7 @@ config.cache_blocks = 512
 config.sync_writes = true
 config.l0_compact_threshold = 4
 config.compact_on_flush = true
+config.checksums = true
 db = Cinderstore::DB.new("data", config)
 ```
 
@@ -359,11 +402,12 @@ spec/                    Test suite
 
 ## Test status
 
-The suite runs with `crystal spec`. It has 97 examples. All pass on Windows
+The suite runs with `crystal spec`. It has 110 examples. All pass on Windows
 and Linux. It covers the skip list, the memory table, the write ahead log,
 the bloom filter, and the block cache. It covers the tables, the iterators,
 and the database. It covers compaction, durability, snapshots, and the
-server protocol.
+server protocol. It covers the checksum-free fast mode and mixed-format
+recovery.
 
 The CI workflow runs on GitHub Actions for Windows and Ubuntu. It checks
 formatting, runs the suite, runs the demo, and builds the binary.
@@ -377,21 +421,23 @@ formatting, runs the suite, runs the demo, and builds the binary.
 - Compaction is a full merge. It is correct and simple, not incremental.
 - No multi-threaded runtime is required. The server uses fibers.
 - Release snapshots before you close the database.
+- Fast mode skips block and record checksums. Corruption goes undetected.
 
 ## Roadmap
 
+Delivered:
+
+- Release 0.3: snapshot iterators and consistent reads. Snapshots give a
+  stable view of the store. Compaction keeps referenced files alive.
+- Release 0.4: optional checksum-free fast mode. Tables and logs skip
+  per-record CRC32. Recovery reads each file's mode from its header.
+
 Planned:
 
 - Release 0.2: incremental compaction by level
-- Release 0.4: optional checksum-free fast mode
 - Release 0.5: batch writes and group commit
 - Release 0.6: secondary indexes
 
-Delivered:
-
-- Release 0.3: snapshot iterators and consistent reads. Snapshots give a
-  stable view of the store. Compaction keeps referenced files alive.
-
 ## License
 
 Cinderstore is licensed under the Apache License 2.0. See the `LICENSE` file.

shard.lock

@@ -1 +1 @@
-version: 0.3.0
+version: 0.4.0

shard.yml

@@ -1,5 +1,5 @@
 name: cinderstore
-version: 0.3.0
+version: 0.4.0
 description: An embeddable key/value store built on a log structured merge tree.
 crystal: ">= 1.10.0"
 license: Apache-2.0

spec/fast_mode_spec.cr

@@ -0,0 +1,189 @@
+require "./spec_helper"
+
+describe "Cinderstore checksum-free fast mode" do
+  it "round trips values with checksums disabled" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-basic", config) do |db, _path|
+      db.put("alpha", "one")
+      db.put("beta", "two")
+      db.delete("beta")
+      db.put("beta", "three")
+      db.get("alpha").should eq("one")
+      db.get("beta").should eq("three")
+      db.scan.map(&.[0]).should eq(%w[alpha beta])
+    end
+  end
+
+  it "flushes and compacts with checksums disabled" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-flush", config) do |db, _path|
+      6.times do |round|
+        20.times { |i| db.put("k%03d" % (round * 20 + i), "r#{round}") }
+        db.flush
+      end
+      db.compact
+      db.stats.tables.should eq(1)
+      db.stats.l1.should eq(1)
+      db.scan.size.should eq(120)
+      db.get("k099").should eq("r4")
+    end
+  end
+
+  it "recovers a checksum-free database after a restart" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db_path("fast-restart") do |path|
+      db = Cinderstore::DB.new(path, config)
+      db.put("a", "1")
+      db.put("b", "2")
+      db.delete("a")
+      db.flush
+      db.close
+
+      reopened = Cinderstore::DB.new(path, config)
+      reopened.get("a").should be_nil
+      reopened.get("b").should eq("2")
+      reopened.stats.tables.should eq(1)
+      reopened.scan.map(&.[0]).should eq(%w[b])
+      reopened.close
+    end
+  end
+
+  it "recovers a checksummed database with checksums disabled" do
+    config = Cinderstore::SpecHelpers.fast_config
+    Cinderstore::SpecHelpers.with_db_path("fast-mix-on-off") do |path|
+      db = Cinderstore::DB.new(path, config)
+      db.put("a", "1")
+      db.put("b", "2")
+      db.flush
+      db.close
+
+      fast = Cinderstore::SpecHelpers.fast_config
+      fast.checksums = false
+      reopened = Cinderstore::DB.new(path, fast)
+      reopened.get("a").should eq("1")
+      reopened.get("b").should eq("2")
+      reopened.scan.map(&.[0]).should eq(%w[a b])
+      reopened.close
+    end
+  end
+
+  it "recovers a checksum-free database with checksums enabled" do
+    fast = Cinderstore::SpecHelpers.fast_config
+    fast.checksums = false
+    Cinderstore::SpecHelpers.with_db_path("fast-mix-off-on") do |path|
+      db = Cinderstore::DB.new(path, fast)
+      db.put("a", "1")
+      db.put("b", "2")
+      db.flush
+      db.close
+
+      checked = Cinderstore::SpecHelpers.fast_config
+      reopened = Cinderstore::DB.new(path, checked)
+      reopened.get("a").should eq("1")
+      reopened.get("b").should eq("2")
+      reopened.stats.tables.should eq(1)
+      reopened.scan.map(&.[0]).should eq(%w[a b])
+      reopened.close
+    end
+  end
+
+  it "serves a snapshot on a checksum-free database" do
+    config = Cinderstore::SpecHelpers.fast_config
+    config.checksums = false
+    Cinderstore::SpecHelpers.with_db("fast-snapshot", config) do |db, _path|
+      db.put("a", "1")
+      db.flush
+      snap = db.snapshot
+      db.put("b", "2")
+      db.flush
+      snap.get("a").should eq("1")
+      snap.get("b").should be_nil
+      snap.scan.map(&.[0]).should eq(%w[a])
+      snap.release
+    end
+  end
+
+  it "reads a mixed set of checksummed and fast tables" do
+    checked = Cinderstore::SpecHelpers.fast_config
+    fast = Cinderstore::SpecHelpers.fast_config
+    fast.checksums = false
+    Cinderstore::SpecHelpers.with_db_path("fast-mixed-tables") do |path|
+      db = Cinderstore::DB.new(path, checked)
+      db.put("a", "checked")
+      db.flush
+      db.close
+
+      other = Cinderstore::DB.new(path, fast)
+      other.put("b", "fast")
+      other.flush
+      other.get("a").should eq("checked")
+      other.get("b").should eq("fast")
+      other.scan.map(&.[0]).should eq(%w[a b])
+      other.close
+    end
+  end
+
+  it "stores fast tables with fewer bytes than checksummed tables" do
+    dir = Cinderstore::SpecHelpers.tmp_db_path("fast-size")
+    Dir.mkdir_p(dir)
+    entries = 300.times.map { |i| Cinderstore::Entry.new("key-%04d" % i, i.to_i64, true, "value-" + "x" * 40) }.to_a
+
+    checked_path = File.join(dir, "000001.sst")
+    File.open(checked_path, "w") do |io|
+      writer = Cinderstore::SstableWriter.new(io, 1_i64, 128, 0.01, true)
+      entries.each { |e| writer.add(e.key, e.value, e.seq, e.alive) }
+      writer.finish
+    end
+
+    fast_path = File.join(dir, "000002.sst")
+    File.open(fast_path, "w") do |io|
+      writer = Cinderstore::SstableWriter.new(io, 2_i64, 128, 0.01, false)
+      entries.each { |e| writer.add(e.key, e.value, e.seq, e.alive) }
+      writer.finish
+    end
+
+    File.size(checked_path).should be > File.size(fast_path)
+
+    reader = Cinderstore::SstableReader.new(fast_path, 2_i64, nil)
+    reader.block_count.should be > 1
+    read_all_entries(reader).should eq(entries)
+    reader.close
+  end
+
+  it "leaves fast mode block corruption undetected by design" do
+    dir = Cinderstore::SpecHelpers.tmp_db_path("fast-corrupt")
+    Dir.mkdir_p(dir)
+    path = File.join(dir, "000003.sst")
+    entries = 100.times.map { |i| Cinderstore::Entry.new("key-%03d" % i, i.to_i64, true, "value-#{i}") }.to_a
+    File.open(path, "w") do |io|
+      writer = Cinderstore::SstableWriter.new(io, 3_i64, 64, 0.01, false)
+      entries.each { |e| writer.add(e.key, e.value, e.seq, e.alive) }
+      writer.finish
+    end
+
+    # Flip a byte inside the first block payload. Fast mode skips CRC32
+    # verification, so the reader must not raise. The structural length
+    # check still passes because the block size is unchanged.
+    File.open(path, "r+") do |f|
+      f.pos = 16
+      byte = f.read_byte.not_nil!
+      f.pos = 16
+      f.write_byte((byte ^ 0xFF).to_u8)
+    end
+
+    reader = Cinderstore::SstableReader.new(path, 3_i64, nil)
+    reader.load_block_entries(0)
+    reader.close
+  end
+end
+
+def read_all_entries(reader : Cinderstore::SstableReader)
+  all = [] of Cinderstore::Entry
+  reader.block_count.times do |i|
+    all.concat(reader.load_block_entries(i))
+  end
+  all
+end