diff --git a/README.md b/README.md index d9f2522..e4b6c55 100644 --- a/README.md +++ b/README.md @@ -6,14 +6,14 @@ Cinderstore is an embeddable key and value store. It is built on a log structured merge tree (LSM tree). It is written in Crystal and uses only the Crystal standard library. -The store keeps a write ahead log for durability. It flushes memory to sorted -files. It merges those files during compaction. It recovers all data after a -restart. It serves `get`, `put`, and `delete` over a local socket. +The store keeps a write ahead log for durability. It flushes memory to +sorted files. It merges those files during compaction. It recovers all data +after a restart. It serves `get`, `put`, and `delete` over a local socket. Snapshots give you a consistent view of the store at one instant. A snapshot never blocks new writes. Release it when you are done. -This is release 0.3.0. It adds point-in-time snapshots and consistent reads. +This is release 0.4.0. It adds an optional checksum-free fast mode. ## Features @@ -27,11 +27,12 @@ This is release 0.3.0. It adds point-in-time snapshots and consistent reads. - Range scans with an iterator API - Crash recovery from the write ahead log - Local TCP server with a line protocol +- Optional checksum-free fast mode - Zero runtime dependencies ## Quick start -You need Crystal 1.10 or newer. +Install Crystal 1.10 or newer. ```console crystal spec @@ -59,7 +60,7 @@ scans, flushes, compacts, deletes, snapshots, and reopens a database. The output is deterministic. ```text -== Cinderstore 0.3.0 demo == +== Cinderstore 0.4.0 demo == Loaded 24 products from ...\fixtures\catalog.csv Database directory: ...\cinderstore-demo @@ -106,12 +107,23 @@ Database directory: ...\cinderstore-demo 8. Reopen the database and verify recovery rows after restart: 21 +9. Compare the checksum modes on 400 synthetic products + checksummed rows: 400, disk bytes: 53573 + checksum-free rows: 400, disk bytes: 53521 + fast mode saves 52 bytes and skips CRC32 work + +10. Reopen the checksum-free database and verify recovery + rows after restart: 400 + Demo complete. ``` Step 7 shows the value of a snapshot. The live store drops SKU-0001 and adds two products. The snapshot still sees the state before those writes. +Steps 9 and 10 show the checksum-free fast mode. The fast store holds the +same rows in fewer bytes. It reads and recovers correctly. + ## Use the library Require the library. @@ -184,6 +196,19 @@ db.flush # Move the memtable into a table. db.compact # Merge tables and drop deleted keys. ``` +### Fast mode + +Disable checksums in the configuration. + +```crystal +config = Cinderstore::DB::Config.new +config.checksums = false +db = Cinderstore::DB.new("data/my-store", config) +``` + +Fast mode skips the CRC32 work. It trades integrity checking for speed. Each +file records its own mode. Recovery always reads files correctly. + ## Command line tool The tool uses a database directory. The default directory is @@ -225,6 +250,12 @@ Start the local server. bin/cinderstore server --db data --port 7654 ``` +Disable checksums for faster writes. + +```console +bin/cinderstore put --key forge-hammer --value steel --no-checksums +``` + Run `bin/cinderstore help` for the full list of commands. ## Wire protocol @@ -291,14 +322,23 @@ the memory table during the merge. Compaction keeps a table file on disk while a snapshot references it. The file is deleted only after the last snapshot releases it. +### Checksum modes + +Checksums detect corruption in the write ahead log and in table blocks. The +default mode writes a CRC32 after every log record and table block. + +Fast mode skips the CRC32 work. It writes fewer bytes. Each file records its +own mode in its header. Recovery reads that header first. This keeps mixed +configurations safe across restarts. + ### Snapshots A snapshot is an immutable view of the store at one instant. Creation copies the active memory table and takes a reference to each table file. Writes, flushes, and compactions after the snapshot do not change what the -snapshot sees. The snapshot reads the table files it holds, so compaction -can replace those files without breaking the snapshot. +snapshot sees. The snapshot reads the table files it holds. Compaction can +replace those files without breaking the snapshot. Snapshots never block writes. Reads over a snapshot use the same merge path as normal reads. Release a snapshot when you are done with it. @@ -326,7 +366,9 @@ Tables use a compact binary format. - A footer stores offsets, a version, and a CRC32. - Each block and each log record carries a CRC32. -Sequence numbers make versions unique. They are per-write and never reused. +In fast mode, the per-block and per-record checksums are omitted. The footer +and the WAL header keep their integrity markers. Sequence numbers make +versions unique. They are per-write and never reused. ## Configuration @@ -341,6 +383,7 @@ config.cache_blocks = 512 config.sync_writes = true config.l0_compact_threshold = 4 config.compact_on_flush = true +config.checksums = true db = Cinderstore::DB.new("data", config) ``` @@ -359,11 +402,12 @@ spec/ Test suite ## Test status -The suite runs with `crystal spec`. It has 97 examples. All pass on Windows +The suite runs with `crystal spec`. It has 110 examples. All pass on Windows and Linux. It covers the skip list, the memory table, the write ahead log, the bloom filter, and the block cache. It covers the tables, the iterators, and the database. It covers compaction, durability, snapshots, and the -server protocol. +server protocol. It covers the checksum-free fast mode and mixed-format +recovery. The CI workflow runs on GitHub Actions for Windows and Ubuntu. It checks formatting, runs the suite, runs the demo, and builds the binary. @@ -377,21 +421,23 @@ formatting, runs the suite, runs the demo, and builds the binary. - Compaction is a full merge. It is correct and simple, not incremental. - No multi-threaded runtime is required. The server uses fibers. - Release snapshots before you close the database. +- Fast mode skips block and record checksums. Corruption goes undetected. ## Roadmap +Delivered: + +- Release 0.3: snapshot iterators and consistent reads. Snapshots give a + stable view of the store. Compaction keeps referenced files alive. +- Release 0.4: optional checksum-free fast mode. Tables and logs skip + per-record CRC32. Recovery reads each file's mode from its header. + Planned: - Release 0.2: incremental compaction by level -- Release 0.4: optional checksum-free fast mode - Release 0.5: batch writes and group commit - Release 0.6: secondary indexes -Delivered: - -- Release 0.3: snapshot iterators and consistent reads. Snapshots give a - stable view of the store. Compaction keeps referenced files alive. - ## License Cinderstore is licensed under the Apache License 2.0. See the `LICENSE` file. diff --git a/shard.lock b/shard.lock index c1dbcd2..8a8a3ae 100644 --- a/shard.lock +++ b/shard.lock @@ -1 +1 @@ -version: 0.3.0 +version: 0.4.0 diff --git a/shard.yml b/shard.yml index e335eb5..5225a79 100644 --- a/shard.yml +++ b/shard.yml @@ -1,5 +1,5 @@ name: cinderstore -version: 0.3.0 +version: 0.4.0 description: An embeddable key/value store built on a log structured merge tree. crystal: ">= 1.10.0" license: Apache-2.0 diff --git a/spec/fast_mode_spec.cr b/spec/fast_mode_spec.cr new file mode 100644 index 0000000..da3093f --- /dev/null +++ b/spec/fast_mode_spec.cr @@ -0,0 +1,189 @@ +require "./spec_helper" + +describe "Cinderstore checksum-free fast mode" do + it "round trips values with checksums disabled" do + config = Cinderstore::SpecHelpers.fast_config + config.checksums = false + Cinderstore::SpecHelpers.with_db("fast-basic", config) do |db, _path| + db.put("alpha", "one") + db.put("beta", "two") + db.delete("beta") + db.put("beta", "three") + db.get("alpha").should eq("one") + db.get("beta").should eq("three") + db.scan.map(&.[0]).should eq(%w[alpha beta]) + end + end + + it "flushes and compacts with checksums disabled" do + config = Cinderstore::SpecHelpers.fast_config + config.checksums = false + Cinderstore::SpecHelpers.with_db("fast-flush", config) do |db, _path| + 6.times do |round| + 20.times { |i| db.put("k%03d" % (round * 20 + i), "r#{round}") } + db.flush + end + db.compact + db.stats.tables.should eq(1) + db.stats.l1.should eq(1) + db.scan.size.should eq(120) + db.get("k099").should eq("r4") + end + end + + it "recovers a checksum-free database after a restart" do + config = Cinderstore::SpecHelpers.fast_config + config.checksums = false + Cinderstore::SpecHelpers.with_db_path("fast-restart") do |path| + db = Cinderstore::DB.new(path, config) + db.put("a", "1") + db.put("b", "2") + db.delete("a") + db.flush + db.close + + reopened = Cinderstore::DB.new(path, config) + reopened.get("a").should be_nil + reopened.get("b").should eq("2") + reopened.stats.tables.should eq(1) + reopened.scan.map(&.[0]).should eq(%w[b]) + reopened.close + end + end + + it "recovers a checksummed database with checksums disabled" do + config = Cinderstore::SpecHelpers.fast_config + Cinderstore::SpecHelpers.with_db_path("fast-mix-on-off") do |path| + db = Cinderstore::DB.new(path, config) + db.put("a", "1") + db.put("b", "2") + db.flush + db.close + + fast = Cinderstore::SpecHelpers.fast_config + fast.checksums = false + reopened = Cinderstore::DB.new(path, fast) + reopened.get("a").should eq("1") + reopened.get("b").should eq("2") + reopened.scan.map(&.[0]).should eq(%w[a b]) + reopened.close + end + end + + it "recovers a checksum-free database with checksums enabled" do + fast = Cinderstore::SpecHelpers.fast_config + fast.checksums = false + Cinderstore::SpecHelpers.with_db_path("fast-mix-off-on") do |path| + db = Cinderstore::DB.new(path, fast) + db.put("a", "1") + db.put("b", "2") + db.flush + db.close + + checked = Cinderstore::SpecHelpers.fast_config + reopened = Cinderstore::DB.new(path, checked) + reopened.get("a").should eq("1") + reopened.get("b").should eq("2") + reopened.stats.tables.should eq(1) + reopened.scan.map(&.[0]).should eq(%w[a b]) + reopened.close + end + end + + it "serves a snapshot on a checksum-free database" do + config = Cinderstore::SpecHelpers.fast_config + config.checksums = false + Cinderstore::SpecHelpers.with_db("fast-snapshot", config) do |db, _path| + db.put("a", "1") + db.flush + snap = db.snapshot + db.put("b", "2") + db.flush + snap.get("a").should eq("1") + snap.get("b").should be_nil + snap.scan.map(&.[0]).should eq(%w[a]) + snap.release + end + end + + it "reads a mixed set of checksummed and fast tables" do + checked = Cinderstore::SpecHelpers.fast_config + fast = Cinderstore::SpecHelpers.fast_config + fast.checksums = false + Cinderstore::SpecHelpers.with_db_path("fast-mixed-tables") do |path| + db = Cinderstore::DB.new(path, checked) + db.put("a", "checked") + db.flush + db.close + + other = Cinderstore::DB.new(path, fast) + other.put("b", "fast") + other.flush + other.get("a").should eq("checked") + other.get("b").should eq("fast") + other.scan.map(&.[0]).should eq(%w[a b]) + other.close + end + end + + it "stores fast tables with fewer bytes than checksummed tables" do + dir = Cinderstore::SpecHelpers.tmp_db_path("fast-size") + Dir.mkdir_p(dir) + entries = 300.times.map { |i| Cinderstore::Entry.new("key-%04d" % i, i.to_i64, true, "value-" + "x" * 40) }.to_a + + checked_path = File.join(dir, "000001.sst") + File.open(checked_path, "w") do |io| + writer = Cinderstore::SstableWriter.new(io, 1_i64, 128, 0.01, true) + entries.each { |e| writer.add(e.key, e.value, e.seq, e.alive) } + writer.finish + end + + fast_path = File.join(dir, "000002.sst") + File.open(fast_path, "w") do |io| + writer = Cinderstore::SstableWriter.new(io, 2_i64, 128, 0.01, false) + entries.each { |e| writer.add(e.key, e.value, e.seq, e.alive) } + writer.finish + end + + File.size(checked_path).should be > File.size(fast_path) + + reader = Cinderstore::SstableReader.new(fast_path, 2_i64, nil) + reader.block_count.should be > 1 + read_all_entries(reader).should eq(entries) + reader.close + end + + it "leaves fast mode block corruption undetected by design" do + dir = Cinderstore::SpecHelpers.tmp_db_path("fast-corrupt") + Dir.mkdir_p(dir) + path = File.join(dir, "000003.sst") + entries = 100.times.map { |i| Cinderstore::Entry.new("key-%03d" % i, i.to_i64, true, "value-#{i}") }.to_a + File.open(path, "w") do |io| + writer = Cinderstore::SstableWriter.new(io, 3_i64, 64, 0.01, false) + entries.each { |e| writer.add(e.key, e.value, e.seq, e.alive) } + writer.finish + end + + # Flip a byte inside the first block payload. Fast mode skips CRC32 + # verification, so the reader must not raise. The structural length + # check still passes because the block size is unchanged. + File.open(path, "r+") do |f| + f.pos = 16 + byte = f.read_byte.not_nil! + f.pos = 16 + f.write_byte((byte ^ 0xFF).to_u8) + end + + reader = Cinderstore::SstableReader.new(path, 3_i64, nil) + reader.load_block_entries(0) + reader.close + end +end + +def read_all_entries(reader : Cinderstore::SstableReader) + all = [] of Cinderstore::Entry + reader.block_count.times do |i| + all.concat(reader.load_block_entries(i)) + end + all +end diff --git a/spec/wal_spec.cr b/spec/wal_spec.cr index 3860506..179f866 100644 --- a/spec/wal_spec.cr +++ b/spec/wal_spec.cr @@ -52,9 +52,10 @@ describe Cinderstore::Wal do writer.append(Cinderstore::Entry.new("alpha", 1_i64, true, "1")) writer.close - # Flip a byte inside the record body. + # Flip a byte inside the record body. The first nine bytes are the + # header, so the payload starts at offset ten. bytes = File.read(path).to_slice.dup - bytes[1] = (bytes[1] ^ 0xFF).to_u8 + bytes[11] = (bytes[11] ^ 0xFF).to_u8 File.open(path, "w") { |f| f.write(bytes) } mem = Cinderstore::MemTable.new @@ -72,4 +73,65 @@ describe Cinderstore::Wal do reopened.close end end + + it "replays a checksum-free log" do + path = File.join(Cinderstore::SpecHelpers.tmp_db_path("wal-fast"), "000001.wal") + Dir.mkdir_p(File.dirname(path)) + writer = Cinderstore::Wal::Writer.new(path, false, false) + writer.append(Cinderstore::Entry.new("a", 1_i64, true, "one")) + writer.append(Cinderstore::Entry.new("b", 2_i64, false, "")) + writer.close + + mem = Cinderstore::MemTable.new + seq = Cinderstore::Wal.recover(path, mem, 0_i64) + seq.should eq(2_i64) + mem.get("a").should eq("one") + mem.get("b").should be_nil + mem.get_entry("b").not_nil!.alive.should be_false + end + + it "stops at a torn tail in a checksum-free log" do + dir = Cinderstore::SpecHelpers.tmp_db_path("wal-fast") + Dir.mkdir_p(dir) + path = File.join(dir, "000001.wal") + writer = Cinderstore::Wal::Writer.new(path, false, false) + writer.append(Cinderstore::Entry.new("alpha", 1_i64, true, "1")) + writer.append(Cinderstore::Entry.new("beta", 2_i64, true, "2")) + writer.append(Cinderstore::Entry.new("gamma", 3_i64, true, "3")) + writer.close + + Cinderstore::SpecHelpers.truncate(path, File.size(path) - 3) + + mem = Cinderstore::MemTable.new + seq = Cinderstore::Wal.recover(path, mem, 0_i64) + seq.should eq(2_i64) + mem.get("alpha").should eq("1") + mem.get("beta").should eq("2") + mem.get("gamma").should be_nil + end + + it "reads a legacy log without a header as checksummed" do + path = File.join(Cinderstore::SpecHelpers.tmp_db_path("wal-legacy"), "000001.wal") + Dir.mkdir_p(File.dirname(path)) + File.open(path, "w") do |io| + io.write(Cinderstore::Wal.encode(Cinderstore::Entry.new("old", 5_i64, true, "v"))) + end + + mem = Cinderstore::MemTable.new + seq = Cinderstore::Wal.recover(path, mem, 0_i64) + seq.should eq(5_i64) + mem.get("old").should eq("v") + end + + it "does not misread a checksum-free log as a legacy one" do + path = File.join(Cinderstore::SpecHelpers.tmp_db_path("wal-header"), "000001.wal") + Dir.mkdir_p(File.dirname(path)) + writer = Cinderstore::Wal::Writer.new(path, false, false) + writer.append(Cinderstore::Entry.new("fast", 3_i64, true, "v")) + writer.close + + mem = Cinderstore::MemTable.new + Cinderstore::Wal.recover(path, mem, 0_i64) + mem.get("fast").should eq("v") + end end diff --git a/src/cinderstore/db.cr b/src/cinderstore/db.cr index f5707f9..8738997 100644 --- a/src/cinderstore/db.cr +++ b/src/cinderstore/db.cr @@ -32,6 +32,12 @@ module Cinderstore property l0_compact_threshold : Int32 = 4 # Start background compaction after a flush when true. property compact_on_flush : Bool = true + # Write CRC32 checksums when true. + # + # Checksums detect torn writes and corrupted blocks. Disable them for + # faster writes and reads. The footer and the WAL header always keep + # their integrity markers, so recovery stays correct in both modes. + property checksums : Bool = true end # A snapshot of database counters for reporting. @@ -374,7 +380,7 @@ module Cinderstore private def create_active_wal : Nil id = @next_id @next_id += 1 - @wal = Wal::Writer.new(wal_path(id), @config.sync_writes) + @wal = Wal::Writer.new(wal_path(id), @config.sync_writes, @config.checksums) end # ------------------------------------------------------------------ @@ -429,7 +435,7 @@ module Cinderstore @next_id += 1 wal_id = @next_id @next_id += 1 - @wal = Wal::Writer.new(wal_path(wal_id), @config.sync_writes) + @wal = Wal::Writer.new(wal_path(wal_id), @config.sync_writes, @config.checksums) end end @@ -459,7 +465,7 @@ module Cinderstore final_path = table_path(table_id) meta = nil File.open(tmp_path, "w") do |io| - writer = SstableWriter.new(io, table_id, @config.block_size, @config.bloom_fpp) + writer = SstableWriter.new(io, table_id, @config.block_size, @config.bloom_fpp, @config.checksums) mem.each_entry do |entry| writer.add(entry.key, entry.value, entry.seq, entry.alive) end @@ -557,7 +563,7 @@ module Cinderstore id end io = File.open(File.join(@path, "#{Util.file_stem(current_id)}#{TMP_SUFFIX}"), "w") - writer = SstableWriter.new(io.not_nil!, current_id, @config.block_size, @config.bloom_fpp) + writer = SstableWriter.new(io.not_nil!, current_id, @config.block_size, @config.bloom_fpp, @config.checksums) end writer.not_nil!.add(entry.key, entry.value, entry.seq, true) written += 1 diff --git a/src/cinderstore/demo.cr b/src/cinderstore/demo.cr index 206635f..7872d20 100644 --- a/src/cinderstore/demo.cr +++ b/src/cinderstore/demo.cr @@ -4,13 +4,14 @@ module Cinderstore # A self-contained walkthrough of the store. # # The demo loads a small product catalog, writes it, scans a range, - # flushes, compacts, and verifies recovery. Its output is deterministic. + # flushes, compacts, and verifies recovery. It then compares the two + # checksum modes on the same data. Its output is deterministic. class Demo - def self.run(db_path : String? = nil, fixture : String? = nil) : Int32 - new(db_path, fixture).run + def self.run(db_path : String? = nil, fixture : String? = nil, checksums : Bool = true) : Int32 + new(db_path, fixture, checksums).run end - def initialize(@db_path : String? = nil, @fixture : String? = nil) + def initialize(@db_path : String? = nil, @fixture : String? = nil, @checksums : Bool = true) end def run : Int32 @@ -25,6 +26,7 @@ module Cinderstore config = DB::Config.new config.sync_writes = false config.compact_on_flush = false + config.checksums = @checksums db = DB.new(path, config) puts "== Cinderstore #{VERSION} demo ==" @@ -103,6 +105,45 @@ module Cinderstore puts " rows after restart: #{count}" reopened.close puts "" + + puts "9. Compare the checksum modes on 400 synthetic products" + fast_path = "#{path}-fast" + checked_path = "#{path}-checked" + checked_config = DB::Config.new + checked_config.sync_writes = false + checked_config.compact_on_flush = false + fast_config = DB::Config.new + fast_config.sync_writes = false + fast_config.compact_on_flush = false + fast_config.checksums = false + checked_db = DB.new(checked_path, checked_config) + fast_db = DB.new(fast_path, fast_config) + synthetic_rows.each do |key, value| + checked_db.put(key, value) + fast_db.put(key, value) + end + checked_db.flush + checked_db.compact + fast_db.flush + fast_db.compact + checked_bytes = checked_db.stats.disk_bytes + fast_bytes = fast_db.stats.disk_bytes + puts " checksummed rows: #{checked_db.scan.size}, disk bytes: #{checked_bytes}" + puts " checksum-free rows: #{fast_db.scan.size}, disk bytes: #{fast_bytes}" + puts " fast mode saves #{checked_bytes - fast_bytes} bytes and skips CRC32 work" + checked_db.close + fast_db.close + puts "" + + puts "10. Reopen the checksum-free database and verify recovery" + reopened_fast = DB.new(fast_path, fast_config) + puts " rows after restart: #{reopened_fast.scan.size}" + reopened_fast.close + puts "" + + FileUtils.rm_rf(fast_path) + FileUtils.rm_rf(checked_path) + puts "Demo complete." 0 end @@ -136,5 +177,16 @@ module Cinderstore end rows end + + # Returns deterministic products spread across several table blocks. + private def synthetic_rows : Array(Tuple(String, String)) + Array.new(400) do |i| + key = "SKU-F%04d" % i + value = %({"name":"Fixture #{i}","price":#{i % 900},"stock":#{i % 50},"note":"#{NOTE_PAD}"}) + {key, value} + end + end + + NOTE_PAD = "x" * 64 end end diff --git a/src/cinderstore/table.cr b/src/cinderstore/table.cr index bcd8bd3..1a697c5 100644 --- a/src/cinderstore/table.cr +++ b/src/cinderstore/table.cr @@ -29,20 +29,27 @@ module Cinderstore # [bloom filter] # [footer] # - # A data block holds the block length, a batch of serialized entries, - # and a CRC32. The block index maps the first key of each block to its - # file offset and total length. The bloom filter lets a reader skip a - # table that cannot contain a key. The footer stores offsets and a CRC32 - # over the entire footer. + # A data block holds the block length and a batch of serialized entries. + # With checksums enabled it also holds a CRC32. The block index maps the + # first key of each block to its file offset and total length. The bloom + # filter lets a reader skip a table that cannot contain a key. The footer + # stores offsets, a version, and a CRC32 over the entire footer. + # + # Version 1 writes a CRC32 after every data block. Version 2 omits those + # checksums. Both versions keep the footer CRC32. The reader chooses the + # layout from the version in the footer, so one database can mix tables of + # both kinds. class SstableWriter - MAGIC = 0x43494E4445525F31_u64 - VERSION = 1_u32 - FOOTER_SIZE = 48 + MAGIC = 0x43494E4445525F31_u64 + VERSION = 1_u32 + VERSION_FAST = 2_u32 + FOOTER_SIZE = 48 @io : IO @id : Int64 @block_size : Int32 @fpp : Float64 + @checksums : Bool @pending : IO::Memory @pending_keys : Array(String) @keys : Array(String) @@ -51,7 +58,8 @@ module Cinderstore @first : String? @last : String? - def initialize(io : IO, @id : Int64, @block_size : Int32 = 4096, @fpp : Float64 = 0.01) + def initialize(io : IO, @id : Int64, @block_size : Int32 = 4096, + @fpp : Float64 = 0.01, @checksums : Bool = true) @io = io @pending = IO::Memory.new @pending_keys = [] of String @@ -100,7 +108,7 @@ module Cinderstore footer.write_bytes(index_length.to_u64, IO::ByteFormat::LittleEndian) footer.write_bytes(bloom_offset.to_u64, IO::ByteFormat::LittleEndian) footer.write_bytes(bloom_length.to_u64, IO::ByteFormat::LittleEndian) - footer.write_bytes(VERSION, IO::ByteFormat::LittleEndian) + footer.write_bytes(@checksums ? VERSION : VERSION_FAST, IO::ByteFormat::LittleEndian) footer_bytes = footer.to_slice @io.write(footer_bytes) @io.write_bytes(Util.crc32(footer_bytes), IO::ByteFormat::LittleEndian) @@ -114,8 +122,11 @@ module Cinderstore offset = @io.pos Util.write_varint(@io, data.size.to_u64) @io.write(data) - @io.write_bytes(Util.crc32(data), IO::ByteFormat::LittleEndian) - total = Util.varint_len(data.size.to_u64) + data.size + 4 + total = Util.varint_len(data.size.to_u64) + data.size + if @checksums + @io.write_bytes(Util.crc32(data), IO::ByteFormat::LittleEndian) + total += 4 + end @index << BlockIndexEntry.new(@pending_keys.first, offset.to_u64, total.to_u64) @pending = IO::Memory.new @pending_keys = [] of String @@ -136,6 +147,7 @@ module Cinderstore @index_length : UInt64 @bloom_offset : UInt64 @bloom_length : UInt64 + @checksums : Bool def initialize(@path : String, @file_id : Int64, @cache : BlockCache? = nil) @file = File.open(@path, "r") @@ -147,6 +159,7 @@ module Cinderstore @index_length = 0_u64 @bloom_offset = 0_u64 @bloom_length = 0_u64 + @checksums = true read_footer read_index read_bloom @@ -215,14 +228,18 @@ module Cinderstore private def read_block_raw(entry : BlockIndexEntry) : Bytes @file.pos = entry.offset data_len = Util.read_varint(@file).to_i - if data_len.to_u64 + Util.varint_len(data_len.to_u64) + 4 != entry.length + expected_len = Util.varint_len(data_len.to_u64) + data_len + expected_len += 4 if @checksums + if entry.length != expected_len.to_u64 raise CorruptDataError.new("block length mismatch in #{@path}") end data = Bytes.new(data_len) @file.read_fully(data) - stored_crc = @file.read_bytes(UInt32, IO::ByteFormat::LittleEndian) - actual_crc = Util.crc32(data) - raise CorruptDataError.new("block checksum mismatch in #{@path}") unless stored_crc == actual_crc + if @checksums + stored_crc = @file.read_bytes(UInt32, IO::ByteFormat::LittleEndian) + actual_crc = Util.crc32(data) + raise CorruptDataError.new("block checksum mismatch in #{@path}") unless stored_crc == actual_crc + end data end @@ -245,7 +262,10 @@ module Cinderstore @bloom_offset = io.read_bytes(UInt64, IO::ByteFormat::LittleEndian) @bloom_length = io.read_bytes(UInt64, IO::ByteFormat::LittleEndian) version = io.read_bytes(UInt32, IO::ByteFormat::LittleEndian) - raise CorruptDataError.new("unsupported version #{version} in #{@path}") unless version == SstableWriter::VERSION + unless version == SstableWriter::VERSION || version == SstableWriter::VERSION_FAST + raise CorruptDataError.new("unsupported version #{version} in #{@path}") + end + @checksums = version == SstableWriter::VERSION end private def read_index : Nil diff --git a/src/cinderstore/version.cr b/src/cinderstore/version.cr index 4b9d872..3b31a8c 100644 --- a/src/cinderstore/version.cr +++ b/src/cinderstore/version.cr @@ -1,3 +1,3 @@ module Cinderstore - VERSION = "0.3.0" + VERSION = "0.4.0" end diff --git a/src/cinderstore/wal.cr b/src/cinderstore/wal.cr index 04735b6..4cdd9b4 100644 --- a/src/cinderstore/wal.cr +++ b/src/cinderstore/wal.cr @@ -1,19 +1,39 @@ module Cinderstore # The write ahead log (WAL). # - # The WAL frames each entry with a length, a payload, and a CRC32. We - # fsync after every append when durability is enabled. A torn write at - # the tail is truncated during recovery. + # Every WAL starts with a nine byte header. The header holds a magic value + # and one flag byte. The flag byte marks whether the records carry CRC32 + # checksums. The writer records its mode in the header. Recovery reads the + # header first, so it parses each file correctly even when the database + # configuration changes between restarts. + # + # With checksums enabled, each record is a length, a payload, and a CRC32. + # A torn write at the tail is detected and truncated during recovery. With + # checksums disabled, each record is only a length and a payload. Recovery + # then stops at the first malformed tail. This mode trades integrity + # checking for speed. + # + # Files without a header are treated as legacy checksummed logs. This keeps + # databases written by release 0.3 readable. class Wal + # Magic value "CINDERWA" written as little endian bytes. + MAGIC = 0x43494E4445525741_u64 + # Flag bit that marks checksummed records. + FLAG_CHECKSUMS = 1_u8 + # Header size in bytes. + HEADER_SIZE = 9 + # Serializes an entry into a framed record. - def self.encode(entry : Entry) : Bytes + def self.encode(entry : Entry, checksums : Bool = true) : Bytes body = IO::Memory.new Cinderstore.write_entry(body, entry) payload = body.to_slice framed = IO::Memory.new Util.write_varint(framed, payload.size.to_u64) framed.write(payload) - framed.write_bytes(Util.crc32(payload), IO::ByteFormat::LittleEndian) + if checksums + framed.write_bytes(Util.crc32(payload), IO::ByteFormat::LittleEndian) + end framed.to_slice end @@ -27,15 +47,18 @@ module Cinderstore return max_seq unless File.exists?(path) File.open(path, "r") do |io| + checksums = detect_checksums(io) loop do break if io.pos >= io.size begin body_len = Util.read_varint(io).to_i body = Bytes.new(body_len) io.read_fully(body) - stored_crc = io.read_bytes(UInt32, IO::ByteFormat::LittleEndian) - actual_crc = Util.crc32(body) - raise CorruptDataError.new("wal checksum mismatch") unless stored_crc == actual_crc + if checksums + stored_crc = io.read_bytes(UInt32, IO::ByteFormat::LittleEndian) + actual_crc = Util.crc32(body) + raise CorruptDataError.new("wal checksum mismatch") unless stored_crc == actual_crc + end entry = Cinderstore.read_entry(IO::Memory.new(body)) if entry.alive mem.put(entry.key, entry.value, entry.seq) @@ -53,18 +76,43 @@ module Cinderstore max_seq end - # Appends framed records to a WAL file. + # Returns true when the log records carry checksums. + # + # Reads the header when present. A file without a header is a legacy + # checksummed log. A headerless empty file carries no records, so either + # mode is safe. + private def self.detect_checksums(io : IO) : Bool + return true if io.size < HEADER_SIZE + header = Bytes.new(HEADER_SIZE) + io.read_fully(header) + magic = IO::Memory.new(header[0, 8]).read_bytes(UInt64, IO::ByteFormat::LittleEndian) + unless magic == MAGIC + # A legacy log has no header. Rewind so the records parse from + # the start of the file. + io.pos = 0 + return true + end + (header[8] & FLAG_CHECKSUMS) != 0 + end + + # Appends framed records to a fresh WAL file. class Writer getter path : String getter size : Int64 - def initialize(@path : String, @sync_each_write : Bool = true) - @file = File.open(@path, "a+") - @size = @file.size + def initialize(@path : String, @sync_each_write : Bool = true, @checksums : Bool = true) + # A WAL is always created fresh. It is replaced after every flush. + @file = File.open(@path, "w+") + @size = 0_i64 + header = IO::Memory.new + header.write_bytes(MAGIC, IO::ByteFormat::LittleEndian) + header.write_byte(@checksums ? FLAG_CHECKSUMS : 0_u8) + @file.write(header.to_slice) + @size += HEADER_SIZE end def append(entry : Entry) : Nil - record = Wal.encode(entry) + record = Wal.encode(entry, @checksums) @file.write(record) @file.flush @file.fsync if @sync_each_write diff --git a/src/cli.cr b/src/cli.cr index 8a91070..b37b079 100644 --- a/src/cli.cr +++ b/src/cli.cr @@ -4,6 +4,12 @@ require "./cinderstore" module Cinderstore # Command line interface for Cinderstore. class CLI + # Options shared by every command that opens a database. + private record Common, db_path : String = "cinderstore-data", checksums : Bool = true + + # A parsed command line and the parser that produced it. + private record Parsed, common : Common, parser : OptionParser + def self.run(args : Array(String)) : Int32 new.run(args) end @@ -36,21 +42,19 @@ module Cinderstore end private def run_server(rest : Array(String)) : Int32 - db_path = "cinderstore-data" host = "127.0.0.1" port = 7654 help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("--host HOST", "Bind address") { |v| host = v } - parser.on("--port PORT", "Listen on this port") { |v| port = v.to_i } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("--host HOST", "Bind address") { |v| host = v } + parser.on("--port PORT", "Listen on this port") { |v| port = v.to_i } + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end - db = DB.new(db_path) + db = DB.new(parsed.common.db_path, config_from(parsed.common)) server = Server.new(db, host, port) server.run db.close @@ -58,22 +62,20 @@ module Cinderstore end private def run_write(rest : Array(String)) : Int32 - db_path = "cinderstore-data" key = "" value = "" help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("--key KEY", "Key to write") { |v| key = v } - parser.on("--value VALUE", "Value to write") { |v| value = v } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("--key KEY", "Key to write") { |v| key = v } + parser.on("--value VALUE", "Value to write") { |v| value = v } + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end raise Error.new("missing --key") if key.empty? - db = DB.new(db_path) + db = DB.new(parsed.common.db_path, config_from(parsed.common)) begin db.put(key, value) puts "ok" @@ -84,20 +86,18 @@ module Cinderstore end private def run_read(rest : Array(String)) : Int32 - db_path = "cinderstore-data" key = "" help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("--key KEY", "Key to read") { |v| key = v } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("--key KEY", "Key to read") { |v| key = v } + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end raise Error.new("missing --key") if key.empty? - db = DB.new(db_path) + db = DB.new(parsed.common.db_path, config_from(parsed.common)) begin if value = db.get(key) puts value @@ -112,20 +112,18 @@ module Cinderstore end private def run_delete(rest : Array(String)) : Int32 - db_path = "cinderstore-data" key = "" help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("--key KEY", "Key to delete") { |v| key = v } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("--key KEY", "Key to delete") { |v| key = v } + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end raise Error.new("missing --key") if key.empty? - db = DB.new(db_path) + db = DB.new(parsed.common.db_path, config_from(parsed.common)) begin db.delete(key) puts "ok" @@ -136,23 +134,21 @@ module Cinderstore end private def run_scan(rest : Array(String)) : Int32 - db_path = "cinderstore-data" start_key = "" finish_key = "" limit = -1 help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("--start KEY", "First key (inclusive)") { |v| start_key = v } - parser.on("--finish KEY", "Last key (exclusive)") { |v| finish_key = v } - parser.on("--limit N", "Maximum rows") { |v| limit = v.to_i } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("--start KEY", "First key (inclusive)") { |v| start_key = v } + parser.on("--finish KEY", "Last key (exclusive)") { |v| finish_key = v } + parser.on("--limit N", "Maximum rows") { |v| limit = v.to_i } + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end - db = DB.new(db_path) + db = DB.new(parsed.common.db_path, config_from(parsed.common)) begin rows = db.scan(start_key, finish_key.empty? ? nil : finish_key, limit) rows.each do |key, value| @@ -165,17 +161,15 @@ module Cinderstore end private def run_simple(action : String, rest : Array(String)) : Int32 - db_path = "cinderstore-data" help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end - db = DB.new(db_path) + db = DB.new(parsed.common.db_path, config_from(parsed.common)) begin case action when "stats" @@ -197,16 +191,15 @@ module Cinderstore db_path = nil fixture = nil help = false - parser = OptionParser.new - parser.on("--db PATH", "Database directory") { |v| db_path = v } - parser.on("--fixture PATH", "CSV fixture file") { |v| fixture = v } - parser.on("-h", "--help", "Show this help") { help = true } - parser.parse(rest) + parsed = parse_common(rest) do |parser| + parser.on("--fixture PATH", "CSV fixture file") { |v| fixture = v } + parser.on("-h", "--help", "Show this help") { help = true } + end if help - puts parser + puts parsed.parser return 0 end - Demo.run(db_path, fixture) + Demo.run(db_path, fixture, parsed.common.checksums) end private def print_help : Nil @@ -230,6 +223,24 @@ module Cinderstore Run "cinderstore --help" for command options. HELP end + + # Parses the shared options plus any command options. + private def parse_common(rest : Array(String), &) : Parsed + common = Common.new + parser = OptionParser.new + parser.on("--db PATH", "Database directory") { |v| common = Common.new(v, common.checksums) } + parser.on("--no-checksums", "Disable CRC32 checksums") { common = Common.new(common.db_path, false) } + yield parser + parser.parse(rest) + Parsed.new(common, parser) + end + + # Returns a database config from the shared options. + private def config_from(common : Common) : DB::Config + config = DB::Config.new + config.checksums = common.checksums + config + end end end