Specification for the on-disk formats used by LSM-tree storage engines conforming to this spec.
This repository is the authority for these formats. It defines them in their own terms, independently of any implementation.
The specification is being reconstructed from existing implementations, but that is research provenance, not an authority relationship. Midge, Pants, and other engines provide evidence about historical intent and interoperability; none defines the format. Normative requirements flow from this specification to implementations.
When an implementation differs from a sound rule, the implementation must change.
When investigation shows a rule itself is wrong, the specification must be amended
deliberately, with rationale and compatibility impact recorded. Normative text must
not be silently changed merely to match current code. Implementation-specific
findings belong in notes/, which is non-normative.
format/— on-disk byte layouts: WAL, SST, manifest, file lease, and the shared value-encoding primitives they share.schema/— machine-checkable schema definitions mirroringformat/.behavior/— behavioral contracts (conflict resolution, durability windows, recovery semantics). Not yet specified.conformance/— requirements for claiming and verifying conformance.notes/— non-normative working notes. Not part of the specification.
The words must, must not, required, should, and may are used in the sense of RFC 2119 wherever they state a requirement, whether or not they appear in capitals.
Each format document ends with a format vs. policy section separating what an
implementation must reproduce for wire compatibility from what it is free to decide.
Points that remain unsettled are marked TODO: verify inline and collected under
that section's Open questions.
0.1.0-dev (see VERSION). Format documents are draft. No conformance fixtures are
published yet; see conformance/README.md.
Semver. A breaking change to any normative rule is a major bump, an additive/
backward-compatible clarification is a minor bump, wording-only fixes are a patch
bump. See CHANGELOG.md.
Licensed under the Apache License, Version 2.0.