Varve

Varve is a Rust library for building append-friendly, segment-oriented custom binary formats from typed block definitions. It is not one fixed file format. The main workflow is format-first: declare a file contract with varve_format!, then use the generated typed reader and writer APIs.

The workspace contains:

• varve: public facade crate
• varve-core: runtime, codecs, file I/O, diagnostics
• varve-macros: varve_format! and derive support

What Varve Provides

• Compile-time format declarations with generated typed reader and writer APIs.
• Fixed and variable blocks with canonical encoding instead of Rust memory layout copying by default.
• Append-friendly record framing, offset-chain metadata, commit policies, checksum hooks, keyed collections, and lazy scan/index support.
• Variable payload compression policies that can be declared globally, per block, or left unspecified for caller-defined behavior.
• Adapter primitives for physical formats that need custom lead-ins, table of contents masks, raw data offsets, next-segment offsets, or externally defined record framing.

Varve should own the reusable binary-format mechanics. Application-specific meaning, domain transforms, and compatibility with an external specification remain caller code.

Start Here

Document	Use it for
Quickstart	shortest path from format declaration to write/read
Declaration And Internals	how the DSL maps to generated Rust API, native bytes, records, fields, indexes, commits, and custom physical layouts
How Varve Works	mental model of generated code, append logs, matrix storage, durability
API Reference	practical public API map
Format Author Guide	policy choices, compression, commit modes, matrix blocks
Self-Check Guide	deciding whether a failure is format, caller, data, feature, environment, or library
Architecture	current architectural outline and implementation status
Requirements Boundary	what Varve owns versus what callers should implement
Adapter Toolkit Design	generic adapter primitives for external binary formats
npTDMS Adapter Boundary	how npTDMS-documented behavior maps to Varve generic APIs versus external adapter code

Minimal Shape

use varve::varve_format;

varve_format! {
    pub format AppFormat {
        magic: b"APP";
        version: 1;
        schema_hash: computed;
        commit: transaction_marker(on_flush);

        blocks {
            fixed Point(id = 1) {
                x: u32,
                y: u32,
            }

            variable User(id = 2, key = [id]) {
                id: u64,
                name: String,
                flags: u32 = default,
            }
        }
    }
}

let mut writer = AppFormat::create_writer("app.varve")?;
writer.push_point(&Point { x: 1, y: 2 })?;
writer.push_user(&User { id: 7, name: "Ada".to_string(), flags: 0 })?;
writer.flush()?;

let reader = AppFormat::open_reader("app.varve")?;
let user = reader.users()?.get(&7)?;

The user declares the format and block types. Varve generates the typed methods, block metadata, required field encoding, record headers, commit handling, offset-chain metadata, and diagnostics.

Examples

Example	Role
crates/varve/examples/bmp_physical.rs	Small physical-format adapter proof for BMP-style headers and raw pixel data, verified against Pillow.
crates/varve/examples/perf_bench.rs	Local throughput check for append, open, scan, and block access paths.
crates/varve/examples/tdms_physical_reader.rs	Thin reader entry point for the TDMS physical adapter example.
crates/varve/examples/tdms_physical_writer.rs	Thin writer entry point for the TDMS physical adapter example.
crates/varve/examples/tdms_physical_adapter.rs	CLI wrapper around the TDMS adapter for write, append, read, read-bytes, and inspect flows.
crates/varve/examples/tdms_physical/common/adapter.rs	Large TDMS physical adapter proof using Varve's generic adapter APIs.
crates/varve/examples/tdms_physical/common/example_data.rs	Multi-channel TDMS scenario fixtures.
crates/varve/examples/tdms_physical/common/verify.rs	Regression checks for TDMS round trips and segment-layout behavior.

The TDMS example is intentionally larger than a quickstart. Its job is to prove that Varve can support a demanding external binary format without hard-coding TDMS-specific behavior into the library. It also acts as a regression harness so combined changes do not silently break segment appends, metadata reuse, scalar types, or cross-tool compatibility.

Self-Check

Generated formats expose diagnostics and end-to-end self-test helpers:

let diagnostics = AppFormat::diagnostics();
let file_report = AppFormat::diagnose_file("data.varve");
let self_test = AppFormat::self_test("self-check.varve")
    .with_block(Point { x: 1, y: 2 })
    .cleanup(true)
    .run();

These reports classify failures as format definition, caller usage, feature gate, file data, environment, or library invariant issues. See Self-Check Guide.

Status

Varve is usable as an alpha library for experimentation and controlled internal projects. It already includes append-log blocks, keyed collections, transaction/footer commit policies, schema manifests, diagnostics, merge and compact helpers, variable-block compression, matrix storage, mmap, and opt-in zero-copy. Treat the wire format as pre-stabilization until the first v0.1 release is cut and pinned.

Local Verification

cargo fmt --check
cargo test
cargo test --all-features
cargo clippy --all-targets --all-features -- -D warnings

Optional compatibility harnesses use Python reference libraries:

.\.venv-tdms\Scripts\python.exe scripts\verify_tdms_with_nptdms.py
.\.venv-tdms\Scripts\python.exe scripts\verify_nptdms_multichannel_with_varve.py
.\.venv-tdms\Scripts\python.exe scripts\verify_bmp_with_pillow.py

License

Varve is licensed under either of:

• MIT License
• Apache License, Version 2.0

at your option. See LICENSE for the short dual-license notice.