Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions docs/PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
*Reverse-written from the shipped code, README, and git history (2026-07-24). Every
current-state claim is grounded in a same-session read of `Cargo.toml`, `src/`, and
`docs/`. The load-bearing engineering decisions live as ADRs
[0001](decisions/0001-lean-core-full-app-split.md)–[0009](decisions/0009-embed-attack-table-drop-forensicnomicon-dep.md)
[0001](decisions/0001-lean-core-full-app-split.md)–[0011](decisions/0011-duckdb-msvc-fmt-floor.md)
under [`docs/decisions/`](decisions/). Product tier: blazehash ships a binary an
examiner runs (the `blazehash` CLI, plus a `tui` mode and an `mcp` server), so it
carries a full PRD per the fleet PRD & ADR standard.*
Expand Down Expand Up @@ -58,7 +58,8 @@ custody.
Grounded in `src/` and `src/cli.rs`:

- **Recursive, parallel hashing** with memory-mapped and direct (uncached) I/O
(`src/hash.rs`), across 25+ algorithms — BLAKE3 (default), the SHA-1/2/3,
(`src/hash.rs`), across 25+ algorithms — BLAKE3 (default,
[ADR 0010](decisions/0010-blake3-default-hashdeep-superset.md)), the SHA-1/2/3,
MD5, Tiger, Whirlpool, BLAKE2, SM3, Streebog, RIPEMD, K12, checksums
(CRC/CRC32C/Adler), fast hashes (xxh3), and fuzzy/similarity hashes (ssdeep,
TLSH) — the last two in `blazehash-core::fuzzy`.
Expand All @@ -75,8 +76,10 @@ Grounded in `src/` and `src/cli.rs`:
Shannon entropy, and VirusTotal batch lookup (`src/vt.rs`).
- **Forensic-image and device input:** EWF/E01 verification (`src/forensic_image/`,
`forensic-image` feature) and raw block-device sizing (`src/device.rs`).
- **Output formats:** the hashdeep text formats plus SQLite, Parquet, DuckDB,
JSON/JSONL, STIX 2.1, and ECS NDJSON (`src/format/`, `src/output.rs`).
- **Output formats:** the hashdeep text formats plus SQLite, Parquet, DuckDB
(`duckdb-output`, bundled engine floored for the MSVC toolchain —
[ADR 0011](decisions/0011-duckdb-msvc-fmt-floor.md)), JSON/JSONL, STIX 2.1, and
ECS NDJSON (`src/format/`, `src/output.rs`).
- **Remote storage:** S3/GCS/Azure/60+ backends via opendal, plus FTP/SFTP, and
Google Drive hash-without-download (`src/remote/`, `remote` feature —
[ADR 0002](decisions/0002-batteries-included-remote-opt-in.md)).
Expand Down Expand Up @@ -152,3 +155,5 @@ Grounded in `src/` and `src/cli.rs`:
| [0007](decisions/0007-dual-release-pipeline.md) | release-plz for the library, `v[0-9]*` tag for the binary |
| [0008](decisions/0008-pure-rust-zip-forensic-core.md) | Pure-Rust `zip-forensic-core` reader; dev-only `zip` for fixtures |
| [0009](decisions/0009-embed-attack-table-drop-forensicnomicon-dep.md) | Embed the ATT&CK table; no compile-time forensicnomicon dep |
| [0010](decisions/0010-blake3-default-hashdeep-superset.md) | BLAKE3 is the default algorithm; hashdeep flags still select the rest |
| [0011](decisions/0011-duckdb-msvc-fmt-floor.md) | Floor `duckdb-output` at DuckDB 1.5.5 (crate 1.10505.0) for the MSVC 14.51 fmt break |
50 changes: 50 additions & 0 deletions docs/decisions/0010-blake3-default-hashdeep-superset.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# 10. BLAKE3 is the default algorithm, while remaining a strict hashdeep superset

Date: 2026-07-27
Status: Accepted

## Context

blazehash's whole positioning is a *drop-in superset of hashdeep*: every hashdeep
flag and output format works unchanged, so an examiner's existing scripts keep
running (README "What's New vs hashdeep"; PRD "Scope"). hashdeep, however, computes
MD5 + SHA-256 when no algorithm is named — the general-purpose cryptographic hashes
of its era. Defaulting to those makes blazehash's headline advantage (BLAKE3 at
~1,640 MB/s, README "Performance") invisible to the most common invocation, where a
user simply points the tool at a tree and takes the default.

The tension: a superset is expected to behave like the original, but the original's
default no longer reflects the fastest, most modern primitive the tool ships. Two
things pull in opposite directions — compatibility (keep hashdeep's default) and the
product's reason to exist (lead with BLAKE3).

## Decision

1. **The zero-argument default algorithm is BLAKE3.** `-c/--compute` defaults to
`"blake3"` (`src/cli.rs:18`), and the fall-through when no algorithm resolves is
`vec![Algorithm::Blake3]` (`src/cli.rs:488`). A plain `blazehash <path>` produces
BLAKE3 digests.
2. **Every hashdeep algorithm remains selectable, unchanged.** `-c` takes a
comma-separated list parsed by `parse_algorithms` into the `Algorithm` enum
(`core/src/algorithm.rs`), which includes MD5, the SHA-1/2/3 families, Tiger,
Whirlpool, and the rest — so `blazehash -c md5,sha256` reproduces hashdeep's
default behaviour exactly. Compatibility is preserved by *keeping the flags*, not
by keeping the default.
3. **The algorithm set lives in the lean engine.** `Algorithm`, its parsing, and
dispatch are `blazehash-core` ([ADR 0001](0001-lean-core-full-app-split.md)); the
app selects among them. The default is an application/CLI decision, not baked into
the engine.
4. **Correctness rests on audited upstream crates.** BLAKE3 via the `blake3` crate,
the rest via RustCrypto; blazehash writes no hand-rolled hash math (PRD "Non-Goals").

## Consequences

- The common path showcases the differentiator: the fastest invocation is also the
one a new user runs first, matching the README's "BLAKE3 by default" tagline.
- A user migrating a hashdeep workflow that relied on the *implicit* MD5+SHA-256
default must name those algorithms explicitly (`-c md5,sha256`). This is the one
visible behavioural divergence from hashdeep, and it is deliberate — the flags
make it a one-token fix, so the superset promise holds for anything that named its
algorithms (the norm in forensic scripting).
- Because the algorithms are all present, changing the default is a low-risk,
reversible CLI choice, not an architectural commitment.
62 changes: 62 additions & 0 deletions docs/decisions/0011-duckdb-msvc-fmt-floor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# 11. Floor the `duckdb-output` dependency at DuckDB 1.5.5 (crate 1.10505.0) for the MSVC 14.51 fmt break

Date: 2026-07-27
Status: Accepted

## Context

The `duckdb-output` feature emits a `.duckdb` manifest via the `duckdb` crate with
the `bundled` feature (`Cargo.toml`; `src/format/duckdb_fmt.rs`), which vendors and
compiles DuckDB's C++ engine at build time rather than linking a system library.
`duckdb-output` is in the `default` feature set, so it is exercised by CI's
`cargo test --all-features` on every platform.

GitHub's `windows-latest` runner moved to Visual Studio 2026, whose MSVC 14.51
toolset **removed `stdext::checked_array_iterator`**. The DuckDB engine bundled by
`duckdb` 1.10501.0 (engine 1.5.1) vendors an old `fmt` whose `format.h:326` still
declares `checked_ptr = stdext::checked_array_iterator<T*>`. The C++ compile of
`libduckdb-sys` therefore fails on the new MSVC with `C2653` ("'stdext' is not a
class or namespace name") and a cascade of syntax errors, reddening the
`test (windows-latest)` job. macOS/Linux (clang) are unaffected, so the failure is
Windows-MSVC-specific.

The manifest requirement `^1.4.4` already *allowed* a fixed version, but the
committed `Cargo.lock` (batteries-included repos commit the lock —
[ADR 0002](0002-batteries-included-remote-opt-in.md) and the fleet lock rule) held
CI at the broken 1.10501.0. This is the classic "lock behind the requirement"
staleness layer: a green-looking requirement with a broken resolved version.

## Decision

1. **Bump the lock and raise the manifest floor to the first fixed release.** DuckDB
dropped the removed `stdext` usage in PR #23261 / #23239, shipped in engine 1.5.5
= crate **1.10505.0**. `Cargo.toml` pins `duckdb = { version = "1.10505.0",
features = ["bundled"], optional = true }` with an inline comment recording the
MSVC 14.51 cause, and `Cargo.lock` is updated to match (commit `4233c08`,
branch `fix/windows-ci-duckdb-msvc-fmt`).
2. **Fix the cause, do not suppress the signal.** The build break is repaired by
moving to the fixed dependency — not by skipping the `duckdb-output` test on
Windows, dropping the feature from `--all-features`, or excluding the runner
(fleet "Root-Cause Over Suppression"). The floor is verified against the bundled
sources: 1.10501.0's `format.h` contains the removed symbol at line 326 (matching
the CI error line); 1.10505.0's `format.h` has zero references.
3. **Scope is the app only.** `duckdb-output` is a `blazehash` (app) feature; the
published `blazehash-core` library has no `duckdb` dependency, so this floor does
not touch the engine's MSRV or its downstream compatibility promise
([ADR 0001](0001-lean-core-full-app-split.md),
[ADR 0006](0006-msrv-floor-vs-pinned-toolchain.md)).

## Consequences

- Windows-MSVC CI compiles `libduckdb-sys` again; `cargo test --all-features` is
green across all three platforms.
- The floor is a lower bound, not a ceiling — Renovate/`cargo update` may advance
`duckdb` further; the requirement only forbids regressing below the MSVC-safe
engine.
- The decision is a dependency floor tied to a real, documented upstream toolchain
discontinuity (a removed MSVC extension), so it is a domain fact, not an arbitrary
pin: it may be revisited only if DuckDB's minimum-supported MSVC changes again.
- *State note: the floor bump lands via commit `4233c08` on branch
`fix/windows-ci-duckdb-msvc-fmt`; on `main` at the time of writing `Cargo.toml`
still reads the pre-fix `duckdb = "1.4.4"`. This ADR records the decision; the two
changes converge when both branches merge.*
Loading