Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 2 additions & 13 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,26 +28,19 @@ jobs:
# hashdb leg) discards the already-built all-features artifacts and the next
# run pays the full cold build again. Persist on failure to break that trap.
cache-on-failure: true
# etcd-client (opendal services-etcd) requires protoc at build time.
# Linux linker: neither mold nor the default GNU ld can link the ~100 huge
# bundled-C++ test binaries on the 7 GB runner — mold DEADLOCKS (6 h hang) and
# GNU ld SIGBUSes (collect2: ld terminated with signal 7, core dumped, proven in
# CI logs). Use lld, which links these large binaries memory-efficiently without
# mold's deadlock; the debuginfo reduction below removes the actual memory driver.
- name: Install protoc + lld (Linux)
- name: Install lld (Linux)
if: runner.os == 'Linux'
run: sudo apt-get update -qq && sudo apt-get install -y protobuf-compiler clang lld
run: sudo apt-get update -qq && sudo apt-get install -y clang lld
- name: Use lld linker (Linux)
if: runner.os == 'Linux'
run: |
mkdir -p ~/.cargo
printf '[target.x86_64-unknown-linux-gnu]\nlinker = "clang"\nrustflags = ["-C", "link-arg=-fuse-ld=lld"]\n' >> ~/.cargo/config.toml
- name: Install protoc (macOS)
if: runner.os == 'macOS'
run: brew install protobuf
- name: Install protoc (Windows)
if: runner.os == 'Windows'
run: choco install protoc --no-progress
# blazehash links ~800 crates + bundled C++ (duckdb/rocksdb/wasmtime/yara-x) into
# each of ~100 integration-test binaries. With full debuginfo (debuginfo=2) the
# linker exhausts the runner's RAM: link.exe LNK1102 on Windows, and collect2/ld
Expand Down Expand Up @@ -81,8 +74,6 @@ jobs:
with:
components: llvm-tools-preview
- uses: taiki-e/install-action@a402910a723481c4c80d006d75298c796a9c8695 # cargo-llvm-cov
- name: Install protoc
run: sudo apt-get update -qq && sudo apt-get install -y protobuf-compiler
- name: Generate coverage
run: cargo llvm-cov --all-features --lcov --output-path lcov.info
- name: Upload to Codecov
Expand All @@ -98,8 +89,6 @@ jobs:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: dtolnay/rust-toolchain@1.88.0
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- name: Install protoc
run: sudo apt-get update -qq && sudo apt-get install -y protobuf-compiler
- run: cargo check --all-features

deny:
Expand Down
72 changes: 14 additions & 58 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,10 @@ signature = "2"
qrcode = { version = "0.14", optional = true }
image = { version = "0.25", optional = true, default-features = false, features = ["png"] }
regex = "1"
# Scoped to forensically-relevant evidence-transfer targets — the stores DFIR
# collectors (Velociraptor, KAPE) read case evidence from and write collections
# back to. Native DB/KV engines, caches, dev/ML/CI-cache, decentralized and
# consumer-drive backends are intentionally excluded (see ADR-0010).
opendal = { version = "0.55.0", optional = true, features = [
# Cloud object storage
"services-s3", # AWS S3, MinIO, R2, Wasabi, Backblaze B2 (S3-compat)
Expand All @@ -158,66 +162,24 @@ opendal = { version = "0.55.0", optional = true, features = [
"services-obs", # Huawei Cloud OBS
"services-oss", # Alibaba Cloud OSS
"services-swift", # OpenStack Swift
"services-upyun", # Upyun CDN storage
# Consumer / enterprise cloud drives
# Cloud drive (the one drive with dedicated wiring in src/remote/gdrive/)
"services-gdrive", # Google Drive
"services-onedrive", # Microsoft OneDrive
"services-dropbox", # Dropbox
"services-aliyun-drive",# Aliyun Drive (Alibaba)
"services-yandex-disk", # Yandex Disk
"services-pcloud", # pCloud
"services-koofr", # Koofr
"services-seafile", # Seafile
# Developer / ML / infra
"services-github", # GitHub (code forensics)
"services-huggingface", # HuggingFace datasets/models
"services-vercel-artifacts", # Vercel build cache
"services-vercel-blob", # Vercel Blob
# Distributed / big data
"services-alluxio", # Alluxio data orchestration
# Hadoop
"services-webhdfs", # WebHDFS REST (no JVM needed)
"services-lakefs", # LakeFS data versioning
"services-dbfs", # Databricks DBFS
"services-ghac", # GitHub Actions Cache (useful in CI forensics)
# Decentralized
"services-ipfs", # IPFS (content-addressed, prost dep is pure Rust)
"services-ipmfs", # IPFS Mutable File System
# In-memory / embedded (useful for testing + pipeline)
"services-memory",
"services-dashmap", # Concurrent in-memory map
"services-mini-moka", # Bounded in-memory cache
"services-moka", # Full-featured in-memory cache with TTL
"services-redb", # Embedded key-value (pure Rust, ACID)
"services-sled", # Embedded key-value (pure Rust)
"services-cacache", # Content-addressable disk cache
# Network key-value / databases
"services-redis", # Redis
"services-redis-native-tls", # Redis with TLS (rediss://)
"services-memcached", # Memcached
"services-etcd", # etcd (gRPC, pure Rust)
"services-tikv", # TiKV distributed KV
"services-mongodb", # MongoDB (pure Rust driver)
"services-gridfs", # MongoDB GridFS
"services-mysql", # MySQL / MariaDB (sqlx, pure Rust)
"services-postgresql", # PostgreSQL (sqlx, pure Rust)
"services-sqlite", # SQLite (sqlx, pure Rust)
# "services-surrealdb" — excluded: pulls async-graphql@7.2.1 which requires rustc 1.89
"services-persy", # Persy embedded (pure Rust)
# Misc
"services-cloudflare-kv", # Cloudflare KV (via REST)
"services-d1", # Cloudflare D1 SQLite (via REST)
# Filesystem
"services-hdfs-native", # HDFS pure-Rust native client (no Java/libhdfs required)
# SQL (sqlx, pure Rust)
"services-mysql", # MySQL / MariaDB
"services-postgresql", # PostgreSQL (README-advertised)
"services-sqlite", # SQLite
# Filesystem / protocol
"services-fs",
"services-http",
"services-webdav",
# services-sftp intentionally omitted: openssh Rust crate is Unix-only.
"services-memory", # in-memory (useful for testing + pipeline)
# services-sftp intentionally omitted: openssh Rust crate is Unix-only.
# SFTP is handled via ssh2 (libssh2 vendored, cross-platform) in src/remote/sftp.rs.
# services-ftp intentionally omitted: async-tls/tokio-rustls conflict.
# FTP/FTPS is handled via suppaftp (blocking + rustls) instead.
"services-hdfs-native", # HDFS pure-Rust native client (no Java/libhdfs required)
"services-compfs", # compio-based async filesystem
# services-monoiofs intentionally omitted from main features: monoio requires Linux io_uring.
# Added via [target.'cfg(target_os = "linux")'.dependencies] below.
"blocking",
] }

Expand All @@ -231,11 +193,6 @@ libc = "0.2"

[target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2"
# monoiofs depends on monoio (io_uring) which only compiles on Linux.
# Optional + gated behind the `remote` feature (same as the main opendal dep):
# the default build must not pull the opendal cloud stack (and its vulnerable
# DNS / quick-xml transitive tree) on Linux either.
opendal = { version = "0.55.0", features = ["services-monoiofs"], optional = true }

[target.'cfg(target_os = "windows")'.dependencies]
windows-sys = { version = "0.59", features = [
Expand Down Expand Up @@ -272,7 +229,6 @@ ots = []
pq = []
qr = ["dep:qrcode", "dep:image"]
remote = ["dep:opendal", "dep:tokio", "dep:suppaftp", "dep:ssh2"]
rocksdb-storage = ["remote", "opendal/services-rocksdb"]

[package.metadata.deb]
maintainer = "SecurityRonin <security-ronin@users.noreply.github.com>"
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Every hashdeep flag works. Your existing scripts keep working. These are the add
| Duplicate detection | Y | — |
| Manifest diff / merge / update | Y | — |
| Live monitoring (watch) | Y | — |
| Remote storage (S3/GCS/Azure/WebDAV/60+ protocols) | Y | — |
| Remote storage (S3/GCS/Azure/SFTP/WebDAV/HDFS/SQL) | Y | — |
| Google Drive hash-without-download | Y | — |
| GPU-accelerated SHA-256/MD5 | Y | — |
| MCP server (AI-assisted workflows) | Y | — |
Expand All @@ -137,7 +137,7 @@ BLAKE3 runs at **1,640–1,780 MB/s** — 2.8x faster than hashdeep's best algor

## Remote Storage

Evidence doesn't live only on disk. blazehash speaks 50+ storage protocols natively — the same command works whether the data is local, on S3, in Google Drive, or on an SFTP server.
Evidence doesn't live only on disk. blazehash reads and writes remote storage natively — the same command works whether the data is local, on S3, in Google Drive, or on an SFTP server.

```bash
# Hash an S3 prefix directly
Expand All @@ -160,7 +160,7 @@ blazehash sftp://admin@192.168.1.10/cases/image.dd

Credentials come from standard environment variables — `AWS_ACCESS_KEY_ID`, `GOOGLE_APPLICATION_CREDENTIALS`, `AZURE_STORAGE_ACCOUNT` — so existing tooling and CI secrets work without changes. For Google Drive, run `blazehash gdrive auth login` once.

Supported: S3, GCS, Azure Blob/Files/ADLS, Backblaze B2, Tencent COS, Huawei OBS, Alibaba OSS, OpenStack Swift, OneDrive, Dropbox, Google Drive, pCloud, Yandex Disk, SFTP, FTP, WebDAV, WebHDFS, Redis, MongoDB, PostgreSQL, and [30+ more →](https://securityronin.github.io/blazehash/remote/)
Supported: S3, GCS, Azure Blob/Files/ADLS, Backblaze B2, Tencent COS, Huawei OBS, Alibaba OSS, OpenStack Swift, Google Drive, HDFS/WebHDFS, MySQL, PostgreSQL, SQLite, SFTP, FTP/FTPS, WebDAV, and HTTP — [full reference →](https://securityronin.github.io/blazehash/remote/)

---

Expand All @@ -174,7 +174,7 @@ cargo install blazehash --all-features

| Flag | Default | Enables |
|------|:-------:|---------|
| `remote` | on | Remote storage (S3/GCS/Azure/SFTP/GDrive/60+ protocols) |
| `remote` | on | Remote storage (S3/GCS/Azure/SFTP/GDrive, HDFS, SQL) |
| `sqlite` | on | `--format sqlite` output + NSRL SQLite database |
| `parquet-output` | on | `--format parquet` output |
| `duckdb-output` | on | `--format duckdb` output |
Expand Down
7 changes: 3 additions & 4 deletions deny.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,10 @@ targets = []
# The `remote` feature (opendal cloud/object-storage stack) is a genuinely-optional
# heavy subsystem — opt-in, NOT default (see Cargo.toml [features]). It drags in a
# vulnerable transitive tree (hickory DNS RUSTSEC-2026-0118/0119, quick-xml
# RUSTSEC-2026-0194/0195, plus sqlx/tikv/rustls chains) that no hashing workload
# RUSTSEC-2026-0194/0195, plus sqlx/rustls chains) that no hashing workload
# exercises. Rather than blanket-ignore all of those advisories, we keep opendal
# out of the audited graph entirely: check every OTHER feature, never `remote`
# (or `rocksdb-storage`, which pulls it in). `cargo deny check` therefore audits
# exactly what a default `cargo build` ships.
# out of the audited graph entirely: check every OTHER feature, never `remote`.
# `cargo deny check` therefore audits exactly what a default `cargo build` ships.
all-features = false
features = [
"forensic-image",
Expand Down
2 changes: 1 addition & 1 deletion docs/PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Grounded in `src/` and `src/cli.rs`:
(`duckdb-output`, bundled engine floored for the MSVC toolchain —
[ADR 0011](decisions/0011-duckdb-msvc-fmt-floor.md)), JSON/JSONL, STIX 2.1, and
ECS NDJSON (`src/format/`, `src/output.rs`).
- **Remote storage:** S3/GCS/Azure/60+ backends via opendal, plus FTP/SFTP, and
- **Remote storage:** S3/GCS/Azure and other object stores via opendal, plus HDFS, SQL, FTP/SFTP, and
Google Drive hash-without-download (`src/remote/`, `remote` feature —
[ADR 0002](decisions/0002-batteries-included-remote-opt-in.md)).
- **Interfaces:** the `blazehash` CLI, an interactive `tui` dashboard
Expand Down
11 changes: 4 additions & 7 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -951,7 +951,7 @@ blazehash completions man > blazehash.1 && man ./blazehash.1

## Remote storage

blazehash accepts remote URIs for both input paths and `-o` output — 50+ protocols supported.
blazehash accepts remote URIs for both input paths and `-o` output — object stores, HDFS, SQL, SFTP/FTP, and WebDAV/HTTP.

**Full protocol reference:** [Remote Storage →](remote.md)

Expand All @@ -974,12 +974,9 @@ blazehash gdrive://1ABCdef...
blazehash sftp://admin@192.168.1.10/evidence/disk.dd
```

Supported schemes include: `s3`, `gcs`, `azblob`, `azdls`, `azfile`, `b2`, `cos`, `obs`, `oss`,
`swift`, `upyun`, `gdrive`, `onedrive`, `dropbox`, `aliyun-drive`, `yandex-disk`, `pcloud`,
`koofr`, `seafile`, `github`, `huggingface`, `vercel-blob`, `alluxio`, `hdfs`, `webhdfs`, `lakefs`,
`ipfs`, `ipmfs`, `redis`, `rediss`, `memcached`, `etcd`, `tikv`, `mongodb`, `mysql`, `postgresql`,
`sqlite`, `rocksdb` (opt-in), `cloudflare-kv`, `d1`, `sftp`, `ftp`, `ftps`, `webdav`, `http`,
`https`, `compfs`, `monoiofs` (Linux), `file`, `mem`.
Supported schemes: `s3`, `gcs`, `azblob`, `azdls`, `azfile`, `b2`, `cos`, `obs`, `oss`,
`swift`, `gdrive`, `hdfs`, `webhdfs`, `mysql`, `postgresql`, `sqlite`, `sftp`, `ftp`, `ftps`,
`webdav`, `http`, `https`, `file`, `mem`.

---

Expand Down
91 changes: 91 additions & 0 deletions docs/decisions/0012-scope-remote-opendal-backends.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# 12. Scope the `remote` OpenDAL backend set to forensically-relevant evidence-transfer targets

Date: 2026-07-27
Status: Accepted

## Context

ADR-0002 gated the whole `opendal` cloud stack behind the opt-in `remote`
feature, so the default binary and the security audit stay opendal-free. It did
not scope *which* OpenDAL services `remote` compiles — every service OpenDAL
publishes (~50) was enabled. That set carries native database/KV engines and an
exotic long tail that no evidence-transfer workflow exercises but that costs real
build time:

- `services-rocksdb` pulls `librocksdb-sys`, a bundled C++ build — the
`link.exe LNK1102` OOM culprit on the Windows leg under a debuginfo build.
- `services-etcd` / `services-tikv` pull `tonic`/`prost` gRPC stacks that need a
`protoc` binary at build time — the sole reason `ci.yml` installs
`protobuf-compiler` on all three OSes.
- `services-mongodb`/`gridfs` and the cache / embedded-KV set (`redis`,
`memcached`, `sled`, `redb`, `persy`, `moka`, `mini-moka`, `dashmap`,
`cacache`) add breadth with no forensic evidence-transfer story.

blazehash's `remote` feature exists so an examiner can hash evidence that lives
on a remote store, and write a collection back to one — the same targets DFIR
collectors (Velociraptor, KAPE) upload to: object storage (S3/Azure/GCS),
SFTP/FTP, WebHDFS/HDFS, and SQL stores. A distributed-KV cache or a
decentralized-filesystem gateway is not where case evidence lives.

Two honesty notes on the analysis behind this decision. An initial review pass
claimed specific dependency-crate-count reductions and a `libsqlite3-sys`
removal; a Codex critic could not reproduce those counts (`--all-features`
measured ~771 crates, not the cited figure) and `libsqlite3-sys` is pulled by
`rusqlite` regardless — so no crate-count figure is asserted here. The
load-bearing, verifiable claim is narrower and structural: dropping
`rocksdb`/`etcd`/`tikv` removes the `librocksdb-sys` C++ build and the
`tonic`/`prost` chains, which lets the `protoc` install step leave CI entirely.
The compile-time and linker-RSS win is confirmed by the before/after CI
wall-clock on the enactment PR, not by a crate count.

## Decision

Scope the `remote` feature's OpenDAL service set to forensically-relevant
evidence-transfer targets. `remote` stays opt-in and remote-free-by-default
(ADR-0002 unchanged); this ADR narrows *which* services it compiles.

**KEEP** — evidence-transfer targets:

- Object storage: `s3`, `gcs`, `azblob`, `azdls`, `azfile`, `b2`, `cos`, `obs`,
`oss`, `swift`
- Cloud drive: `gdrive` — the one drive with dedicated wiring
(`src/remote/gdrive/`)
- Hadoop: `webhdfs`, `hdfs-native` (pure-Rust, no JVM)
- SQL: `mysql`, `postgresql`, `sqlite` (sqlx; PostgreSQL is README-advertised)
- Local / protocol: `fs`, `memory`, `http`, `webdav`, plus SFTP (`ssh2`) and
FTP (`suppaftp`), which are non-opendal by design (ADR-0002 §4)

**DROP**:

- Native DB/KV + caches: `rocksdb` (and the `rocksdb-storage` feature), `etcd`,
`tikv`, `mongodb`, `gridfs`, `redis` (+`redis-native-tls`), `memcached`,
`sled`, `redb`, `persy`, `moka`, `mini-moka`, `dashmap`, `cacache`
- Dev / ML / CI-cache: `github`, `huggingface`, `vercel-blob`,
`vercel-artifacts`, `ghac`
- Big data beyond WebHDFS/HDFS: `alluxio`, `lakefs`, `dbfs`
- Decentralized: `ipfs`, `ipmfs`
- Alternate-runtime filesystems: `compfs`, `monoiofs`
- Cloudflare: `cloudflare-kv`, `d1`
- Regional / consumer drives: `onedrive`, `dropbox`, `aliyun-drive`,
`yandex-disk`, `pcloud`, `koofr`, `seafile`, `upyun`

On the drives specifically: GDrive is kept because it carries dedicated wiring;
OneDrive and Dropbox had only generic scheme wiring (no CLI or test path), so
absent a concrete case they ship as unexercised capability and are dropped —
one line to re-add if a real need appears.

## Consequences

- The `protoc` build-tool install step is removed from `ci.yml` (no
`etcd`/`tikv` gRPC stack remains to need it).
- `librocksdb-sys` leaves the `--features remote` graph, removing the
Windows-linker memory pressure it contributed.
- `rocksdb-storage` is removed from `[features]`, and `deny.toml`'s per-feature
enumeration drops it.
- `operator_for_uri` loses the ~25 dropped scheme arms; the KEEP schemes and the
informative SFTP/FTP bail arms remain.
- The cost win — smaller `remote` compile, lower Windows-linker RSS — is proven
by the enactment PR's CI wall-clock against the prior run, not asserted as a
crate-count figure.
- Re-adding any dropped backend is a one-line feature plus one match arm; nothing
here is one-way.
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

The only open-source forensic hashing tool that answers every question a court asks about digital evidence — *what* changed, *who* handled it, *when* it was sealed, and *in what context* — in a single binary that's drop-in compatible with hashdeep.

Now with **50+ remote storage backends** built in via Apache OpenDAL: hash from S3, GCS, Azure Blob, WebDAV, SFTP, and more — no extra flags required. Hash Google Drive files in memory via `gdrive://` URI no local copy needed.
Read and write evidence directly from **remote storage** via Apache OpenDALS3, GCS, Azure Blob, WebDAV, SFTP, HDFS, and SQL stores, the targets DFIR collectors actually use — built into the release binaries, no flag at runtime. Hash Google Drive files in memory via a `gdrive://` URI, no local copy needed.

```bash
# Acquire evidence with chain-of-custody metadata
Expand Down
Loading
Loading