backup-integrity-verifier

Read-only backup verification for operators who need confidence before a restore — not another backup tool.

A cross-platform Python CLI that inspects local backup targets for freshness, integrity, archive readability, and safe restore capability. It never modifies, deletes, or rewrites source backups. Reports are emitted as standalone JSON and HTML files suitable for automation and audit trails.

The screenshot and example reports contain synthetic demonstration data only. No real backup, user, device, network, or credential information is included.

Problem being solved

Backup jobs can appear healthy while producing stale, empty, truncated, unreadable, or structurally invalid files. A file on disk does not prove recoverability: archives may fail CRC checks, checksum manifests may drift, duplicates may mask retention gaps, and path-traversal payloads may hide inside otherwise normal-looking archives.

Most teams discover these problems only during an incident. backup-integrity-verifier closes that gap with scheduled, read-only checks that produce structured pass/fail results before you need to restore.

Project objectives

Provide a read-only verifier that never alters source backups
Detect common backup failures: missing targets, stale files, undersized archives, hash mismatches, corrupt ZIP/TAR members, and unsafe archive content
Support safe temporary extraction inside managed directories with traversal and bomb protections
Emit machine-readable JSON and human-readable HTML reports with clear status semantics
Run on Windows and Linux with zero runtime dependencies beyond the Python standard library
Ship with synthetic demo data, tests, CI, and operator documentation suitable for a public portfolio release

Main features

JSON-driven configuration with per-target overrides and validation
Glob-based backup discovery with include/exclude patterns
Freshness checks with separate warning and critical age thresholds
File size and readability validation
Streamed SHA-256 and SHA-512 hashing without loading entire files into memory
Checksum manifest verification (<hash> <relative-path> format)
ZIP and TAR/TAR.GZ integrity inspection with required-member checks
Optional safe test extraction into auto-cleaned temporary directories
Archive bomb limits (member count, uncompressed size, compression ratio)
Duplicate backup detection by digest (report-only; never deletes files)
Standalone JSON and HTML reports plus a concise console summary
Automation-friendly exit codes with optional --fail-on-warning
Synthetic demo generator for reproducible demonstrations and tests

Supported backup formats

Format	Support in 1.0
Plain files	Full checks (size, freshness, hashing, permissions)
Directories	Discovery and metadata checks on contained files
ZIP (`.zip`)	Integrity (`testzip()`), member validation, optional safe extraction
TAR (`.tar`)	Member enumeration, unsafe special-member rejection, optional extraction
TAR.GZ / TGZ (`.tar.gz`, `.tgz`)	Same as TAR via gzip decompression
GZIP (`.gz`)	Practical checks when the payload is a single compressed stream (not a nested archive)
Encrypted ZIP	Reported as Unavailable — password-protected extraction is not supported

Archive type detection uses magic bytes first, with extension-based fallback.

Checks performed

The verifier runs up to 15 independent check categories per configured target. Each produces one or more CheckResult records with a status, summary, and optional details.

#	Category	Purpose
1	Target existence	Confirm the configured path exists (`required` vs optional targets)
2	Backup discovery	Find matching backups via glob, include, and exclude patterns
3	Minimum backup count	Ensure at least `minimum_file_count` backups are present
4	Freshness	Detect stale backups using newest file mtime vs age thresholds
5	File size	Flag empty or undersized files
6	Hash calculation	Streamed SHA-256/SHA-512 digests for discovered files
7	Checksum manifest verification	Compare files against a `<hash> <relative-path>` manifest
8	ZIP integrity	Open archive, validate members, run CRC checks via `testzip()`
9	TAR integrity	Enumerate members and reject unsafe special entries
10	Safe test extraction	Extract into a managed temporary directory when enabled
11	Archive bomb protections	Enforce member count, uncompressed size, and compression-ratio limits
12	Required archive members	Verify expected paths exist inside archives
13	Duplicate backup detection	Identify identical hashes across discovered files
14	Permissions and readability	Confirm the process can read selected files
15	Restore-test result	Report the outcome of safe extraction (readability ≠ application restore)

See docs/checks-reference.md for status rules, configuration keys, and operator responses.

Quick start

git clone https://github.com/salvomazzaglia/backup-integrity-verifier.git
cd backup-integrity-verifier

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux/macOS
source .venv/bin/activate

python -m pip install -U pip
python -m pip install -e ".[dev]"

backup-integrity-verifier generate-demo
backup-integrity-verifier verify --config config/demo.config.json

Reports are written to reports/ by default. Open the generated HTML file in a browser or inspect the JSON for automation.

Installation

Requirements: Python 3.11 or newer. No third-party runtime packages.

python -m pip install -e .

For development (pytest, Ruff, mypy, build tools):

python -m pip install -e ".[dev]"

The CLI entry point is backup-integrity-verifier. You can also invoke the module directly:

python -m backup_integrity_verifier --help

Usage examples

Verify configured targets

backup-integrity-verifier verify --config config/demo.config.json

Write only JSON, only HTML, or both (default):

backup-integrity-verifier verify --config config/demo.config.json --format json
backup-integrity-verifier verify --config config/demo.config.json --format html
backup-integrity-verifier verify --config config/demo.config.json --format all

Custom output directory, verbose diagnostics, and strict warning handling:

backup-integrity-verifier verify \
  --config config/demo.config.json \
  --output-directory reports \
  --verbose \
  --fail-on-warning

Force safe extraction on or off regardless of configuration:

backup-integrity-verifier verify --config config/demo.config.json --test-extraction
backup-integrity-verifier verify --config config/demo.config.json --no-test-extraction

Validate configuration

Validate a JSON configuration file without running checks:

backup-integrity-verifier validate-config --config config/demo.config.json

Invalid configuration prints a precise error and exits with code 3.

Generate synthetic demo backups

Create reproducible demonstration fixtures under examples/demo-backups/:

backup-integrity-verifier generate-demo
backup-integrity-verifier generate-demo --output examples/demo-backups

Module invocation

python -m backup_integrity_verifier verify --config config/demo.config.json
python -m backup_integrity_verifier validate-config --config config/demo.config.json
python -m backup_integrity_verifier generate-demo
python -m backup_integrity_verifier --version

Configuration examples

Configuration is JSON. See docs/configuration.md for the full schema.

Demo configuration — config/demo.config.json — targets synthetic fixtures under examples/demo-backups/ with report.synthetic: true:

{
  "report": {
    "formats": ["json", "html"],
    "include_absolute_paths": false,
    "synthetic": true
  },
  "defaults": {
    "max_age_hours": 48,
    "warning_age_hours": 36,
    "critical_age_hours": 48,
    "minimum_size_bytes": 50,
    "hash_algorithm": "sha256",
    "test_extraction": true,
    "compute_hashes": true,
    "detect_duplicates": true
  },
  "archive_security": {
    "maximum_members": 10000,
    "maximum_total_uncompressed_bytes": 1073741824,
    "maximum_compression_ratio": 200
  },
  "targets": [
    {
      "id": "database-daily",
      "name": "Daily database backup",
      "path": "examples/demo-backups/database",
      "pattern": "*.zip",
      "required": true,
      "required_members": ["demo-database.sql"]
    }
  ]
}

Operator template — config/default.config.json — starting point for real deployments. Keep private paths in a gitignored config/local.config.json.

Report examples

Pre-generated synthetic reports (safe to browse and commit):

File	Description
`examples/sample-report.json`	Machine-readable verification summary
`examples/sample-report.html`	Human-readable report with synthetic banner

Regenerate sample reports after code changes:

python scripts/generate_demo_data.py
python scripts/generate_sample_reports.py

Each report includes overall status, per-check results, target summaries, exit code, and a privacy notice. Synthetic reports display a clear SYNTHETIC DEMONSTRATION DATA banner.

Exit codes

Code	Meaning
0	All checks are Healthy or Unavailable (non-failure for automation)
1	At least one Warning, no Critical or Error
2	At least one Critical
3	Configuration, validation, execution, or unexpected internal error

`--fail-on-warning`

By default, an overall Warning status exits with code 1. With --fail-on-warning, Warning is treated as a hard failure and returns exit code 2 — useful for strict CI pipelines that must fail on any degradation.

Overall status is the highest-severity status among all checks:

Error > Critical > Warning > Unavailable > Healthy

See docs/results-explained.md for status semantics and operator guidance.

Archive-security protections

Archive handling is designed for untrusted inputs. The verifier never writes into source backup directories.

Protection	Behavior
Path traversal blocking	Rejects `..`, absolute paths, home-relative paths, Windows drive prefixes, UNC paths, and null bytes in member names
Extraction root confinement	Resolved paths must stay under the temporary extraction root (`commonpath` validation)
Special-member rejection	TAR symlinks, hard links, device nodes, and FIFOs are blocked
Archive bomb limits	Configurable caps on member count, total uncompressed bytes, and compression ratio block extraction before expansion
Managed temporary directories	Extraction uses `tempfile.TemporaryDirectory` and is always cleaned up, including after failures
Manifest path validation	Checksum manifest entries with traversal or absolute paths are rejected
Symlink policy	Discovery does not follow symlinks by default (`follow_symlinks: false`)
Pre-extraction precheck	Limits are evaluated before any bytes are written to disk

Configure limits under the top-level archive_security key. See docs/security-and-privacy.md and SECURITY.md.

Architecture

flowchart TD
  CLI[CLI] --> CFG[Configuration loader]
  CFG --> DISC[Target discovery]
  DISC --> CHK[Backup checks]
  CHK --> SEC[Archive-security validation]
  SEC --> EXT[Optional temporary extraction]
  EXT --> NORM[Result normalization]
  NORM --> AGG[Overall assessment]
  AGG --> REP[JSON and HTML reporting]

Module	Responsibility
`cli.py`	Argument parsing and command dispatch
`config.py`	JSON configuration load and validation
`discovery.py`	Glob-based backup discovery
`verification.py`	Orchestrates all checks per target
`hashing.py`	Streamed digests and manifest parsing
`archives.py`	ZIP/TAR detection and integrity
`safe_extract.py`	Path validation and safe extraction
`aggregation.py`	Status and exit-code logic
`json_report.py` / `html_report.py`	Report writers
`console.py`	Human-readable summary
`demo.py`	Synthetic demo fixture generation

Detailed design notes: docs/architecture.md.

Testing instructions

python -m pip install -e ".[dev]"

# Lint and format
python -m ruff check src tests scripts
python -m ruff format --check src tests scripts

# Unit tests
python -m pytest

# Full local CI script (lint + format + pytest)
python scripts/run_ci.py

CI matrix: GitHub Actions runs on Ubuntu and Windows with Python 3.11 and 3.12.

Tests cover configuration validation, discovery, freshness and size thresholds, hashing and manifests, archive integrity and corruption, safe extraction and path rejection, aggregation and exit codes, JSON/HTML encoding, and CLI flows. Archive fixtures are generated dynamically in tmp_path where possible.

See docs/testing.md for fixture strategy and cross-platform notes.

Security and privacy

Read-only by design — the verifier does not delete, rename, overwrite, chmod, or chown backup files
No credentials required — do not place secrets in configuration committed to git
Sensitive reports — real reports may reveal backup layout and naming conventions; treat them as confidential
Absolute paths — omitted from reports unless include_absolute_paths is explicitly enabled
Synthetic examples — tracked samples use synthetic names and display a clear demonstration banner
Responsible disclosure — see SECURITY.md

Full policy: docs/security-and-privacy.md.

Known limitations

No password-protected archive extraction — encrypted ZIP files are reported as Unavailable
No cloud or object-storage connectors — only local filesystem paths are supported
No backup creation, rotation, or deletion — this is a verifier, not a backup engine
Archive readability ≠ application restore success — a successful extraction does not prove database import, schema compatibility, or transactional consistency
Checksum match ≠ business recoverability — matching digests confirm byte integrity, not RTO/RPO compliance
GZIP-only payloads — practical checks apply to single-stream gzip files; nested or multi-member gzip layouts may be limited
Screenshot generation — the optional screenshot script requires a local Edge, Chrome, or Chromium installation

Roadmap

Planned directions for future releases (subject to change):

Read-only verification of password-protected archives where feasible without modifying backups
Additional archive format support (e.g., .7z, .xz) with the same safety model
Configurable notification hooks (email, webhook) driven by report exit codes
Report comparison/diff between verification runs for trend analysis
Optional JUnit/XML output for CI dashboards

Track progress via GitHub Issues and CHANGELOG.md.

Skills demonstrated

This project showcases practical skills across automation, security, and operations:

Python automation — typed, standard-library CLI with structured error handling
Backup validation — freshness, count, size, hashing, and manifest verification
Disaster-recovery concepts — distinguishing presence, integrity, and recoverability
Safe archive handling — traversal blocking, bomb limits, and managed extraction
SHA-256 / SHA-512 — streamed hashing without excessive memory use
Structured error handling — precise configuration errors and per-check status semantics
JSON / HTML reporting — offline, self-contained audit artifacts
pytest — deterministic unit tests with dynamic fixtures
Ruff — linting and formatting in CI and local development
GitHub Actions — cross-platform CI matrix (Windows + Ubuntu, Python 3.11/3.12)
Cross-platform development — pathlib-based paths, Windows and POSIX behavior
Security-conscious engineering — read-only guarantees, synthetic data policy, SECURITY.md
Technical documentation — architecture, checks reference, operator guides

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request.

Do not commit: real backup archives, production reports, secrets, or personal filesystem paths in tracked samples.

python -m ruff check src tests scripts
python -m ruff format --check src tests scripts
python -m pytest

License

This project is licensed under the MIT License.

Author

Salvatore Mazzaglia

GitHub: @salvomazzaglia
Repository: github.com/salvomazzaglia/backup-integrity-verifier

Name		Name	Last commit message	Last commit date
.github		.github
config		config
docs		docs
examples		examples
scripts		scripts
src/backup_integrity_verifier		src/backup_integrity_verifier
tests		tests
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
VERSION		VERSION
pyproject.toml		pyproject.toml

Navigation Menu

Repository files navigation

backup-integrity-verifier

Problem being solved

Project objectives

Main features

Supported backup formats

Checks performed

Quick start

Installation

Usage examples

Verify configured targets

Validate configuration

Generate synthetic demo backups

Module invocation

Configuration examples

Report examples

Exit codes

`--fail-on-warning`

Archive-security protections

Architecture

Testing instructions

Security and privacy

Known limitations

Roadmap

Skills demonstrated

Contributing

License

Author

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

backup-integrity-verifier

Problem being solved

Project objectives

Main features

Supported backup formats

Checks performed

Quick start

Installation

Usage examples

Verify configured targets

Validate configuration

Generate synthetic demo backups

Module invocation

Configuration examples

Report examples

Exit codes

--fail-on-warning

Archive-security protections

Architecture

Testing instructions

Security and privacy

Known limitations

Roadmap

Skills demonstrated

Contributing

License

Author

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

`--fail-on-warning`