Purge oversized binary artifacts and leaked secrets from Git history

[felipebogaertsm]

Goal

Shrink the repository and remove leaked secrets by purging never-should-have-been-committed artifacts from the entire Git history. Today .git is ~63 MB (pack ~39.6 MB) across 1,526 commits and ~150 refs, even though the current HEAD is already clean. Every offender below was deleted from the working tree long ago but still lives in old commits (and in stale local branches that keep the blobs reachable). After the purge, .git should drop to roughly 10–15 MB.

This is a history-rewrite task, not a working-tree cleanup. It cannot be delivered as an ordinary pull request, because rewriting history changes every commit hash on every branch. See "Execution and coordination" below.

⚠️ Security — do this first, independently of the rewrite

This repository is public, and its history exposes real credentials. Rotate these regardless of when the rewrite happens — a history rewrite does not un-expose anything already pushed, cloned, fetched, or indexed:

1. The Gmail account used as EMAIL_HOST_USER, and its hardcoded EMAIL_HOST_PASSWORD, present in four historical settings.py files (web/config/settings.py, backend/srm_solver_app/settings.py, srm_solver_app/srm_solver_app/settings.py, srm_solver_web/srm_solver_web/settings.py). Change the password / revoke the app password.
2. The hardcoded Django SECRET_KEY in srm_solver_web/srm_solver_web/settings.py.
3. srm_solver_web/db.sqlite3 contains a Django auth_user row with a pbkdf2_sha256 password hash. Treat that account's password as compromised.

(The literal values are intentionally not reproduced here to avoid re-indexing them. They are recoverable from history if needed for confirmation.)

What to purge (verified, ranked by cumulative uncompressed size)

Artifact / path	What it is	~Cumulative (uncompressed)
scatter.html, temp-plot.html	Plotly interactive-plot exports committed at the repository root	59.1 MB
output/ (+ legacy mirrors srm_solver/output/, core/output/, srm_solver_web/srm_solver/output/)	Simulation result plots (PNG) and data dumps (CSV/ENG/TXT)	48.2 MB
The built MkDocs site (lives on the stale gh-pages branch): api/, assets/javascripts/, assets/stylesheets/, assets/_mkdocstrings.css, search/, objects.inv, sitemap.xml, sitemap.xml.gz, index.html, 404.html	Generated documentation site; source is docs/**/*.md	11.1 MB
examples/olympus_srm/hot_fire_olympus_1/test_data.csv	Orphaned result CSV (added and removed in consecutive commits)	0.2 MB
poetry.lock	Dead lockfile, orphaned after the migration to uv	2.1 MB
frontend/	Removed Next.js application (lockfiles, favicons, source)	1.3 MB
srm_solver_web/db.sqlite3	Django development database (see Security above)	0.7 MB
web/config/settings.py, backend/srm_solver_app/settings.py, srm_solver_app/srm_solver_app/settings.py, srm_solver_web/srm_solver_web/settings.py	Historical Django settings with hardcoded credentials	small (security, not size)
UI/, unused_files/	Legacy Qt Designer desktop GUI tooling	0.25 MB
.coverage	coverage.py binary cache	0.16 MB
*.pyc, **/__pycache__/**	Compiled bytecode (includes a Dropbox "conflicted copy")	0.08 MB
.idea/	JetBrains per-developer configuration	0.01 MB
assets/images/favicon.png	Stray favicon under an abandoned tree	tiny

What must SURVIVE (do not strip)

The globs above are written to avoid these, but verify they remain after the rewrite:

• uv.lock — the active lockfile (only poetry.lock is being removed).
• docs/assets/logo/** — legitimate brand assets, tracked in HEAD. Note this is under docs/assets/, not the root assets/ that is being stripped.
• machwave/models/propellants/formulations/** — source data (propellant formulation JSON files), tracked in HEAD.

Tooling

Use git-filter-repo (preferred). Install with pipx install git-filter-repo or pip install git-filter-repo. The BFG Repo-Cleaner is an acceptable alternative for glob deletions but cannot handle path prefixes as flexibly.

Procedure (self-contained script)

Run from the repository root, on a clean working tree, with all current work committed or stashed:

#!/usr/bin/env bash
set -euo pipefail

# 0. Preconditions
[ -d .git ] || { echo "Run from the repository root." >&2; exit 1; }
command -v git-filter-repo >/dev/null || { echo "Install git-filter-repo first." >&2; exit 1; }
echo ">>> Size before:"; du -sh .git

# 1. Full mirror backup — do not skip. This is the recovery path.
BACKUP="../machwave-backup.git"
git clone --mirror . "$BACKUP"
echo ">>> Backup mirror created at $BACKUP"

# 2. Stop tracking the still-tracked IDE file (kept on disk via --cached).
#    Commit this with the prevention changes below; the history filter alone
#    will not drop a path that is present in HEAD.
if git ls-files --error-unmatch .vscode/settings.json >/dev/null 2>&1; then
  git rm --cached .vscode/settings.json
fi

# 3. Delete the stale gh-pages branch — the 11 MB MkDocs site is reachable
#    only through it. The path globs below also strip it defensively.
git show-ref --verify --quiet refs/heads/gh-pages && git branch -D gh-pages || true

# 4. Strip the artifacts from ALL history (--invert-paths => delete matches).
git filter-repo --force --invert-paths \
  --path scatter.html \
  --path temp-plot.html \
  --path output/ \
  --path srm_solver/output/ \
  --path core/output/ \
  --path srm_solver_web/srm_solver/output/ \
  --path examples/olympus_srm/hot_fire_olympus_1/test_data.csv \
  --path api/ \
  --path assets/javascripts/ \
  --path assets/stylesheets/ \
  --path assets/_mkdocstrings.css \
  --path search/ \
  --path objects.inv \
  --path sitemap.xml \
  --path sitemap.xml.gz \
  --path index.html \
  --path 404.html \
  --path poetry.lock \
  --path frontend/ \
  --path srm_solver_web/db.sqlite3 \
  --path web/config/settings.py \
  --path backend/srm_solver_app/settings.py \
  --path srm_solver_app/srm_solver_app/settings.py \
  --path srm_solver_web/srm_solver_web/settings.py \
  --path UI/ \
  --path unused_files/ \
  --path .coverage \
  --path .idea/ \
  --path-glob '*.pyc' \
  --path-glob '**/__pycache__/**' \
  --path-glob 'assets/images/favicon.png'

# 5. Reclaim space.
git reflog expire --expire=now --all
git gc --aggressive --prune=now
echo ">>> Size after:"; du -sh .git

Note: git filter-repo intentionally drops the origin remote after running; it is re-added during the force-push step.

Verification (must pass before force-pushing)

• du -sh .git shows roughly 10–15 MB (down from ~63 MB).
• KEEP paths survive in HEAD: git ls-files | grep -E 'uv.lock|docs/assets/logo/|machwave/models/propellants/formulations/' returns all three groups.
• No purged paths remain anywhere in history: e.g. git rev-list --all --objects | grep -E 'scatter.html|/output/|db.sqlite3|poetry.lock|frontend/' returns nothing.
• Source history intact: git log --oneline | wc -l is still in the thousands; recent machwave commits and their messages are present.
• The test suite passes against the rewritten HEAD (see the Makefile for the canonical test target), confirming no source files were removed.

Execution and coordination

• This rewrites all ~150 refs. After verifying, re-add the remote and force-push everything:

  git remote add origin https://github.com/felipebogaertsm/machwave.git
  git push --force --all
  git push --force --tags

• Every collaborator must re-clone; existing clones cannot be fast-forwarded.
• Old commit hashes referenced in issues/PRs will no longer resolve. This is unavoidable with any history rewrite.
• An agent can prepare, run, and verify everything locally, but the actual force-push should be a deliberate, coordinated step taken by the maintainer.

Prevention (a normal pull request, separate from the rewrite)

The history rewrite removes the past; this stops it recurring. The existing .gitignore already covers db.sqlite3, .coverage, *.pyc, __pycache__/, .idea/, and site/. Add the missing rules and stop tracking the IDE file:

# Simulation result/output directories
output/
srm_solver/output/
core/output/
srm_solver_web/srm_solver/output/

# Plotly write_html exports dumped at the repository root
/scatter.html
/temp-plot.html
/plot.html

# Rocket-motor thrust-curve result exports
*.eng

# Editor configuration that should not be shared
.vscode/

# SQLite databases (the existing rule only covers the literal db.sqlite3)
*.sqlite3
*.sqlite
*.db

# coverage.py rotated/parallel data files
.coverage.*

Also run git rm --cached .vscode/settings.json (it is currently tracked in HEAD) and commit it with the .gitignore change. Caveat: machwave/services/eng.py reads and writes .eng files, so confirm no .eng file is a needed fixture before relying on the *.eng rule (none are currently tracked).

Repository conventions for the implementing agent

• Do not add any AI/assistant attribution to commits or pull requests (no Co-Authored-By, no "Generated with" footers).
• For the prevention pull request, use a branch named 374-harden-repo-against-artifact-recurrence (the repository's {issue_number}-short-summary convention).
• Spell things out fully in commits, the pull request, and any comments; avoid abbreviations.

[felipebogaertsm]

Validated the purge end to end on a mirror clone, so nothing in the working repository was rewritten. Three corrections to the procedure in the description.

1. A fifth output mirror was missed. rocketsolver/output/ survives the glob list as written. Add --path rocketsolver/output/. The full set of output prefixes in history is output/, core/output/, srm_solver/output/, rocketsolver/output/, srm_solver_web/srm_solver/output/.

2. git push --force --all and --force --tags are wrong here. Origin has 2 branches and 33 tags; this clone has 150 branches and 56 tags. The blanket flags would publish 150 stale branches and 23 legacy unprefixed tags (0.1.0 through 0.5.0; origin uses v-prefixed tags). Push only what already exists on origin:

git push --force origin master
git push --force origin 'refs/tags/v*:refs/tags/v*'

3. refs/original/refs/heads/268-extract-feed-system-component-specs is left over from an earlier git filter-branch run and keeps old objects reachable. Delete it alongside gh-pages.

Also note rocketsolver/example_data/hot_fire_olympus_1/test_data.csv is the same 0.2 MB file listed under its later examples/ path, but it was genuine package content across many commits rather than an added-and-removed orphan, so I left it in place. Ask if it should go too.

Measured results with the corrected list: .git mirror 74 MB to 3.6 MB. All three keep groups survive in master. No purged path remains anywhere in history. 1156 commits on master, 1733 across all refs. The rewritten master tree hash is byte identical to the current one (828de14f8b0bbfe777ffb1c26b2e859ab50da19e), so the working tree is provably untouched; make test gives 1044 passed, 2 skipped.

Prevention pull request: #475. It is deliberately narrower than the description proposed, see the pull request for why.

Still outstanding and not something the rewrite addresses: the credential rotation in the Security section above.