Raw mirror stores full copies of growing JSONL and SQLite sources on every change

[joedevon]

Summary

The raw mirror stores a complete new blob whenever a mutable source file changes. For large SQLite databases and append-only JSONL transcripts, small updates therefore create another full-file copy instead of storing the changed or appended bytes.

On one macOS archive, two source paths account for 31.93 GB of a 52.46 GB raw mirror:

• Cursor globalStorage/state.vscdb: 28 unique versions totaling 22,567,227,392 bytes. The current file is approximately 806 MB.
• One growing Codex rollout JSONL: 63 unique versions totaling 9,358,615,387 bytes.

Together, those two paths consume 60.9% of the raw mirror.

Evidence

This was measured read-only from the raw-mirror manifests and blob metadata. Identical content is deduplicated, but every changed version has a different content hash and is retained as a separate full blob.

The behavior remains present in v0.6.26 source:

• the capture cache key includes full file size and timestamps;
• capture copies and hashes the complete source file;
• a changed content hash creates a new blob;
• raw-mirror compression is none.

The local archive accumulated under recent 0.6.x releases; I did not run a mutating v0.6.26 reproduction against it.

Why mirror prune is not enough

#221 added a needed cleanup mechanism for unbounded raw-mirror growth. This report is about the amplification that happens before pruning: a frequently appended transcript or mutable database can consume many times its current size even when the underlying conversation content grew only modestly.

Manual age- or size-based pruning can also discard the only preserved version of a conversation after the provider rotates it out, so cleanup alone cannot safely distinguish redundant historical file versions from sole-copy evidence.

Expected behavior

Any of these would bound the amplification while preserving archive-first behavior:

• append-tail capture for verified append-only JSONL files;
• content-defined chunking or delta storage for changed files;
• special handling or exclusion guidance for large mutable SQLite databases;
• a per-source version/size budget with a safe default;
• diagnostics showing per-source amplification before disk use becomes excessive.

At minimum, Cass should warn when stored versions of one source path materially exceed that source's current size.

Environment

• macOS arm64
• Cass archive created across recent 0.6.x releases
• v0.6.26 source inspected at tag v0.6.26

Related: #221

[Dicklesworthstone]

Fixed on main by 402515f893de3a166b029f222bc2be2c257316f9 (the report and fix crossed by roughly one minute; the issue inspected v0.6.26). Large mutable sources now use 4 MiB content-addressed fixed_chunks_v1 storage: unchanged complete JSONL chunks and unchanged SQLite byte regions are reused across versions, while every snapshot retains a full-source BLAKE3 identity and remains byte-exact reconstructable. Existing whole-blob manifests remain readable. Prune understands shared descriptor/chunk references and will not reclaim a chunk while any retained manifest references it; apply mode revalidates the selected manifests and every dependent blob before removal. cass doctor --json also reports per-source amplification (amplified_sources, referenced/current/excess bytes and ratio) without exposing exact source paths. Regression coverage includes growing append, partial-tail replacement, sparse SQLite-like mutation, byte-exact reconstruction, same-size corruption rejection, shared-chunk prune retention/reclamation, and bounded doctor amplification diagnostics. This is merged source behavior, not yet a claim that a new release has been cut; reopen if a current-main capture still writes a whole-file snapshot for a source above the 8 MiB threshold.