Repair report and version indexes and clean up stale Litestream files#922
Draft
epompeii wants to merge 1 commit into
Draft
Repair report and version indexes and clean up stale Litestream files#922epompeii wants to merge 1 commit into
epompeii wants to merge 1 commit into
Conversation
Add a migration that rebuilds all indexes on the report and version tables. Long-lived databases can carry index corruption (wrong number of entries in an index) that silently drops rows from index-driven queries. REINDEX is a no-op when the indexes are already consistent. The migration also removes report_benchmark rows (and their dependent metric, boundary, and alert rows) orphaned by deletes performed before ON DELETE CASCADE was enforced (see 2023-01-15-185835_perf_cascade). Migrations run with foreign_keys = OFF, so the delete chain is explicit and bottom-up, using temp tables to scan each unindexed table only once. At API server startup, remove stale Litestream files: the generations state from Litestream 0.3 that 0.5+ no longer reads, and leftover restore output files from an interrupted litestream restore.
Contributor
|
Bencher Report| Branch | reindex-report-version |
| Testbed | intel-v1 |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result microseconds (µs) (Result Δ%) | Upper Boundary microseconds (µs) (Limit %) |
|---|---|---|---|
| Adapter::Json | 📈 view plot 🚷 view threshold | 4.60 µs(-0.50%)Baseline: 4.62 µs | 4.90 µs (93.82%) |
| Adapter::Magic (JSON) | 📈 view plot 🚷 view threshold | 4.45 µs(-1.02%)Baseline: 4.50 µs | 4.72 µs (94.29%) |
| Adapter::Magic (Rust) | 📈 view plot 🚷 view threshold | 25.92 µs(+1.48%)Baseline: 25.54 µs | 26.64 µs (97.29%) |
| Adapter::Rust | 📈 view plot 🚷 view threshold | 3.53 µs(+1.19%)Baseline: 3.49 µs | 3.60 µs (97.97%) |
| Adapter::RustBench | 📈 view plot 🚷 view threshold | 3.53 µs(+1.24%)Baseline: 3.48 µs | 3.59 µs (98.15%) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migration:
reindex_report_versionLong-lived databases can carry index corruption (
wrong # of entries in index) that silently drops rows from index-driven queries, while the underlying table data remains intact. This migration rebuilds all indexes on thereportandversiontables.REINDEXis a no-op when the indexes are already consistent, so this is safe for every instance.The migration also removes
report_benchmarkrows orphaned by deletes performed beforeON DELETE CASCADEwas enforced (see2023-01-15-185835_perf_cascade), along with their dependentmetric,boundary, andalertrows. Migrations run withforeign_keys = OFF, so cascades do not fire and the delete chain is explicit, bottom-up. Sincemetric.report_benchmark_idandboundary.metric_idare not indexed, the orphaned ids are collected into temp tables so each table is scanned only once.Validated end-to-end on a multi-gigabyte database exhibiting this corruption: the migration completes in ~15 seconds, after which
PRAGMA quick_checkreturnsokandPRAGMA foreign_key_checkreturns no rows, with all report rows intact.Startup cleanup: stale Litestream files
Litestream 0.3 kept its state in a
generations/directory and agenerationpointer file; Litestream 0.5+ only readsltx/. Instances that upgraded through 0.3 carry that dead state on the data volume forever. An interruptedlitestream restorecan also leave<db>.tmp*output files behind.The API server now removes these stale files before spawning Litestream, when nothing has them open. Failures are logged and ignored; cleanup never blocks startup. Covered by unit tests, including that the live
ltx/state and the database itself are untouched.