fix(storage): a missing SSTable component is a defect, not a later problem

[bkearns]

The defect

Startup SSTable repair checked whether a critical component was zero-byte and quarantined the generation if so, but let a missing component fall through:

Ok(meta) if meta.len() == 0 => { quarantine = true; }
Err(_) => {
    // Missing component — will fail in open_sstable_from_dir.
}

It does fail there. What reaches the operator is:

storage: I/O error: No such file or directory (os error 2)

No table, no generation, no path — and it fails every read of that table, not just the one generation. Missing is strictly worse than zero-byte, and it was the one case not caught.

How it showed up

On the ferrosa-memory native cluster, agent_memory.mobile_control_cursor_state and agent_memory.knowledge_by_state were unreadable on all three nodes. 87 of 89 tables were fine, including the 80,917-entity corpus, so this was not general corruption — it was two tables whose manifests referenced generations whose component files were gone.

The downstream damage is what made it expensive to diagnose. The cursor allocator lost its recent value, reverted to an older surviving generation, and regressed ~10,600 cursors. A mobile client then refused a high-water below the cursor it had already committed, and its event feed went silent. The client's guard was correct; the store had moved backwards underneath it. Working that out took reading generation directories on disk, because the error named nothing.

The change

The decision moves into sstable_health.rs as a pure function over probed components. It needed no engine, manifest or disk — only the directory walk around it did — and it was the part that was wrong.

• missing, zero-byte, and unreadable all withhold the generation
• Rows.db stays out of the critical set: the writer legitimately emits it empty for simple partitions, and treating it as critical would quarantine healthy SSTables
• both startup log lines now name the component and the reason, so the next occurrence doesn't require disk archaeology

Tests

Written against the old behaviour first, so the defect showed up as red rather than as an assertion I made about it. Two failed before the fix:

a_missing_data_file_is_unusable
  left: None                             right: Some(("Data.db", Missing))

the_first_defect_in_order_is_the_one_reported
  left: Some(("Partitions.db", Empty))   right: Some(("Data.db", Missing))

The second is the sharper one: given a missing Data.db and a zero-byte Partitions.db, the old logic stepped over the missing file and reported the empty one.

8 cases total, covering the healthy generation, both critical components, each defect kind, deterministic ordering when several are defective, and a guard that Rows.db is not critical.

Verification

• cargo test -p ferrosa-storage --lib — 1089 passed, 0 failed, 0 ignored
• cargo fmt --check — passes
• cargo clippy -p ferrosa-storage --all-targets -- -D warnings — passes

Crate docs updated as the repo requires: README.md gains the module, and specs/fmea.md gains ST-18 (RPN 256 → 24).

What this does not fix

Stated plainly, and recorded in ST-18:

• The manifest is not reconciled. The quarantined generation stays referenced, so it is re-probed and re-quarantined on every boot. Idempotent, but it means the manifest and disk stay disagreeing.
• The cause of the missing files is still unknown. This fix stops a manifest/disk divergence from taking out a whole table and makes it say so; it does not explain what removed the components. That investigation is tracked separately (t_d573e7c3), including whether the absent S3 path matters — the compaction warning explicitly mentions "no remote component length hook confirmed object-storage availability", and this cluster has no working upload path, so there is no remote copy to fall back to.

[bkearns]

Root cause found, and it is a second instance of the same asymmetry — pushed as a7927b6f.

The trigger

Ben's hint (new tables, look at what changed) sent me to the on-disk layout, which was a dead end — plenty of healthy tables mix flat and nested generations. The node log had it instead:

compaction: swap complete table_id=agent_memory.mobile_control_cursor_state
  pre_swap_count=6 post_swap_count=3 removed=4

Three SSTables in the post-swap set, and one generation on disk. The other two are 1787845599164710 and 1787844609442814 — exactly the pair the logs report skipping every single cycle:

compaction planning: skipping SSTable because required on-disk component files
are missing or empty and no remote component length hook confirmed
object-storage availability

Why it was permanent

That skip is return None out of the planning closure. It drops the SSTable from that compaction plan and nothing else:

• it stays in the live read set, so every read opens a file that is gone → bare ENOENT, whole query fails
• planning skips it again next cycle, so it is never compacted away

Immortal. Two tables sat like that for days while a complete generation sat on disk next to them. That is why a restart would have "fixed" it and why I initially reached for DROP — both were treating the symptom.

Note the sibling branch: when sstable_compaction_remote_component_available confirms an object-storage copy, compaction rehydrates instead. This cluster has no working S3 upload path, so that branch is never taken — which is why the fault landed here and not somewhere with S3 wired up.

The fix

Planning now quarantines before skipping, reusing the existing in-memory quarantine that the read path already consults in three places. Reads skip the generation instead of failing on it, and the quarantine record carries the token range so anti-entropy repair can refill it.

The test

store::tests::planning_quarantines_an_sstable_whose_components_are_gone builds the phantom directly: a valid in-memory reader whose on-disk directory holds no components.

Before the fix its first assertion passed — planning does skip it — and the second failed. That gap is the defect, stated as a test.

Full crate suite: 1090 passed, 0 failed, 0 ignored. fmt and clippy -D warnings clean. FMEA ST-19 (RPN 288 → 24), README updated.

Scope of the two commits

• 74ea45e8 — startup: a missing component is a defect, not a later problem
• a7927b6f — runtime: a file-less SSTable is quarantined, not merely skipped

The first covers the boot path, the second the running engine. Neither explains what removed the component files — that is still open on t_d573e7c3. These stop a manifest/disk divergence from silently becoming a permanently unreadable table.