Switch CSV writer to per-ticker layout with overwrite guard

[cswaney]

Goal

Replace the current date-only, append-based CSV write strategy with a per-ticker, fresh-write layout plus an overwrite guard. This makes a parse run's output deterministic (independent of prior runs), prevents silent data accumulation, and gives tvi convert (#12) well-defined per-ticker inputs to combine.

Problem with current behavior

Each flush writes data/{kind}/{date}.csv opened create(true).append(true) (src/writer/csv.rs). The filename has no ticker component, so:

• Re-running tvi parse on the same date with a different --tickers set appends, accumulating rows across runs with no warning.
• What ends up in a CSV depends on run history, which makes the parquet skip rule in Add tvi convert subcommand (CSV → Parquet) #12 unsafe (a later run can yield a stale or subset parquet silently).

New layout

data/orders/{date}/{ticker}.csv
data/trades/{date}/{ticker}.csv
data/noii/{date}/{ticker}.csv
data/books/{date}/{ticker}.csv

• Files written fresh (truncate), never appended.
• --tickers * writes a single combined data/{kind}/{date}/_all.csv rather than fanning out to one file per (~8000) symbols. Per-ticker files are produced only when tickers are explicitly listed.

Overwrite guard

Before parsing, resolve the ticker set (known up front from --tickers) and stat the target {ticker}.csv files:

• If any exist and --overwrite is not set → error, listing the colliding files. No partial work is done.
• --overwrite is per-file (surgical): tvi parse --tickers MSFT --overwrite replaces only MSFT.csv, leaving AAPL.csv untouched.
• For *, the guard checks {date}/_all.csv existence.

Add an --overwrite flag to tvi parse (src/main.rs Cli).

Scope of change

• src/writer/csv.rs: path construction ({kind}/{date}/{ticker}.csv), open mode create_new/truncate instead of append, header always written (file is always fresh). The writer needs the ticker in scope per row/batch — currently it derives the filename from date only and serializes whichever ticker's messages arrive. Decide whether to (a) group messages by ticker before flush, or (b) key writers by ticker.
• src/main.rs: add --overwrite, perform the up-front collision check, resolve the combined-file path for *.

Books caveat

Book CSVs are variable-width by levels_count / --depth (src/writer/csv.rs). Per-ticker book files in the same date dir written with different --depth across runs would disagree in width; the overwrite guard prevents the same-ticker case, but convert-combine (#12) must reconcile or refuse on width mismatch.

Backward incompatibility

Output directory structure changes ({date}.csv → {date}/{ticker}.csv). Existing downstream consumers of the flat layout must be updated. Flag for the v0.2.0 notes.

Relationship

• Prerequisite for Add tvi convert subcommand (CSV → Parquet) #12 (tvi convert combines {date}/*.csv → {date}.parquet).
• Feeds the overwrite/auto-convert semantics in Add --format parquet to tvi parse (auto-convert) #13.

[cswaney]

Writer restructure design (from review)

Notes on how the per-ticker layout is implemented in the writer, settled during code review. Covers the buffering structure and the memory-bounding eviction strategy.

Per-ticker buffering

Today Writer holds a flat Vec per message kind (src/writer.rs), flushed in size-based batches; within a batch, messages are interleaved across tickers, so writing per-ticker files would mean re-grouping an interleaved batch and opening N files on every flush (open-churn).

Instead, buffer per ticker in the writer:

• HashMap<String, Vec<OrderMessage>> keyed by ticker (and likewise for trades, noii, snapshots).
• Flush iterates the map and writes each ticker's vec to its own file — no per-flush regrouping.
• --tickers * collapses to a single _all key (one logical "ticker"), so the map has one entry and the wide-fanout concerns below don't apply to the wildcard case. Per-ticker vecs only proliferate under an explicit, large ticker list.

Memory bound — why naive per-ticker thresholds fail

A per-ticker flush threshold (flush a vec when it crosses buffer_size) gives the fattest writes but makes total buffered memory scale with ticker count: buffer_size × active_tickers. Worst case is ~8000 explicit tickers each holding ~buffer_size rows — unbounded for practical purposes.

So use an aggregate ceiling (total buffered rows across all vecs), not a per-vec threshold. Memory is then bounded by the ceiling regardless of ticker count.

Eviction strategy (two-tier)

When the aggregate ceiling trips:

1. Normal sweep: scan the map once and flush every vec whose length ≥ min_flush_size, until total buffered drops to a target (below the ceiling). No sorted structure, no heap — a single O(n) linear scan with a length comparison. min_flush_size is the knob that keeps thin vecs (e.g. 3-row tickers) from triggering file opens: a file open is only "worth it" for a sufficiently fat write.

2. Pressure fallback: the normal sweep can fail to reach target if too many vecs sit just under min_flush_size (thousands of tickers each holding min_flush_size − 1 rows → nothing qualifies, memory still over ceiling). When the sweep doesn't get under the ceiling, lower the threshold / flush-all to restore the bound, accepting thinner writes. This path is rare and only fires under genuine wide-fanout pressure — exactly where thin writes are an acceptable cost for staying bounded.

This keeps the common case (a handful of explicit tickers, or * → one _all vec) on the cheap fat-write path, while preserving a hard memory bound in the pathological wide-fanout case.

Rejected alternatives

• Maintain vecs sorted by length (BTree/re-sort): taxes every push (hot path) to make rare eviction O(1). Wrong cost asymmetry — pushes are constant, eviction is rare.
• Binary heap keyed by length: O(log n) pop-largest without scanning, but pushing onto an existing vec changes its length and staleness its heap position; BinaryHeap can't re-prioritize in place, so it needs lazy stale-entry handling or an indexed PQ. More machinery than a rarely-fired path warrants.
• Evict only the single largest vec per trigger: drops just under the ceiling, so the next push re-trips it → scan-per-push in steady state. Batch eviction to a target avoids this.

Append vs. truncate note

Even with per-ticker vecs, a long parse flushes the same ticker's file multiple times, so within a run the file is opened append across flushes. The #15 "fresh write" guarantee comes from the up-front overwrite guard (create-new/truncate on first touch, or the pre-run collision check), not from the open mode. Intra-run flushes still append to that run's own files.

Prerequisite tweak

OrderMessage derives Getters with #[getset(get = "pub")] on date only (src/message.rs), so it has no public ticker(). The per-ticker grouping needs it — add a getter for ticker (TradeMessage/NOIIMessage already expose ticker() via struct-level get = "pub").