Stream tweets to CSV as fetched (crash-safe, lower memory)#1
Merged
Conversation
Previously main() accumulated every row in an in-memory list and wrote the CSV only after all scraping finished. A mid-run failure (network error, rate-limit exhaustion, Ctrl-C) therefore lost the entire run despite credits already spent, and memory grew with result size. Now the DictWriter is opened up front inside a with-block and each deduped row is written as it arrives, so the file is flushed/closed (preserving partial output) when an exception unwinds. Only the seen-id set and a list of reply-target ids are held in memory, not the full row dicts. - Add _write_new (dedup + write) and _stream_replies, replacing the mutate-a-list _collect_replies; fold the triplicated dedup into _stream_query. - Output ordering (search, then user, then replies) is unchanged. - Add end-to-end main() tests incl. partial-output-survives-crash; replace the _collect_replies unit tests with _write_new/_stream_replies tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Streams scraped tweets to the output CSV as they are fetched, instead of buffering every row in memory and writing once at the end.
Why
The old
main()accumulated all rows in an in-memory list and only opened the CSV after all scraping finished. A mid-run failure (network error, rate-limit exhaustion, Ctrl-C) therefore lost the entire run despite credits already spent, and memory grew with the result size.What changed
main()opens thecsv.DictWriterup front inside awithblock and writes each deduped row immediately; the block flushes/closes the file as an exception unwinds, so already-fetched tweets survive a crash.seenid set + a small list of reply-target ids are held in memory — not full row dicts._write_newand_stream_repliesreplace the mutate-a-list_collect_replies; this also folds away the triplicated dedup logic.Testing
Full validation loop green from
scraper/:ruff check,ruff format --check,mypy,pytest --cov— 56 passed. Added end-to-endmain()tests includingtest_main_persists_rows_fetched_before_a_midrun_crash, which fails on the old code and passes now (proves the crash-safety guarantee). Tests are fully mocked (no network).