Parse large CSVs off the main thread (PapaParse worker)#7
Merged
Conversation
parseCSV ran Papa.parse synchronously, so a big paste/upload froze the UI while parsing. Enable worker:true above a 512KB threshold so large inputs parse on a background thread; small inputs keep the synchronous main-thread path (the worker's serialization overhead makes it slower for them). PapaParse falls back to synchronous parsing when no Worker global exists (verified in Node), so the existing tests and any non-browser use are unaffected. The Promise contract and the normalize/enrich post-processing are unchanged. Adds a >512KB parse test (30k rows) exercising the worker path. 41 tests green.
cybermejo
pushed a commit
that referenced
this pull request
Jun 18, 2026
PR #10 (config/query/api/cli split + __main__.py) had merged into its stacked base branch but not into main, which still carried the flat tweet_scraper.py. Bring the split to main. Touches only scraper/CI/docs — no Studio files, so the PR #6/#7 Studio work is untouched. CLAUDE.md auto-merged (kept the PR #8 Studio note + the split's python -m updates). Validation loop green: 65 tests, mypy on the package, python -m tweet_scraper verified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parses large CSVs on a background worker so a big paste/upload no longer freezes the UI.
Why
parseCSVcalledPapa.parsesynchronously on the main thread. A large file blocked rendering until parsing finished.What changed
worker: trueonly when the input exceeds 512 KB. Small inputs keep the synchronous main-thread path — the worker's serialization overhead makes it slower for them, and the freeze only matters for large files.Promisecontract and thenormalize→filter→enrichpost-processing are unchanged (the worker only does the parse; enrichment still runs on the main thread on the parsed rows).Safety / compatibility
Workerglobal exists (Node / vitest), PapaParse falls back to synchronous parsing and still firescompletewith correct rows — so existing tests and any non-browser use are unaffected.Testing
Adds a >512 KB parse test (30k rows) that exercises the worker path end-to-end (full normalize/enrich pipeline, correct row count + values). 41 tests pass; build green.