Skip to content

Split tweet_scraper.py into a config/query/api/cli package#10

Merged
cybermejo merged 2 commits into
worktree-scraper-package-11from
worktree-scraper-split-package
Jun 18, 2026
Merged

Split tweet_scraper.py into a config/query/api/cli package#10
cybermejo merged 2 commits into
worktree-scraper-package-11from
worktree-scraper-split-package

Conversation

@cybermejo

Copy link
Copy Markdown
Owner

Summary

Splits the 630-line tweet_scraper.py into a config / query / api / cli package — the last remaining README Roadmap item for the scraper. Behavior-preserving: same 65 tests pass.

Stacked on #9 (base worktree-scraper-package-11, the logging+packaging PR). GitHub will retarget this to main once #9 merges.

Layout (flat public API preserved)

from tweet_scraper import X and the tweet_scraper:main entry point are unchanged — __init__.py re-exports the full public API (including the _underscore helpers the tests import).

Module Contents
config.py DEFAULTS, _CONFIG_TYPES, validate_config, load_config, date parsing
query.py build_query, handle_from_url
api.py endpoint URLs, get_api_key, _extract_*, paged_get (retry/cursor), search_tweets, user_tweets, tweet_replies
cli.py CSV_COLUMNS, flatten, streaming writers, main
__main__.py enables python -m tweet_scraper

Mock-patch retargeting (the tricky part)

The suite has 20 patch targets on the flat namespace. Since the patched objects moved into api, the strings were retargeted to where names are actually looked up — tweet_scraper.api.{requests,time,paged_get,tweet_replies} — and cli calls fetchers via the api. namespace so the patches intercept.

Verification (beyond the green loop)

  • Falsification test: deliberately pointing one patch at a wrong target makes the tests fail, and restoring it makes them pass — proving the mocks genuinely intercept rather than passing inertly.
  • 3-agent adversarial review: confirmed byte-for-byte logic equivalence vs. the original, complete public-API re-export, and an acyclic import graph with correct patch indirection. Two minor findings it raised (missing python -m entry, stale docs) are fixed in this PR.

Also updated

  • CI mypy tweet_scraper.pymypy tweet_scraper.
  • README + CLAUDE.md: invocation → python -m tweet_scraper, project-structure tree, data-flow, and Roadmap (shipped items marked done).

Testing

Full CI-equivalent loop green from scraper/: ruff, ruff format --check, mypy tweet_scraper (6 files), pytest --cov65 tests pass. python -m tweet_scraper --dry-run verified without install.

Christian-Bermejo added 2 commits June 18, 2026 10:30
The 630-line single module is now a package with the same flat public API
(re-exported from __init__, so 'from tweet_scraper import X' and the
'tweet_scraper:main' entry point are unchanged):

  config  vars.json load/validate + date parsing
  query   build_query + handle extraction
  api     HTTP client: get_api_key, paged_get (retry/cursor), fetchers
  cli     flatten -> CSV, streaming writes, main

Behavior-preserving — same 65 tests pass. Test mock patch targets were retargeted
to where names are now looked up (tweet_scraper.api.requests/.time/.paged_get/
.tweet_replies); cli calls fetchers via the api namespace so those patches resolve.
CI mypy target updated tweet_scraper.py -> tweet_scraper; CLAUDE.md architecture
note updated for the package layout.
Adversarial verification flagged two minor regressions from the split:
- The original supported 'python tweet_scraper.py'; the package had no __main__,
  so the README's no-install run path broke. Add __main__.py so
  'python -m tweet_scraper' works (mirrors the tweet-scraper console script).
- README + CLAUDE.md still said 'python tweet_scraper.py' and 'mypy
  tweet_scraper.py', and the project-structure tree / data-flow / Roadmap were
  stale. Update invocation to 'python -m tweet_scraper', mypy target to the
  package, the structure tree to the new modules, and mark the now-shipped
  Roadmap items done.

Full loop green: 65 tests, mypy on 6 files, python -m verified.
@cybermejo
cybermejo merged commit ed9de01 into worktree-scraper-package-11 Jun 18, 2026
4 checks passed
cybermejo pushed a commit that referenced this pull request Jun 18, 2026
PR #10 (config/query/api/cli split + __main__.py) had merged into its stacked
base branch but not into main, which still carried the flat tweet_scraper.py.
Bring the split to main. Touches only scraper/CI/docs — no Studio files, so the
PR #6/#7 Studio work is untouched. CLAUDE.md auto-merged (kept the PR #8 Studio
note + the split's python -m updates). Validation loop green: 65 tests, mypy on
the package, python -m tweet_scraper verified.
@cybermejo
cybermejo deleted the worktree-scraper-split-package branch June 18, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant