Scraper: structured logging + packaging (console entry point)#9
Merged
Conversation
added 2 commits
June 18, 2026 09:47
Progress/warning output went through print(..., file=sys.stderr). Switch to a module logger configured (INFO -> stderr) in main(), so callers/importers control verbosity and handlers. Dry-run output stays on stdout via print() — it is the program's actual result, not progress chatter. Updates the unknown-key warn test from capsys to caplog. 59 tests green.
pyproject.toml was tooling-config only, with a comment deferring packaging. Add a [project] table (name/version/deps/requires-python), a setuptools build-system, and a console_scripts entry point so 'pip install -e .' exposes a 'tweet-scraper' command. Verified the CLI installs and the entry point runs --dry-run. Validation loop still green (59 tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Knocks out two of the three remaining README Roadmap items for the scraper: structured logging and packaging. (The third — splitting
tweet_scraper.pyinto a package — is deliberately deferred to its own PR; see note below.)1. Logging
print(..., file=sys.stderr)progress/warning calls with a module logger (logging.getLogger("tweet_scraper")), configuredINFO → stderrinmain().stdoutviaprint()— it's the program's actual result, not progress chatter, so it remains pipeable.capsystocaplog.2. Packaging
[project]table (name/version/deps/requires-python), a setuptoolsbuild-system, and a[project.scripts]console entry point.pip install -e .now exposes atweet-scrapercommand. Verified end-to-end: installs in a clean venv and the entry point runs--dry-run.Testing
Full validation loop green from
scraper/(ruff,ruff format --check,mypy,pytest) — 59 tests pass.Deferred (separate PR)
Splitting the 630-line module into
config/api/query/clisubmodules is not included here. It would force rewriting ~20 test mock patch targets (patch("tweet_scraper.requests.get")etc.) since those names would move into submodules — a test-internals change worth isolating in its own reviewable PR. The module's seams are already clean, so it stays cheap to do later.