Split tweet_scraper.py into a config/query/api/cli package

.github/workflows/ci.yml

@@ -36,7 +36,7 @@ jobs:
         run: ruff format --check .
 
       - name: Type check (mypy)
-        run: mypy tweet_scraper.py
+        run: mypy tweet_scraper
 
       - name: Tests (pytest + coverage)
         run: pytest --cov=tweet_scraper --cov-report=term-missing

CLAUDE.md

@@ -7,19 +7,22 @@ hard rules come first because they matter most.
 
 A two-part tweet toolkit (monorepo — two independent subprojects):
 
-- **Scraper** — `scraper/` (`tweet_scraper.py`), a Python 3.11+ CLI that pulls
-  tweets from twitterapi.io (pay-per-use) into a flat CSV. Config schema mirrors
-  Apify Tweet Scraper V2.
+- **Scraper** — `scraper/` (`tweet_scraper/` package), a Python 3.11+ CLI that
+  pulls tweets from twitterapi.io (pay-per-use) into a flat CSV. Config schema
+  mirrors Apify Tweet Scraper V2.
 - **Studio** — `studio/` (Vite + React 18 + Tailwind 4), a browser app that
   ingests the CSV for tone/sentiment tagging and draft assistance.
 
-Data flow: `scraper/vars.json → scraper/tweet_scraper.py → tweets.csv → Tweet Studio`.
+Data flow: `scraper/vars.json → python -m tweet_scraper → tweets.csv → Tweet Studio`.
 
 ## Architecture
 
-- `scraper/tweet_scraper.py` — single-module CLI. Key seams: `build_query`
-  (search syntax), `paged_get` (HTTP + retry + cursor paging), `flatten`
-  (tweet → CSV row), `main` (CLI orchestration).
+- `scraper/tweet_scraper/` — the CLI package (flat public API preserved via
+  `__init__` re-exports, so `from tweet_scraper import X` still works). Modules:
+  `config` (vars.json load/validate + date parsing), `query` (`build_query`,
+  handle extraction), `api` (HTTP client: `paged_get` retry/cursor paging,
+  per-endpoint fetchers — patch HTTP here, e.g. `tweet_scraper.api.requests`),
+  `cli` (`flatten` tweet → CSV row, streaming writes, `main`).
 - `scraper/tests/` — pytest, fully mocked (no network, no API key).
   `scraper/conftest.py` puts the scraper dir on `sys.path`.
 - `studio/src/App.jsx` — the entire Studio UI (lexicons, tone patterns, CSV upload).
@@ -45,7 +48,7 @@ Run the full loop from `scraper/` and make sure every step passes:
 
 1. `ruff check .` — lint, zero errors
 2. `ruff format --check .` — formatting
-3. `mypy tweet_scraper.py` — type check
+3. `mypy tweet_scraper` — type check
 4. `pytest --cov=tweet_scraper` — all tests pass
 
 (CI runs the same four steps from `scraper/` — see `.github/workflows/ci.yml`.)
@@ -54,7 +57,7 @@ Run the full loop from `scraper/` and make sure every step passes:
 
 - When I say **"validate"** or **"ship it"** → run all four Validation steps and
   report results before doing anything else.
-- When I say **"dry run"** → `python tweet_scraper.py --dry-run` (never spends credits).
+- When I say **"dry run"** → `python -m tweet_scraper --dry-run` (never spends credits).
 - When I say **"scrape"** → `--dry-run` first, show me the queries, wait for OK,
   then run for real.
 

README.md

@@ -9,7 +9,7 @@ A two-part toolkit for collecting tweets and turning them into content insights:
    browser and does tone/sentiment tagging, engagement stats, and draft assistance.
 
 ```
-scraper/vars.json ──▶ scraper/tweet_scraper.py ──▶ tweets.csv ──▶ Tweet Studio ──▶ insights
+scraper/vars.json ──▶ python -m tweet_scraper ──▶ tweets.csv ──▶ Tweet Studio ──▶ insights
 ```
 
 The two halves are independent (separate toolchains); work on them from their
@@ -42,15 +42,18 @@ pip install -r requirements.txt          # runtime dep: requests
 export TWITTERAPI_IO_KEY="your_key_here"  # key comes from the env, never from a file
 
 cp vars.example.json vars.json            # then edit vars.json for your run
-python tweet_scraper.py                   # reads ./vars.json, writes ./tweets.csv
+python -m tweet_scraper                   # reads ./vars.json, writes ./tweets.csv
 ```
 
+(`pip install -e .` also installs a `tweet-scraper` console command equivalent to
+`python -m tweet_scraper`.)
+
 Options:
 
 ```bash
-python tweet_scraper.py --config path/to.json   # use a different config
-python tweet_scraper.py --out results.csv        # change output path
-python tweet_scraper.py --dry-run                # print the queries, don't call the API (free)
+python -m tweet_scraper --config path/to.json   # use a different config
+python -m tweet_scraper --out results.csv        # change output path
+python -m tweet_scraper --dry-run                # print the queries, don't call the API (free)
 ```
 
 **Always `--dry-run` first** — it shows the exact queries that would be billed,
@@ -112,7 +115,7 @@ pip install -r requirements-dev.txt
 
 ruff check .                 # lint
 ruff format --check .        # formatting
-mypy tweet_scraper.py        # type check
+mypy tweet_scraper           # type check
 pytest --cov=tweet_scraper   # tests + coverage
 ```
 
@@ -129,12 +132,18 @@ pip install pre-commit && pre-commit install
 
 ```
 scraper/                  # Python CLI scraper
-  tweet_scraper.py        # the CLI
+  tweet_scraper/          # the CLI package (flat public API via __init__)
+    __init__.py           #   re-exports the public API
+    __main__.py           #   enables `python -m tweet_scraper`
+    config.py             #   vars.json load/validate + date parsing
+    query.py              #   build_query + handle extraction
+    api.py                #   twitterapi.io HTTP client (paging/retry, fetchers)
+    cli.py                #   flatten → CSV, streaming writes, main
   tests/                  # pytest suite (no network / no API key needed)
   conftest.py             # makes tweet_scraper importable from tests
   requirements.txt        # runtime deps
   requirements-dev.txt    # dev/validation deps
-  pyproject.toml          # ruff / mypy / pytest config (tooling only)
+  pyproject.toml          # [project] packaging + ruff / mypy / pytest config
   vars.example.json       # template config — copy to vars.json (git-ignored)
   tweets.sample.csv       # tiny anonymized example output
 studio/                   # Tweet Studio (Vite + React)
@@ -146,10 +155,13 @@ CLAUDE.md                 # AI-assistant rules & validation loop
 
 ## Roadmap
 
-- Packaging: add a `[project]` table + console entry point (`tweet-scraper`).
-- Split `tweet_scraper.py` into a package as it grows (config / api / query / cli).
-- Config schema validation (reject unknown keys / bad types early).
-- Move to the `logging` module instead of `print(..., file=sys.stderr)`.
+Done: packaging (`[project]` + `tweet-scraper` console entry point), the
+`config / query / api / cli` package split, config-schema validation, and the
+move to the `logging` module. Possible next steps:
+
+- Tweet Studio: list virtualization for very large CSVs (the Tweets tab caps the
+  rendered list and shows a "top N of M" notice today).
+- Retry/backoff tuning and configurable rate limits for `paged_get`.
 
 ## License
 

scraper/pyproject.toml

@@ -14,7 +14,7 @@ dependencies = ["requests>=2.31.0"]
 tweet-scraper = "tweet_scraper:main"
 
 [tool.setuptools]
-py-modules = ["tweet_scraper"]
+packages = ["tweet_scraper"]
 
 [tool.ruff]
 line-length = 100

scraper/tests/test_main_streaming.py

@@ -59,7 +59,7 @@ def test_main_writes_scraped_tweets_to_csv(tmp_path):
     page = FakeResp(200, {"tweets": [_tweet("1"), _tweet("2")], "has_next_page": False})
     with (
         patch.dict(os.environ, {"TWITTERAPI_IO_KEY": "k"}),
-        patch("tweet_scraper.requests.get", return_value=page),
+        patch("tweet_scraper.api.requests.get", return_value=page),
     ):
         rc = _run_main(config, out)
     assert rc == 0
@@ -72,7 +72,7 @@ def test_main_writes_user_timeline_tweets(tmp_path):
     page = FakeResp(200, {"tweets": [_tweet("1"), _tweet("2")], "has_next_page": False})
     with (
         patch.dict(os.environ, {"TWITTERAPI_IO_KEY": "k"}),
-        patch("tweet_scraper.requests.get", return_value=page),
+        patch("tweet_scraper.api.requests.get", return_value=page),
     ):
         rc = _run_main(config, out)
     assert rc == 0
@@ -87,8 +87,8 @@ def test_main_persists_rows_fetched_before_a_midrun_crash(tmp_path):
     side_effect = [page1] + [requests.ConnectionError()] * 4
     with (
         patch.dict(os.environ, {"TWITTERAPI_IO_KEY": "k"}),
-        patch("tweet_scraper.time.sleep"),
-        patch("tweet_scraper.requests.get", side_effect=side_effect),
+        patch("tweet_scraper.api.time.sleep"),
+        patch("tweet_scraper.api.requests.get", side_effect=side_effect),
         pytest.raises(requests.ConnectionError),
     ):
         _run_main(config, out)
@@ -104,7 +104,7 @@ def test_main_streams_replies_after_main_tweets(tmp_path):
     reply_page = FakeResp(200, {"tweets": [_tweet("99")], "has_next_page": False})
     with (
         patch.dict(os.environ, {"TWITTERAPI_IO_KEY": "k"}),
-        patch("tweet_scraper.requests.get", side_effect=[search_page, reply_page]),
+        patch("tweet_scraper.api.requests.get", side_effect=[search_page, reply_page]),
     ):
         rc = _run_main(config, out)
     assert rc == 0
@@ -119,7 +119,7 @@ def test_main_skips_reply_fetch_when_reply_count_zero(tmp_path):
     search_page = FakeResp(200, {"tweets": [_tweet("1", replyCount=0)], "has_next_page": False})
     with (
         patch.dict(os.environ, {"TWITTERAPI_IO_KEY": "k"}),
-        patch("tweet_scraper.requests.get", side_effect=[search_page]) as mock_get,
+        patch("tweet_scraper.api.requests.get", side_effect=[search_page]) as mock_get,
     ):
         rc = _run_main(config, out)
     assert rc == 0

scraper/tests/test_paging.py

@@ -63,7 +63,7 @@ def test_extract_pagination_missing_is_false_empty():
 
 def test_paged_get_single_page():
     resp = FakeResp(200, {"tweets": [{"id": "1"}, {"id": "2"}], "has_next_page": False})
-    with patch("tweet_scraper.requests.get", return_value=resp):
+    with patch("tweet_scraper.api.requests.get", return_value=resp):
         out = list(paged_get("http://x", {}, "key", limit=10))
     assert [t["id"] for t in out] == ["1", "2"]
 
@@ -72,15 +72,15 @@ def test_paged_get_respects_limit():
     resp = FakeResp(
         200, {"tweets": [{"id": "1"}, {"id": "2"}, {"id": "3"}], "has_next_page": False}
     )
-    with patch("tweet_scraper.requests.get", return_value=resp):
+    with patch("tweet_scraper.api.requests.get", return_value=resp):
         out = list(paged_get("http://x", {}, "key", limit=2))
     assert [t["id"] for t in out] == ["1", "2"]
 
 
 def test_paged_get_follows_cursor_across_pages():
     page1 = FakeResp(200, {"tweets": [{"id": "1"}], "has_next_page": True, "next_cursor": "c1"})
     page2 = FakeResp(200, {"tweets": [{"id": "2"}], "has_next_page": False})
-    with patch("tweet_scraper.requests.get", side_effect=[page1, page2]) as mock_get:
+    with patch("tweet_scraper.api.requests.get", side_effect=[page1, page2]) as mock_get:
         out = list(paged_get("http://x", {"query": "q"}, "key", limit=10))
     assert [t["id"] for t in out] == ["1", "2"]
     # second request must carry the cursor returned by page 1
@@ -91,8 +91,8 @@ def test_paged_get_retries_on_429_then_succeeds():
     bad = FakeResp(429, {})
     good = FakeResp(200, {"tweets": [{"id": "1"}], "has_next_page": False})
     with (
-        patch("tweet_scraper.time.sleep"),
-        patch("tweet_scraper.requests.get", side_effect=[bad, good]),
+        patch("tweet_scraper.api.time.sleep"),
+        patch("tweet_scraper.api.requests.get", side_effect=[bad, good]),
     ):
         out = list(paged_get("http://x", {}, "key", limit=10))
     assert [t["id"] for t in out] == ["1"]
@@ -102,8 +102,8 @@ def test_paged_get_retries_on_500_then_succeeds():
     bad = FakeResp(500, {})
     good = FakeResp(200, {"tweets": [{"id": "1"}], "has_next_page": False})
     with (
-        patch("tweet_scraper.time.sleep"),
-        patch("tweet_scraper.requests.get", side_effect=[bad, good]),
+        patch("tweet_scraper.api.time.sleep"),
+        patch("tweet_scraper.api.requests.get", side_effect=[bad, good]),
     ):
         out = list(paged_get("http://x", {}, "key", limit=10))
     assert [t["id"] for t in out] == ["1"]
@@ -112,16 +112,16 @@ def test_paged_get_retries_on_500_then_succeeds():
 def test_paged_get_retries_on_connection_error():
     good = FakeResp(200, {"tweets": [{"id": "1"}], "has_next_page": False})
     with (
-        patch("tweet_scraper.time.sleep"),
-        patch("tweet_scraper.requests.get", side_effect=[requests.ConnectionError(), good]),
+        patch("tweet_scraper.api.time.sleep"),
+        patch("tweet_scraper.api.requests.get", side_effect=[requests.ConnectionError(), good]),
     ):
         out = list(paged_get("http://x", {}, "key", limit=10))
     assert [t["id"] for t in out] == ["1"]
 
 
 def test_paged_get_stops_when_no_tweets():
     resp = FakeResp(200, {"tweets": [], "has_next_page": True, "next_cursor": "c1"})
-    with patch("tweet_scraper.requests.get", return_value=resp):
+    with patch("tweet_scraper.api.requests.get", return_value=resp):
         out = list(paged_get("http://x", {}, "key", limit=10))
     assert out == []
 

scraper/tests/test_tweet_scraper.py

@@ -66,7 +66,7 @@ def test_tweet_replies_calls_paged_get_with_correct_args():
         "author": {"userName": "replier", "name": "Replier"},
         "text": "great tweet",
     }
-    with patch("tweet_scraper.paged_get", return_value=iter([mock_reply])) as mock_pg:
+    with patch("tweet_scraper.api.paged_get", return_value=iter([mock_reply])) as mock_pg:
         results = list(tweet_replies("123", "test_key", 20))
     mock_pg.assert_called_once_with(
         TWEET_REPLIES_URL,
@@ -79,7 +79,7 @@ def test_tweet_replies_calls_paged_get_with_correct_args():
 
 
 def test_tweet_replies_returns_empty_when_no_replies():
-    with patch("tweet_scraper.paged_get", return_value=iter([])):
+    with patch("tweet_scraper.api.paged_get", return_value=iter([])):
         results = list(tweet_replies("123", "test_key", 20))
     assert results == []
 
@@ -150,7 +150,7 @@ def test_stream_replies_writes_replies_for_tweet():
         "author": {"userName": "replier", "name": "Replier"},
         "text": "reply text",
     }
-    with patch("tweet_scraper.tweet_replies", return_value=iter([mock_reply])):
+    with patch("tweet_scraper.api.tweet_replies", return_value=iter([mock_reply])):
         written = _stream_replies(writer, seen, "100", "api_key", 100)
     assert written == 1
     assert writer.rows[0]["id"] == "999"
@@ -165,7 +165,7 @@ def test_stream_replies_deduplicates_already_seen_replies():
         "author": {"userName": "r", "name": "R"},
         "text": "dupe",
     }
-    with patch("tweet_scraper.tweet_replies", return_value=iter([mock_reply])):
+    with patch("tweet_scraper.api.tweet_replies", return_value=iter([mock_reply])):
         written = _stream_replies(writer, seen, "100", "api_key", 100)
     assert written == 0
     assert writer.rows == []