Skip to content

Validate scraper config (warn on unknown keys, reject bad types)#3

Merged
cybermejo merged 1 commit into
mainfrom
worktree-scraper-config-validation
Jun 17, 2026
Merged

Validate scraper config (warn on unknown keys, reject bad types)#3
cybermejo merged 1 commit into
mainfrom
worktree-scraper-config-validation

Conversation

@cybermejo

Copy link
Copy Markdown
Owner

Summary

Adds config validation to the scraper so typos and wrong types are caught up front instead of silently ignored or crashing mid-run.

Why

load_config did {**DEFAULTS, **cfg} with no checks. A typo like searchTerm (missing s) was silently dropped, and a wrong type such as "maxItems": "100" only failed deep inside the run after the config looked fine.

What changed

  • New validate_config():
    • Unknown keys → warn (not error). The config schema mirrors the Apify Tweet Scraper V2 input, which is a superset of what this script implements, so a real Apify vars.json must still load unchanged. Warning surfaces likely typos without breaking compatibility.
    • Known keys with the wrong type → hard error (sys.exit) reporting the key, expected type, and what was received.
    • A non-object top-level config is rejected up front.
  • _CONFIG_TYPES maps each known key to its accepted type(s) (type(None) marks nullable fields).

Behavior (smoke-tested)

example config            -> runs clean
unknown keys (Apify cfg)  -> "[warn] unknown config key 'addUserInfo' — ignored." then runs
maxItems: "lots"          -> "Config error: 'maxItems' must be int or float, got str." (exit 1)

Testing

Full validation loop green from scraper/ (ruff, ruff format --check, mypy, pytest). Adds tests/test_config.py (9 tests); fully offline.

Independent of #1 — touches load_config, a different region of tweet_scraper.py than the streaming change.

load_config previously did {**DEFAULTS, **cfg} with no checks, so a typo like
'searchTerm' was silently ignored and a wrong type (e.g. maxItems: "100") only
blew up deep inside the run.

Add validate_config():
- unknown keys WARN (the schema mirrors Apify Tweet Scraper V2, a superset of
  what this script implements, so a real Apify vars.json must still load)
- known keys with the wrong type are a hard error (sys.exit) with the key,
  expected type, and what was received
- a non-object top-level config is rejected up front

Adds tests/test_config.py (9 tests). Full validation loop green.
@cybermejo cybermejo self-assigned this Jun 17, 2026
@cybermejo cybermejo added the enhancement New feature or request label Jun 17, 2026
@cybermejo
cybermejo merged commit 1e6f162 into main Jun 17, 2026
4 checks passed
@cybermejo
cybermejo deleted the worktree-scraper-config-validation branch June 17, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant