Validate scraper config (warn on unknown keys, reject bad types)#3
Merged
Conversation
load_config previously did {**DEFAULTS, **cfg} with no checks, so a typo like
'searchTerm' was silently ignored and a wrong type (e.g. maxItems: "100") only
blew up deep inside the run.
Add validate_config():
- unknown keys WARN (the schema mirrors Apify Tweet Scraper V2, a superset of
what this script implements, so a real Apify vars.json must still load)
- known keys with the wrong type are a hard error (sys.exit) with the key,
expected type, and what was received
- a non-object top-level config is rejected up front
Adds tests/test_config.py (9 tests). Full validation loop green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds config validation to the scraper so typos and wrong types are caught up front instead of silently ignored or crashing mid-run.
Why
load_configdid{**DEFAULTS, **cfg}with no checks. A typo likesearchTerm(missings) was silently dropped, and a wrong type such as"maxItems": "100"only failed deep inside the run after the config looked fine.What changed
validate_config():vars.jsonmust still load unchanged. Warning surfaces likely typos without breaking compatibility.sys.exit) reporting the key, expected type, and what was received._CONFIG_TYPESmaps each known key to its accepted type(s) (type(None)marks nullable fields).Behavior (smoke-tested)
Testing
Full validation loop green from
scraper/(ruff,ruff format --check,mypy,pytest). Addstests/test_config.py(9 tests); fully offline.