Refactor collector into modular package; add source health, validation, exporter, tests, and CI - #13
Refactor collector into modular package; add source health, validation, exporter, tests, and CI#13MahanKenway wants to merge 2 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fa9ae01a30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not payload: | ||
| return None | ||
| data = json.loads(payload) | ||
| host = data.get("add") |
There was a problem hiding this comment.
When a fetched VMess line decodes to valid JSON that is not an object, for example vmess://W10= ([]), data.get raises AttributeError, which is not caught by parse. Since ConfigCollector.collect() parses every untrusted source line before testing, one malformed VMess entry aborts the whole update instead of being skipped; check that data is a dict or catch this case before reading fields.
Useful? React with 👍 / 👎.
| return None | ||
| data = decoded | ||
| data = data.split("?", 1)[0] | ||
| match = re.search(r"@([^:/]+):(\d+)$", data) |
There was a problem hiding this comment.
Shadowsocks/SIP002 plugin URLs commonly include the optional slash before the query, e.g. ss://cipher:password@example.com:443/?plugin=...; after the query is stripped this leaves ...:443/, so the end-anchored regex does not match and the config is silently dropped before reachability testing. This removes otherwise valid SS plugin configs from both ss.txt and mix.txt; allow the optional trailing slash/path when extracting host and port.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c29841f39
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| host, port, type=socket.SOCK_STREAM | ||
| ) | ||
| ] | ||
| except socket.gaierror: |
There was a problem hiding this comment.
When a feed includes a syntactically parseable config with an invalid IDNA hostname, such as a single 300-character label, socket.getaddrinfo raises UnicodeEncodeError rather than socket.gaierror. Since ConfigCollector.collect() consumes executor.map() directly, that worker exception aborts the whole scheduled update instead of skipping the bad config; catch UnicodeError here or validate hostname length before resolving.
Useful? React with 👍 / 👎.
1 participant
Motivation
scripts/collector.pyinto a maintainable package to improve testability and safety.Description
src/package with modules:core(collector, parser, tester, exporter, notifier, deduplicate, source_health, static_validator),models,config,utils, andmainto drive the pipeline.scripts/collector.pyrunner to callsrc.main, and addconfig/sources.yaml,src/config/settings.py, and a.env.examplefor runtime configuration.schemas/config.schema.json), and prioritized source quarantine/health management.ABUSE.md,SECURITY.md,CONTRIBUTING.md,docs/*), a projectLICENSE,.gitignore, test configuration (pytest.ini), and a GitHub Actions workflow that runspython -m src.main, commits generatedconfigs/, and uploads logs on failure.Testing
tests/covering parser, deduplication, tester, source health, and static validation and ranpytestto execute the suite.pytest(13 tests) and all tests completed successfully.Codex Task