feat: add normalized findings, jsonl/stdout output and workflow docs - #7
feat: add normalized findings, jsonl/stdout output and workflow docs#7theoffsecgirl wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR evolves webxray into a workflow-friendly producer by emitting normalized findings in JSON/JSONL, supporting stdout-safe piping, and documenting pipeline integration.
Changes:
- Add a normalized findings schema plus JSON/JSONL serialization and
--stdoutoutput mode. - Route logs/progress output to
stderrto keepstdoutclean for piping/ingestion. - Update READMEs with workflow/pipeline usage and new output parameters.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
webxray.py |
Adds finding normalization + JSON/JSONL output helpers, introduces --format/--stdout, and moves banner/logs/tqdm output to stderr. |
README.md |
Reworks documentation to highlight pipeline workflow and new output modes. |
README.es.md |
Updates Spanish documentation with normalized output examples and workflow integration guidance. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| """ | ||
|
|
||
| __version__ = "1.1.0" | ||
| __version__ = "1.2.0" |
There was a problem hiding this comment.
__version__ is bumped to 1.2.0 here, but pyproject.toml still declares project.version = "1.1.0". This will make the installed package version disagree with the CLI/banner version. Consider updating the packaging version (or deriving __version__ from package metadata) to keep them in sync.
| # Show version | ||
| webxray --version | ||
| ``` | ||
| Normalized findings ready for pipeline ingestion. |
There was a problem hiding this comment.
Since the PR’s key behavior change is “logs/progress to stderr so stdout can be piped safely”, the English README’s Output section should mention that logs/progress go to stderr and only normalized findings are emitted on stdout when using --stdout (or --json-output -). Without this, users may think the tool is noisy on stdout.
| Normalized findings ready for pipeline ingestion. | |
| When using `--stdout` or `--json-output -`, logs and progress messages are sent to `stderr`, and only normalized findings are emitted on `stdout`. | |
| This keeps `stdout` safe for piping into other tools. |
| elif finding_type == "waf_xss": | ||
| normalized["vector"] = "xss" | ||
| normalized["evidence"] = ["waf-oriented payload reflected in response"] | ||
| normalized["tags"] = ["waf-bypass", "get-param"] | ||
| normalized["raw"]["waf"] = finding.get("waf") |
There was a problem hiding this comment.
normalize_finding stores the original finding dict under raw and then mutates it (normalized["raw"]["waf"] = ...). Even though current WAF findings already include waf, keeping raw as the same object is a surprising side effect and can lead to accidental mutations later. Consider storing a shallow/deep copy in raw and removing the redundant assignment.
| def serialize_findings(findings: List[dict], fmt: str) -> str: | ||
| if fmt == "jsonl": | ||
| return "\n".join(json.dumps(f, ensure_ascii=False) for f in findings) | ||
| return json.dumps(findings, indent=2, ensure_ascii=False) |
There was a problem hiding this comment.
For JSONL output, it’s usually safer to end the payload with a trailing newline so concatenation and line-oriented tools behave consistently (and so the last record is newline-terminated when writing to a file). Consider appending "\n" when fmt == "jsonl" (including the empty-findings case).
| normalized_findings = [normalize_finding(f) for f in all_findings] | ||
|
|
||
| if args.json_output or args.stdout: |
There was a problem hiding this comment.
normalized_findings is computed unconditionally, even when neither --json-output nor --stdout is requested. This adds unnecessary work (and retains extra references) on large scans. Consider computing/normalizing only inside the if args.json_output or args.stdout: block.
| normalized_findings = [normalize_finding(f) for f in all_findings] | |
| if args.json_output or args.stdout: | |
| if args.json_output or args.stdout: | |
| normalized_findings = [normalize_finding(f) for f in all_findings] |
| ```text | ||
| [*] Starting scan → https://example.com | ||
| [*] Crawling depth: 2 | Timeout: 10s | ||
|
|
||
| [+] URLs found: 47 | ||
| [+] Forms found: 8 | ||
|
|
||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| XSS | ||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| [!] Reflected XSS → https://example.com/search?q=<script>alert(1)</script> | ||
| Parameter: q | Type: GET | ||
|
|
||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| SQLi | ||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| [!] Possible SQLi → https://example.com/product?id=1' | ||
| Parameter: id | Error: MySQL syntax | ||
|
|
||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| Headers | ||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| [!] Missing: Content-Security-Policy | ||
| [!] Missing: X-Frame-Options | ||
| [+] Present: Strict-Transport-Security | ||
| [+] Present: X-Content-Type-Options | ||
|
|
||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| WAF | ||
| ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ||
| [*] WAF detected: Cloudflare | ||
| [*] Applying bypass payloads... | ||
| [!] Possible bypass → https://example.com/search?q=<img src=x onerror=alert(1)> | ||
|
|
||
| [+] Results saved → results.json | ||
| [*] Scan completed in 12.4s | ||
| --format json|jsonl | ||
| --stdout | ||
| --json-output | ||
| ``` |
There was a problem hiding this comment.
The README "Params" snippet only lists the new output flags, but omits the required -u/--url and other commonly used options (-d, -t, --no-*, --waf-xss). This makes it hard to run the tool from the README alone; consider expanding this list or pointing readers to webxray --help.
| ### Guardar a fichero e ingerir en `bb-copilot` | ||
|
|
||
| ```bash | ||
| webxray -u https://target.com -d 2 --format jsonl --json-output out.jsonl | ||
| bbcopilot ingest webxray out.jsonl |
There was a problem hiding this comment.
README.es.md refers to ingesting into bb-copilot, but the commands use bbcopilot (no hyphen). Consider making the tool name consistent to avoid confusion when users copy/paste.
This PR upgrades webxray from a standalone scanner into a workflow-friendly producer.
Changes
--format json|jsonl--stdoutmode--json-outputWhy
This makes webxray compatible with
bbcopilot ingestand enables a real bug bounty pipeline:webxray -u https://target.com --format jsonl --stdout > out.jsonl bbcopilot ingest webxray out.jsonl bbcopilot correlate bbcopilot auto-triage bbcopilot exploit-planNotes
Findings are signals/candidates, not confirmed vulnerabilities.