add sitemap crawling to help maximise URLs found

[a11ya11y]

this is currently happening with 2 sites: Tax Policy Home and Tax Technical - Inland Revenue NZ

They each have 1000s of pages, but Tax Policy Home is only getting 236 pages scanned, and Tax Technical - Inland Revenue NZ even fewer. I get the same results on a 2 separate installs of CWAC on different infrastructure.

Running LibreCrawl over Tax Policy and it got a 200 response from 1000 URLs, so the site can be crawled.

Here's the output from a 300-page scan of these 2 sites, where taxpolicy.ird.govt.nz only had 236 pages scanned, and taxtechnical.ird.govt.nz had 109:

2026-07-22_11-19-35_taxpolicy-taxtecnical.log
audit_log.csv
chromedriver.log
config.json
pages_scanned.csv
progress.csv

[a11ya11y]

After some more testing, LibreCrawl enables sitemap crawling by default. Turning that off, it finds only 244 URLs for taxpolicy.ird.govt.nz. That's consistent with CWAC's crawl numbers.

@G-Rath Here's a copilot-generated proof of concept with sitemap crawling added to crawler.py. It appears to work: https://github.com/GOVTNZ/cwac/tree/fix/sitemap-crawling

[G-Rath]

@a11ya11y ok cool that is what I suspect was happening - I think it could make sense to have CWAC interact with sitemaps, but probably something we need to do a bit of discover for to ensure it doesn't surprise users.

I'll make sure it's included in our internal work