Currently, URLs with a fragment (e.g. https://site.com/#main) are filtered out (see function url_filter_fragment in filters.py).
However, in SPAs where the hash fragment is used for client-side routing, ignoring the hash means that CWAC misses major portions of the website's views or "pages".
Example of the Issue
On the following site, the content changes entirely based on the hash, but the crawler currently skips these links:
Base URL: https://aria.stats.govt.nz/aria/
Target Resource: https://aria.stats.govt.nz/aria/#ClassificationView:uri=http://stats.govt.nz/cms/ClassificationVersion/CARS5587
Preferred behavior
CWAC should be able to distinguish between:
-
Actual fragment identifiers: Same-page links to id or name values should be ignored.
-
Hash-based routing: Links that load new content (e.g., #StandardView:uri=hvybhrsoA0CNa1Dq) should be followed.
Currently, URLs with a fragment (e.g. https://site.com/#main) are filtered out (see function
url_filter_fragmentin filters.py).However, in SPAs where the hash fragment is used for client-side routing, ignoring the hash means that CWAC misses major portions of the website's views or "pages".
Example of the Issue
On the following site, the content changes entirely based on the hash, but the crawler currently skips these links:
Base URL: https://aria.stats.govt.nz/aria/
Target Resource: https://aria.stats.govt.nz/aria/#ClassificationView:uri=http://stats.govt.nz/cms/ClassificationVersion/CARS5587
Preferred behavior
CWAC should be able to distinguish between:
Actual fragment identifiers: Same-page links to
idornamevalues should be ignored.Hash-based routing: Links that load new content (e.g., #StandardView:uri=hvybhrsoA0CNa1Dq) should be followed.