Describe the bug
After updating from 0.2.5 to latest, markdown-checker is now hitting rate limiters on major domains.
To Reproduce
Steps to reproduce the behavior:
- Run
markdown-checker URL checker on any significant number of GitHub or Wikipedia URLs.
Expected behavior
It should run successfully.
Desktop (please complete the following information):
- OS: MacOS / GitHub CI
- Browser: Waterfox (not related)
Additional context
There are two updates that may be causing issues here.
httpx uses http-2, which is faster, but it has different connection symantics, and is quite possibly triggering rate-limiters as a result.- There is a subtle difference between the way head requests are handled between
requests and httpx, with regard to allowing redirects. It appears that you are allowing redirects, but it may not be working properly.
Describe the bug
After updating from 0.2.5 to latest,
markdown-checkeris now hitting rate limiters on major domains.To Reproduce
Steps to reproduce the behavior:
markdown-checkerURL checker on any significant number of GitHub or Wikipedia URLs.Expected behavior
It should run successfully.
Desktop (please complete the following information):
Additional context
There are two updates that may be causing issues here.
httpxuseshttp-2, which is faster, but it has different connection symantics, and is quite possibly triggering rate-limiters as a result.requestsandhttpx, with regard to allowing redirects. It appears that you are allowing redirects, but it may not be working properly.