Support crawling websites behind a login

[staabm]

It would be great if the initial request could be a http-post with additional fields like username/password etc. After the initial post linkinator should handle the http-session (like receive/sending a session cookie) and crawl all other links via get-method (like it does already)

That way one could check links even if behind a login

[JustinBeckwith]

Thanks for the feature request! Implementing something like this would require really changing the entire way the library works. This would mean moving from stateless HTTP requests that are driven from node, to using something like puppeteer to make the request and maintain state. I don't have immediate plans to implement something like this, but I'd be happy to talk through a PR proposal.

[leviwheatcroft]

what about exposing the request so headers et cetera could be set?

linkChecker.check(url, {
  configureRequest (request) {
    request.headers.some-header = 'some value'
  }
})

[JustinBeckwith]

That's probably something that could be pretty easily done. If we did it though, I would only want to support it via the API (not the CLI). Would that get you where you wanna go?

[leviwheatcroft]

Actually it turned out that the issue I was trying to solve was nothing to do with missing headers.

That said, I still think that exposing the request like this would make the package more versatile and I think it would benefit a lot of users.

[Veraxus]

This could be easily solved by allowing cookies to be sent with every request. Either allow a cookies.txt file (e.g. linkchecker/linkchecker supports this) or allow cookies to be set from the config file or via the CLI. Requests are still stateless - you just send the cookies with every request.

[Uzlopak]

Actually this could be solved via a dispatcher in undici.