Excluding URIs from crawl by regex doesn't work

[LeoI07]

(Apparently) due to this phenomenon, the ExtractorJS module interprets what appear to be references to object properties in the JavaScript code (of the form <object name>.<property name>) of a site I'm trying to crawl as relative URLs, which results in an obnoxious amount of unwanted URLs being crawled. Fortunately, in this case, the vast majority of these share only a handful of object names, so I tried using both the forceStrictIfUrlMatchingRegexList and rejectRelativeMatchingRegexList properties of the ConfigurableExtractorJS module and the MatchesListRegexDecideRule bean to exclude them from the crawl, using regexes prefixed with ^/, / and neither for the former, but the URLs I was trying to exclude continued to show up in the crawl logs. Any fixes?

[ato]

MatchesListRegexDecideRule matches against the full absolute URL, so may not be useful for your situation.

I've never used it but looking at the code rejectRelativeMatchingRegexList looks like it should work but note it's a regex match not a regex search. So if you want to reject anything starting with / I think you'll want to use /.* not ^/.

The script console may be useful for checking things:

// Script Engine: Groovy
htmlOut.println(appCtx.getBean("extractorJs").isOnRejectList("/foo"));

[LeoI07]

The script console may be useful for checking things:

// Script Engine: Groovy
htmlOut.println(appCtx.getBean("extractorJs").isOnRejectList("/foo"));

Clicking on "Scripting Console" in the menu gives me a 404 error. I also get the same error if I attempt to execute a script via the REST API.