How to get the HTML content from CostAssignmentPolicy instance

[cgr71ii]

Hi!

I've implemented a CostAssignmentPolicy instance and I'd like to use CLD2 in order to get the language of the document which has been already downloaded. I've checked out the CrawlURI API, and the closest methods related to my goal are getContent*, but I wonder if there is something similar to getContent() itself which, if I'm not wrong, it should return the HTML of the already downloaded document.

Thank you!

[cgr71ii]

@ato Any suggestion? Thank you!

[ato]

Getting the html content from a CostAssignmentPolicy seems not possible in the current architecture as the policy is invoked by FrontierPreparer before the html is fetched from the server. Perhaps after fetching updating the cost from a custom/subclassed html extractor by calling CrawlURI.setHolderCost() would work.

[cgr71ii]

Thank you for the reply! I think I explained myself wrong. I meant the HTML from the document which discovered the current URI which is under evaluation of CostAssignmentPolicy. In CostAssignmentPolicy there is available an instance of CrawlURI, which has the methods getUURI(), which returns the URI of the document which hasn't already been downloaded and the cost is being assigned, and getVia(), which returns the URI of the document which discovered the present URI whose cost is being calculated. I meant this last one, the URI from getVia() which has already been downloaded since it's been parsed and the URI from getUURI() has been discovered. Is there any way to obtain the HTML easily from this "via" URI? If not, I'll give it a try to the method you proposed.

[ato]

Oh! I see. In that case, I think this should work:

try (InputStream stream = curi.getFullVia().getRecorder().getContentReplayInputStream()) {
    // do something with stream
}

[ato]

These examples might also be relevant:

https://github.com/internetarchive/heritrix3/blob/master/engine/src/test/java/org/archive/crawler/selftest/KeyWordProcessor.java
https://github.com/internetarchive/heritrix3/blob/master/engine/src/test/java/org/archive/crawler/selftest/KeyWordUriPrecedencePolicy.java

[cgr71ii]

It worked, thank you!!! A last minor detail: is there a way to be sure that the "via" URI is HTML? I'd like to only apply CLD2 to those and not other files like "dns:".