|
I didn't found much on this topic in Google. Out of curiosity, does it recrawl same url multiple times? |
Answered by
ato
Feb 15, 2021
Replies: 1 comment
|
With the default configuration no, at least not within a single job launch. The most common workflow is to periodically relaunch the job, perhaps using one of the deduplication mechanisms to avoid storing multiple identical records. However there is a primitive ReschedulingProcessor which in theory can be configured to revisit on a fixed delay. Some users like the UK web archive have more elaborate setups for revisiting. |
0 replies
Answer selected by
ato
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With the default configuration no, at least not within a single job launch. The most common workflow is to periodically relaunch the job, perhaps using one of the deduplication mechanisms to avoid storing multiple identical records.
However there is a primitive ReschedulingProcessor which in theory can be configured to revisit on a fixed delay. Some users like the UK web archive have more elaborate setups for revisiting.