SurtPrefixedDecideRule doesnt seem to work as expected

[naveen17797]

Heritrix version : 3.4.0-20210803

I have added my full configuration file here

https://gist.github.com/naveen17797/b119d62ef3c0a20e656cae5ea56d5821

 <bean id="acceptSurts" class="org.archive.modules.deciderules.surt.SurtPrefixedDecideRule">
  <property name="decision" value="ACCEPT"/>
  <!-- crawl only from seed domains -->
  <property name="seedsAsSurtPrefixes" value="true" />
  <!-- <property name="alsoCheckVia" value="false" /> -->
  <!-- <property name="surtsSourceFile" value="" /> -->
  <!-- <property name="surtsDumpFile" value="${launchId}/surts.dump" /> -->
  <!-- <property name="surtsSource">
        <bean class="org.archive.spring.ConfigString">
         <property name="value">
          <value>
           # example.com
           # http://www.example.edu/path1/
           # +http://(org,example,
          </value>
         </property> 
        </bean>
       </property> -->
 </bean>

For example i have added reddit.com to the seeds list, but i find urls from other domains are present on the crawl log, but since seedsAsSurtPrefixes is set to true the expected behaviour would be to filter the urls only from the seed domain ( if i am not mistaken )

[ato]

The decide rules work such that the last rule that matches wins. The ACCEPT TransclusionDecideRule will override the acceptSurts rule in order to fetch embedded resources such as images, stylesheets, javascript etc that are needed to render the page. To disable this behaviour and follow acceptSurts strictly even for embedded content then delete or comment out the TransclusionDecideRule.

    <!-- ...but ACCEPT those more than a configured link-hop-count from start... -->
    <bean class="org.archive.modules.deciderules.TransclusionDecideRule">
     <!-- <property name="maxTransHops" value="2" /> -->
     <!-- <property name="maxSpeculativeHops" value="1" /> -->
    </bean>

[ldko]

More information about how crawl scope works and the individual rules is at: https://heritrix.readthedocs.io/en/stable/configuring-jobs.html#crawl-scope

[naveen17797]

Thanks @ato , this fixed the issue, i was under the impression that RejectDecideRule rejects all the discovered urls, then all the rules below decides whether the url can be accepted or not.