Specifying custom cookies?

[818S]

How do I specify custom cookie(s) in the configuration (bean) for a job?

[818S]

I still haven't found a way to declare them in the XML but managed to add it to the running crawl. I had to figure it out myself since the code in the documentation is outdated and doesn't work but the API docs helped. Here it is for anyone who may also be looking for it:

// Groovy
cookies = appCtx.getBean("cookieStore");

// Create a new Cookie with its Name and Value
epochSeconds = Long.parseLong("2094586238"); // Expiration in 2036
expirationDate = (epochSeconds >= 0 ? new Date(epochSeconds * 1000) : null);
cookie = new org.apache.http.impl.cookie.BasicClientCookie("COOKIE_NAME", "COOKIE_VALUE");
cookie.setDomain("COOKIE_DOMAIN");
cookie.setExpiryDate(expirationDate);
cookie.setSecure(true);
cookie.setPath("/");

rawOut.println(cookie);
cookies.addCookie(cookie);

// Print all cookies
cookies = appCtx.getBean("cookieStore").getCookies().toArray();
cookies.each{ rawOut.println("${it}\n") }

[ato]

Thanks for sharing. I've added your updated version to the wiki page.

As for the original question, I don't see an obvious way to add cookies inline in the XML but it looks like you might be able to load them from a separate cookies.txt file using something like this (modifying the existing cookieStore bean in the default config):

 <bean id="cookieStore" class="org.archive.modules.fetcher.BdbCookieStore">
   <property name="cookiesLoadFile">
      <bean class="org.archive.spring.ConfigFile">
        <property name="path" value="cookies.txt" />
      </bean>
   </property>
 </bean>

The format of cookies.txt is described in the AbstractCookieStore.readCookies() javadoc:

The input is text in the Netscape's 'cookies.txt' file format. Example entry of cookies.txt file:

www.archive.org FALSE / FALSE 1311699995 details-visit texts-cralond

Each line has 7 tab-separated fields:

DOMAIN: The domain that created and have access to the cookie value.
FLAG: A TRUE or FALSE value indicating if hosts within the given domain can access the cookie value.
PATH: The path within the domain that the cookie value is valid for.
SECURE: A TRUE or FALSE value indicating if to use a secure connection to access the cookie value.
EXPIRATION: The expiration time of the cookie value, or -1 for no expiration
NAME: The name of the cookie value
VALUE: The cookie value

[TheTechRobo]

@ato I get "Bean name 'cookieStore' is already used in this element" when trying that.

[TheTechRobo]

Nevermind - I didn't notice that there was a cookieStore bean stored previously. My bad.