Skip to content

Bundle Installation

Jump to bottom Edit New page
Victor Harbo Johnston edited this page Jul 4, 2023 · 2 revisions

SolrWayback bundle installation guide

Through this bundle installation you will have a fully working SolrWayback installation. Here you are able to index, search and playback web pages from your WARC files. The bundle contains Solr, the warc-indexer tool and SolrWayback installed on a Tomcat webserver. Unzip the bundle and copy two property files to your home directory and explore your WARC files.

The installation consists of the following steps:

Download

Download the newest version of the bundle here.

Unzip and follow the instructions below.

Initial Setup

  • Setup: Copy the two property files: properties/solrwayback.properties and properties/solrwaybackweb.properties to your HOME folder (or the home-folder for Tomcat user).
  • Optional: For screenshot previews to work you may have to edit the file solrwayback.properties and change the value of the last two properties : chrome.command and screenshot.temp.imagedir. Chrome(Chromium) must be installed for preview of images to work.

If you encounter any errors when running a script during installation or setup, try change the permissions for the file (startup.sh etc.). On Linux and mac, this can be done with the following command: chmod +x filename.sh

Starting SolrWayback

SolrWayback requires both Solr and Tomcat to be running. These processes are started and stopped separately with the following commands:

  • Step 1: Navigate to the location of the bundle on your computer.

For Linux and Mac:

  • Step 2.1: Start tomcat with this command: apache-tomcat-8.5.60/bin/startup.sh
  • Step 2.2: Start solr with this command: solr-7.7.3/bin/solr start

For Windows:

  • Step 2.1: To start tomcat navigate to apache-tomcat-8.5.60/bin/ and type startup.bat

  • Step 2.2: To start solr navigate to solr-7.7.3/bin/ and type solr.cmd start

  • Step 3: To see that tomcat and solr is running open the following links: http://localhost:8080/solrwayback/ and http://localhost:8983/solr/#/netarchivebuilder. If these are not throwing errors the services have been started successfully.

Useful commands

These commands are all commands that are needed to start and stop Solr and Tomcat.

Tomcat Command
Start tomcat (Linux/mac) apache-tomcat-8.5.60/bin/startup.sh
Stop tomcat (Linux/mac) apache-tomcat-8.5.60/bin/shutdown.sh
Start tomcat (Windows) Navigate to apache-tomcat-8.5.60/bin/ and type startup.bat
Stop tomcat (Windows) Navigate to apache-tomcat-8.5.60/bin/ and type shutdown.bat
Check tomcat is running http://localhost:8080/solrwayback/
Solr Command
Start solr (Linux/mac) solr-7.7.3/bin/solr start
Stop solr (Linux/mac) solr-7.7.3/bin/solr stop -all
Start solr (Windows) Navigate to solr-7.7.3/bin/ and type solr.cmd start
Stop solr (Windows) Navigate to solr-7.7.3/bin/ and type solr.cmd stop -all
Check solr is running http://localhost:8983/solr/#/netarchivebuilder

INDEXING

SolrWayback uses a Solr index of WARC files to support freetext search and more complex queries.
If you do not have existing WARC files, see steps below on harvesting with wget.

Indexing can take up to 20 minutes for 1GB warc-files. After indexing, the warc-files must stay in the same folder since SolrWayback is using them during playback etc.

Having whitespace characters in WARC file names can result in pagepreviews and playback not working on some systems. There can be up to 5 minutes delay before the indexed files are visible from search.

The script warc-indexer.sh in the indexing-folder allows for multiprocessing and keeps track of already indexed files, so the collection can be extended by adding more WARCs and running the script again.

For more information about the warc-indexer see: https://github.com/ukwa/webarchive-discovery/wiki/Quick-Start

How to index on Linux and Mac:

Call indexing/warc-indexer.sh -h for usage and how to adjust the number of processes to use for indexing. We recommend moving your WARC files to the folders warcs1 or warcs2 and use the example below for your first time indexing. Here is an example that will index all WARC-files in the warcs1 folder:

THREADS=2 ./warc-indexer.sh warcs1/*

This will start indexing files from the folder warcs1 using 2 threads. Assigning a higher number of threads than CPU cores available will result in slower indexing. Each indexing job require 1GB of RAM, so this can also be a limiting factor.

To create custom collections in your index, you can populate the collection and collectionid field in Solr with custom values. This can be done with the following command during indexing:

THREADS=4 INDEXER_CUSTOM="--collection_id  collection1 --collection corona2021" ./warc-indexer.sh warcs1/*

You can then enable faceting on these fields in the property file: solrwaybackweb.properties.

How to index on Windows:

Indexing works a little different on Windows. This also works on Linux and Mac, however we recommend using the warc-indexer.sh as above.

  • Step 1: Copy ARC/WARC files into the folder: indexing/warcs1
  • Step 2: To index the files call indexing/batch_warcs1_folder.bat (batch_warcs1_folder.sh for Linux and Mac)
  • Note: There is a batch_warcs2_folder.sh similar script to show how to easily add new WARC files to the collection without indexing the old ones again.

As mentioned above there can be a 5-minute delay for files to be visible in SolrWayback after indexing has finished. To remove this wait time visit this url after index job have finished to commit them instantly: http://localhost:8983/solr/netarchivebuilder/update?commit=true. Your WARCS should now be visible in SolrWayback at: http://localhost:8080/solrwayback/

Index logging:

The indexing script keeps track of processed files by checking if a log from a previous analysis is available. The logs are stored in the status-folder (this can be changed using the STATUS_ROOT variable). To re-index a WARC file, delete the corresponding log file.

Add a custom footer
Add a custom sidebar

Clone this wiki locally