GUIDE: Flyscrape Dedicated Server

[dynabler]

Using generic hardware, Ubuntu server and Cronicle, you can set up a Flyscrape server. The only consideration is memory, storage, and CPU.

About this guide

This guide makes assumptions to keep in mind:

• new, unused system, so a system which already is used for anything else may cause issues and requires you to adjust as you go
• for setting up one main server
• later additions of worker servers is possible, but not discussed
• memory, storage, cpu sections are only for reference if you wish to build your own system.

Memory

Each running flyscrape instance uses 80MB of memory. So, if you run 10x Flyscrape instance = 800MB memory usage. If you have 200 Flyscrape scripts, but only 5 are running at the same time, that's only 400MB of memory.

Storage

How much space you need depends on Flyscrape cache config and usage of download function.

For final JSON, each page will take 0.0017 MB of space.

For cache, each page will take up 0.2 MB of space. Example: 13000 pages * 0.2 = 2.6 GB (10x less than average of today's webpage size)

For download function, it depends on the image sizes and quality.

CPU

This part is not my forte and has to be tested over a longer period. But for now, each running Flyscrape instance uses on average 10% of a single core with rate: 60. In practice, you can run 9 flyscrape instances at the same time on a single core. An example, a dual-core processor allows for 18 instances safely, a quad-core allows for 36 instances.

If you use Flyscrape download function, you can substract 1 flyscrape instance for each non-download instance. Write to disk will be a bottleneck. An Example: You have 1 single core, 8 non-download scripts, 1 download script. You can only run 7 scripts at the same time safely.

You can most definitely run far more scripts with/without download function, at the same time, but at some point you will notice it goes much slower that anticipated. Just something to keep in mind.

(You can use Cronicle multi-server setup to expand, see below for more info on Cronicle).

Operating System

Ubuntu server LTS seems to be a solid choice, due to its Long-Term Support. You can enable Ubuntu Pro to get the latest non-free updates for proprietary drivers. Ubuntu Pro is free to up to 5 systems.

The network is managed by netplan (instead of the old ifconfig or ipconfig)

Stuff to look out for

1. Ubuntu Server installation will only make half (or 100GB) of your storage available by default. During installation, take a look at the partition suggestion the installer gives you and make sure it uses all the space.

Steps: Use an entire disk > Done > Storage Configuration > Used Devices > ubuntu-vg (new) / ubuntu-lv 100GB > Select ubuntu-lv > Edit > Size (max Nxxx) > Type in the max given > Save > Done

If you forgot the previous step, use this guide: https://packetpushers.net/blog/ubuntu-extend-your-default-lvm-space/

• note: the dash - used in the guide differs from the dash a terminal outputs, copy the one from your terminal when needed

2. Don't install OpenSSH server, OpenSSH client is installed by default

Software

The software is divided into 2 categories: required and optional.

Required software forms the basis for your Flyscrape dedicated server.

Optional are suggestions and can be installed or replaced with your preferred software.

Required: OpenSSH

OpenSSH client is installed when installing Ubuntu. How To Use SSH to Connect to a Remote Server

You can now log in remotely and continue this guide.

Required: Cronicle

Cronicle is a task scheduler and runner, with a web based front-end UI.

It handles both scheduled, repeating and on-demand jobs, with real-time stats and live log viewer.

Not recommended are the usual web server management software, like webmin and Cockpit, since they don't have a dedicated task scheduler and runner and Flyscrape server is not a web server.

Cronicle supports multi-server setup:

• 1x main server (called Main node)
• 1 or more slave servers (called Worker nodes)
• backup server (if main server fails)

The main node divides the jobs among the worker nodes based on settings you provide (random, available memory/storage on a slave etc.) In a nutshell, it's fast to set up and expandable later on.

Install Cronicle

Cronicle requires nodejs. You can install it using the guide here: https://deb.nodesource.com/

The installation instructions for Cronicle is at the end of this page: https://cronicle.net/

Cronicle also ships with an autostart script, which starts at server reboot. No need to add a daemon.

Required: Flyscrape

Flyscrape can be installed using this guide: https://flyscrape.com/docs/installation/

Optional: Rclone

Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

Optional: FileBrowser (+ edit scripts in browser)

A difficult suggestion to make since it all depends on your preferences for security and current network setup. A suggestion would be FileBrowser.net

In a nutshell, FileBrowser is a file server with a web ui. You can browse, upload, delete, add, rename files. It also allows you to edit files.

You install a single executable binary, specify settings in filebrowser.json and add it as a service, so it runs/autostarts upon reboot.

FileBrowser Installation

In order to install and get FileBrowser running you need to:

1. install filebrowser
2. initiate filebrowser for database creation (filebrowser.db)
3. make a config file (filebrowser.json)
4. autostart filebrowser (filebrowser.service)

1. Installation

Follow the steps here: https://filebrowser.org/installation

2. Database creation

Go to the folder which will hold the filebrowser.db file. Example:

$ mkdir /home/your_username/filebrowser
$ cd /home/your_name/filebrowser
$ filebrowser config init

3. FileBrowser Config

Filebrowser looks for a config file in /, $HOME and /etc/filebrowser/ and is named .filebrowser.json (note the dot). You can also specify a different location later when doing autostart (see next step).

The most basic config file filebrowser.json is as follows:

{
  "address": "machine_ip_eg_10.0.0.1",
  "port": "your_prefered_port_eg_8080",
  "root": "/path/to/share_eg/home/user_name/flyscrape/scripts",
  "database": "/path/to/database_filebrowser.db"
}

4. FileBrowser Autostart

FileBrowser does not have autostart script/daemon. You can use the following commands/codes to get it daemonized:

$ sudo nano /etc/systemd/system/filebrowser.service

[Unit]
Description=Filebrowser
After=network-online.target

[Service]
User=root
Group=root

ExecStart=/usr/local/bin/filebrowser -c /path/to/your/config/file

[Install]
WantedBy=multi-user.target

If you choose NOT to use the default locations FileBrowser searches for, specify it as shown in script above -c /path/to/your/config/file

Then do

$ sudo systemctl start filebrowser
$ sudo systemctl enable filebrowser
$ sudo systemctl restart filebrowser

Required: sudo chown -R username /path/to/directory

If everything goes well, you now have a dedicated Flyscrape server. You can continue copying your Flyscrape scripts using FileBrowser, and make Cronicle run it.

Before adding your scripts to Cronicle, test to make sure everything is working. The majority of problems in (Linux) Ubuntu are caused by file permissions. It suffices to transfer ownership from root to your_username using sudo chown -R username /path/to/flyscrape/scripts where directory is where all your Flyscrape scripts are located.

For more file permission on Linux: https://linuxconfig.org/fixing-the-permission-denied-error-on-linux

Optional: dos2unix

If you work on a Windows machine, then dos2unix removes (invisible) characters like \n \r from bash scripts. You can install it with sudo apt install dos2unix and simply run dos2unix script.js. It will overwrite the input file.

Adding Flyscrape scripts to Cronicle

Cronicle offers a variety of ways to add jobs, but the easiest method is running bash scripts. You need to use absolute paths in your scripts.

Example folder structure used with flyscrape as your main scraper

247_technology/
├─ tablets.js
│─ computers.js
│─ phones.js
│─ update.sh

update.sh looks like this:

#!/bin/bash

# Paths, parent_paths makes your scripts run anywhere its invoked
parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )
flyscrape_bin=/home/your_username/.flyscrape/flyscrape

cd "$parent_path"

# Iterate through each file in the current directory
for file in *.js
do
    # Check if the item is a file (not a directory)
    if [ -f "$file" ]; then
        $flyscrape_bin run $file	
    fi
done

In Cronicle you add a new event, choose plugin Shell Script and add the following line of code: (Cronicle --> schedule --> event --> plugin --> Shell Script)

/home/your_username/flyscrape/247_technology/update.sh

And that's about it :-) Have suggestions or improvements? Feel free to comment.

Cronicle Timeout

By default any Cronicle job will end after an hour. You can disable this for long running scraping project.

Cronicle --> schedule --> event --> Timeout --> 0 (0 to disable)

Progress Bar Flyscrape in Cronicle

If you like some sort of display of progress in Cronicle, you can add the following code to your scripts. It only counts the number of scripts in a folder and shows a percentage of how far it progressed. It doesn't count the actual Flyscrape progress.

#!/bin/bash

# Paths, parent_paths makes your scripts run anywhere its invoked
parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )
flyscrape_bin=/home/your_username/.flyscrape/flyscrape

cd "$parent_path"

# Count the total number of files in the current directory
TOTAL_FILES=$(find . -maxdepth 1 -type f -name "*.js" | wc -l)

# Initialize a counter for completed files
COMPLETED_FILES=0

# Iterate through each file in the current directory
for file in *.js
do
    # Check if the item is a file (not a directory)
    if [ -f "$file" ]; then
        $flyscrape_bin run $file

        # Increment the completed files counter
        ((COMPLETED_FILES++))
        
        # Calculate the percentage of completion
        PERCENTAGE=$((COMPLETED_FILES * 100 / TOTAL_FILES))
        
        # Echo the progress
        echo "$PERCENTAGE%"
    fi
done