Install and Configure ML

Build the Spark application

Build spot-ml Spark application.

[soluser@edge-server]$ cd spot-ml
[soluser@edge-server]$ sbt assembly

Copy the entire spot-ml folder to the primary ML node, the node that will launch Spark application. For simplicity, this is often the lowest-numbered node in this role. Log in to the primary ML node.

[soluser@edge-server]$ scp –r spot-ml node-04:/home/solution_user/.
[soluser@edge-server]$ ssh node-04
[soluser@node-04]$cd /home/solution_user/spot-ml

Edit configuration variables in /etc/spot.conf

If it hasn't been done, copy /etc/spot.conf to ML node.

Once /etc/spot.conf is in ML node, make sure the following configurations are set:

 SPK_EXEC - Number of Spark executors
 SPK_EXEC_MEM - Total memory per executor
 SPK_DRIVER_MEM - Total driver memory
 SPK_DRIVER_MAX_RESULTS - Total memory for driver max results
 SPK_EXEC_CORES - Number of cores per executor
 SPK_DRIVER_MEM_OVERHEAD - Driver memory overhead
 SPAK_EXEC_MEM_OVERHEAD - Executor memory overhead
 TOL - Results threshold

More details about how to set up Spark properties con be found here.