-
Notifications
You must be signed in to change notification settings - Fork 5
Setup
This guide will get you up and running with Paper Machete (PM) on a fresh Ubuntu Linux install. Nearly all Linux distros and macOS will work. We use Ubuntu as a standard to illustrate package management commands in this guide. Please adapt the commands to reflect your specific environment as needed.
GRAKN.AI is Linux and macOS only. If you would like to use PM on Windows we suggest looking into container solutions like Docker or a Linux VM.
We have deployed PM on everything from a dinky Linux VM with 4GB RAM and 2 cores, all the way up to a bare metal 2x Intel Xeon E5-2697v4 with 1TB RAM and 72 total cores. If you have the processing power, use it. Our goal is to keep PM usable for individual researchers on a modern laptop, all the way up to organizations that have large compute resources. You will earn reduced analysis and migration time with increased computing power. But that doesn't mean you can't use PM to target specific functions in a reasonable amount of time without a lot of computing power.
For the purpose of simplicity, we assume you are installing PM on Ubuntu Linux. We also assume you are installing GRAKN.AI, Binary Ninja, and Paper Machete under ~/pm/. Just keep this in mind and tweak the commands as needed to reflect your own paths.
Let's make a directory to hold everything we need for PM.
$ mkdir ~/pm
$ cd ~/pm
And grab the Java JRE and git.
$ sudo apt-get update
$ sudo apt-get install default-jre git
The current GRAKN.AI Python driver requires Python 3.6. Yeah, that's right. Not 2.7, not 3.5, Python 3.6. At the time of writing, the repos have up to Python 3.5, so we add another repo with Python 3.6 packages. Feel free to diverge from the script here and install Python from Python.org downloads.
$ sudo add-apt-repository ppa:jonathonf/python-3.6
$ sudo apt-get update
$ sudo apt-get install python3.6
Once we have Python 3.6 installed, we'll grab the GRAKN Python driver using pip. For more detials check: https://github.com/graknlabs/grakn-python
$ python3.6 -m pip install grakn
As a sanity check, make sure you can import grakn in python3.6.
$ ~/pm$ python3.6
Python 3.6.3 (default, Oct 6 2017, 08:44:35)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import grakn
>>> # look no errors! it worked!
>>> exit()
Let's grab GRAKN version 1.0.0 from their GitHub releases. For more details check: https://dev.grakn.ai/docs/get-started/setup-guide
$ cd ~/pm
$ https://github.com/graknlabs/grakn/releases/download/v1.0.0/grakn-dist-1.0.0.zip
$ unzip grakn-dist-1.0.0.zip
$ rm grakn-dist-0.16.0.zip
And now let's pull down Paper Machete from GitHub.
$ cd ~/pm
$ git clone https://github.com/cetfor/PaperMachete
PM has a config file that holds the location to your extracted GRAKN.AI dist package. At this point just use your text editor of choice and change:
[PATHS]
GRAKN=
to this:
[PATHS]
GRAKN=/home/<username>/pm/grakn-dist-1.0.0
Obviously, you'll replace <username> with your own username in this case.
You're doing great! We just need to install Binary Ninja and setup the Python bindings. When installing Binary Ninja on Linux, you simply extract the package and run the binaryninja executable. We'll assume you've extracted Binary Ninja to ~/binaryninja. So update your the following paths to reflect your binja location.
Let's add a line to the very bottom of your ~/.profile file that will add Binary Ninja's Python bindings path to PYTHONPATH automatically for you each time you login. So open ~/.profile in your text editor of choice and at the very end, add this line:
export PYTHONPATH="$PYTHONPATH:$HOME/binaryninja/python"
Sweet! Now because we don't want to logout and in again, we'll just source this file and we won't ever have to worry about this again.
source ~/.profile
Let's do another sanity check and make sure that worked.
~/pm$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import binaryninja
>>> # look no errors!
>>> exit()
If you get an error importing binaryninja double check your path in ~/.profile is correct. If not, change it and resource your profile file.
We're so close! We just need to start GRAKN and we're ready to rock!
=============================================================================
________ _____ _______ __ __ __ __ _______ ___
| __ || _ \ | _ || | / /| \ | | | _ || |
| | |__|| | | | | | | || | / / | \ | | | | | || |
| | ____ | |_| / | |_| || |/ / | \| | | |_| || |
| ||_ || _ \ | _ || _ \ | _ | | _ || |
| |__| || | \ \ | | | || | \ \ | | \ | __ | | | || |
|________||__| \__\|__| |__||__| \__\|__| \__||__||__| |__||___|
Grakn: The Database For AI
=============================================================================
Starting Storage......SUCCESS
Starting Queue.....SUCCESS
Starting Grakn......SUCCESS
$ _
Awesome, you're done! Remember, if you want to stop GRAKN use:
$ cd ~/pm/grakn-dist-1.0.0/
$ grakn server stop
=============================================================================
________ _____ _______ __ __ __ __ _______ ___
| __ || _ \ | _ || | / /| \ | | | _ || |
| | |__|| | | | | | | || | / / | \ | | | | | || |
| | ____ | |_| / | |_| || |/ / | \| | | |_| || |
| ||_ || _ \ | _ || _ \ | _ | | _ || |
| |__| || | \ \ | | | || | \ \ | | \ | __ | | | || |
|________||__| \__\|__| |__||__| \__\|__| \__||__||__| |__||___|
Grakn: The Database For AI
=============================================================================
Stopping Grakn.....................................SUCCESS
Stopping Queue....SUCCESS
Stopping Storage......SUCCESS
$ _
If you want to completely clean all data from GRAKN and restart it, do this:
$ cd ~/pm/grakn-dist-1.0.0/
$ grakn server clean
Stopping engine
Stopping cassandra
Are you sure you want to delete all stored data and logs? [y/N] y
Deleted data in /home/user/Projects/grakn/grakn-dist-0.16.0/db
Deleted logs in /home/user/Projects/grakn/grakn-dist-0.16.0/logs
Cleaning redis
OK
$ bin/grakn.sh start
To explore your data in GRAKN, visit http://127.0.0.1:4567/ in a local web browser.
To get started with Paper Machete, slap some binaries you want to analyze in ~/pm/PaperMachete/analysis/ and start the CLI:
$ cd ~/pm/PaperMachete
$ python paper_machete.py
Follow the prompts. If you want more detailed information on options in the CLI, check out the rest of this wiki!
According to the GRAKN.AI deployment documentation, the Java maximum heap size should be configured to at least 3GB if you have the RAM.
Minimum Requirements
Grakn can run on default Java settings (heap of 768MB, 1GB machine) if the graph is small enough. Recommended production settings are at least 4GB machine with 3GB heap.
Cassandra will need at least 1GB of memory by itself if run on same machine.
If you do not have at least 4GB of RAM to devote to Paper Machete, we highly recommend you do not analyze entire binaries! Instead, we suggest you analyze a few functions of interest within the target binary. Both pmanalyze.py and paper_machete.py allow you to specify a list of functions to analyze. This will make your graph much smaller and require less overhead from Grakn.
This typically becomes a problem during migration. To set Java's heap limit, modify <grakn_dist>/bin/migration.sh and change the following:
...snip...
elif [ "$1" == "json" ]
then
java -cp ${CLASSPATH} -Dgrakn.dir="${GRAKN_HOME}/bin" ai.grakn.migration.json.JsonMigrator ${1+"$@"}
...snip...
...snip...
elif [ "$1" == "json" ]
then
java -Xmx3G -cp ${CLASSPATH} -Dgrakn.dir="${GRAKN_HOME}/bin" ai.grakn.migration.json.JsonMigrator ${1+"$@"}
...snip...