Skip to content

derekburgess/jaws

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

374 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JAWS

hehe

2026

JAWS is now an MCP server first. The Semantic Kernel jaws-agent and the older smol.py experiment have been removed — instead of bundling agents, JAWS exposes its pipeline as MCP tools (jaws-mcp) that any MCP client (e.g. Claude Code) can drive. Deploy to a Raspberry Pi at the edge of a network (passive LAN taps work great) and run the MCP server from it.

Reworked the analysis model: traffic is now aggregated into one endpoint profile per IP address (labeled with its organization), describing inbound/outbound bytes, packets, peers, ports, protocols, and inter-packet timing. Anomaly detection clusters those endpoints, blending the embedding with standardized behavioral features so volume/fan-out outliers (e.g. unusual outbound traffic) actually surface — and every endpoint now gets an interpretable anomaly score (aggregate robust-z over the raw behavioral features) with per-feature reasons in real units, so a flag explains itself (high outbound bytes vs. suspiciously regular beacon-like timing). A dedicated host-outbound view ranks the capture host's own upload destinations from raw packets, answering the exfiltration/beaconing question directly, and jaws-finder --ablate quantifies how much the text embedding actually contributes versus the numeric features (text-only vs. numeric-only vs. blended, with silhouette and Jaccard agreement). On the MCP side, inspect_endpoint drills from any flagged IP back to its peers and raw packets. Local embeddings moved to sentence-transformers with a small model registry (config.PACKET_MODELS), so swapping/adding models is a one-line change.

2025

Refactored the experience to be more agentic — CLI commands plus experimental agents that collected data, used tools, and produced analysis. smol.py used smolagents to orchestrate a manager/analyst hand-off; jaws-agent used Microsoft Semantic Kernel with a Gradio command center. Both have since been removed in favor of the MCP server (see 2026).

Context

JAWS is a Python based shell pipeline for analyzing the shape and activity of networks for the purpose of identifying outliers. It also works as a Graph RAG, utilizing OpenAI or local Transformers. It gathers and stores packets/osint in a graph database (Neo4j). It provides a set of commands to transform and process packets into plots and reports using: PCA, DBSCAN, OpenAI, Jina Code embeddings, etc. It is intended to run locally using "open" models, but is set to run using OpenAI by default for demos and ease of use.

Prerequisites and initial setup

This part of the guide is assuming a clean install and mainly exists as a guide for myself when setting up new systems.

CUDA Support

If you plan on running local models against a NVIDIA GPU, you will need the CUDA Toolkit installed. You can configure an installer or guide here -- On Ubuntu, if you installed the additional drivers for NVIDIA, you can run:

apt install nvidia-cuda-toolkit

Wireshark

JAWS uses pyshark which requires termshark, which can be installed with Wireshark. Termshark is an optional installation and bundled with the executables for Windows and Mac. On Ubuntu you can install both using:

apt install wireshark and apt install termshark

Wireshark will ask you about adding non-root users to the Wireshark group. It is recommended that you say Yes. If you said No, then you can run:

dpkg-reconfigure wireshark-common

In addition, the installation and adding non-root users is suppose to add your user to the Wireshark group and set permissions, but I have found that it doesn't always do this. You may need to run:

adduser $USER wireshark and chmod +x /usr/bin/dumpcap

Neo4j DBMS

JAWS also uses Neo4j as the graph database. You can run the provided Neo4j docker container (See below), or install and run the Neo4j DBMS/Desktop app on Windows/Mac/Linux.

On Ubuntu, you can follow these instructions for installing the package:

https://neo4j.com/docs/operations-manual/current/installation/linux/debian/#debian-installation

To use the desktop application on Linux, you will need to set its permissions:

chmod +x neo4j...

Ubuntu may complain about lack of FUSE.

Ubuntu may also complain about lack of sandbox... So far I have only found running the app image with --no-sandbox appended.

Additionally, the problem appears to stem from changes made to Ubuntu 24.04, this command, recommended in the Docker documentation, which appears to disable "AppArmor", also resolves the issue:

sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

Conda/Anaconda/Miniconda

I tend to use Anaconda and prefer their installation script over the guide...

If you used the installation script, add Conda to bash: nano ~/.bashrc and append export PATH=~/anaconda3/bin:$PATH to the end of the file, replacing /anaconda3/bin with your actual installation path.

Some other useful(and basic) Conda commands:

conda create --name env_name python=version

conda activate env_name and conda deactivate

conda env config vars list and conda env config vars set ENV_VAR=value

Finally, if you want to use the navigator GUI, it can be installed using:

conda install anaconda-navigator and anaconda-navigator

Docker

As mentioned above, JAWS optionally uses Docker. Again, easy enough to figure out.

Some useful (and basic) Docker commands:

docker ps -a and docker volume ls

docker inspect and docker stop

docker rm and docker volume rm

docker system prune -a --volumes

Set Neo4j Environment Variables

NEO4J_URI (bolt://localhost:7687)

NEO4J_USERNAME (neo4j)

NEO4J_PASSWORD (you set)

Set Environment Variables for Additional Services

To run jaws-ipinfo, you will need to sign up for a free account with ipinfo, and create an env variable for:

IPINFO_API_KEY

jaws-compute uses OpenAI (text-embedding-3-large) by default. This requires that you have an OpenAI account (not free) and create an env variable for:

OPENAI_API_KEY

Note: the MCP server (jaws-mcp) defaults to transformers instead, since it typically runs on a GPU host — the CLI default stays openai for easy demos.

Optional: Since OpenAI is not free, by passing --api transformers, or jaws-utils --model jina-code, jaws can download and run on device models from Hugging Face (see config.PACKET_MODELS; the default is jinaai/jina-embeddings-v2-base-code). The bundled models are public and download without any API key — you only need an env variable if you add a gated model to the registry:

HUGGINGFACE_API_KEY

The command jaws-finder displays several plots using Matplot, but also saves those plots to a directory/endpoint of your choice, using:

JAWS_FINDER_ENDPOINT

Install the JAWS Python Package

From the /jaws root directory, install JAWS (dependencies are pulled in automatically from requirements.txt):

pip install .

If you are using the Neo4j dbms and GUI, that is it, you can skip the Docker steps and run jaws-guide for the rest of the instructions and command overview.

Neo4j Docker Container

This Docker container operates as a local/headless Neo4j database. You can run all commands against it by default and easily connect to and view the graph using the Neo4j GUI.

From the /jaws/harbor directory run:

docker build -t jaws-neodbms --build-arg NEO4J_USERNAME --build-arg NEO4J_PASSWORD --build-arg DEFAULT_DATABASE=captures .

Then run:

docker run --name captures -p 7474:7474 -p 7687:7687 --detach jaws-neodbms

If you plan to run the Hugging Face models on your local machine that is it, you can skip the next step and run jaws-guide for the rest of the instructions and command overview.

JAWS Compute Docker Container

This Docker container operates as a full instance of JAWS. However, the intended purpose is for providing a deployable container for compute resources.

From the /jaws/ocean directory run:

docker build -t jaws-image --build-arg NEO4J_URI --build-arg NEO4J_USERNAME --build-arg NEO4J_PASSWORD --build-arg IPINFO_API_KEY --build-arg OPENAI_API_KEY --build-arg HUGGINGFACE_API_KEY .

docker run --gpus 1 --network host --name jaws-container --detach jaws-image

To pull the Hugging Face models, run jaws-utils with the model argument.

docker exec -it jaws-container jaws-utils --model jina-code

To use the container run:

docker exec -it jaws-container jaws-compute --api "transformers"

Usage

jaws-guide

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

1 watching

Forks

Contributors