LargeGraphViz

LargeGraphViz is a Python package and CLI toolset for processing, embedding, analyzing, and visualizing large-scale complex web networks. The web networks are created based on the Open Web Index (OWI) dataset.

graph.mp4

Features

• Preprocessing of Parquet datasets to create graphs.
• Node embeddings using deep learning and network-based methods (TENE, DeepWalk, ASNE).
• Community detection (e.g., Louvain) and tracking communities over time.
• Topic modeling from node metadata.
• Export to JSON for WebGL-based visualizations.

Installation

pip install git+https://github.com/It4innovations/LargeGraphViz.git

Requires Python >= 3.10.

CLI Usage

After installing, you can use the provided commands:

owi-topic input_dir result_dir
owi-parquet-to-graph input_dir result_dir --min-node-degree 1
owi-community nodes.parquet edges.parquet --with-force-layout
owi-embed nodes.parquet edges.parquet output_file_path.parquet --embed-method tene --dim 3
owi-save-graph nodes.parquet edges.parquet output_graph.json

or use the owi-main command to run the entire pipeline:

owi-main input_dir result_dir --min-node-degree 1 --embed-method tene --dim 3 --with-force-layout

You can also run the commands with --help to see all available options.

Acknowledgement

This work has received funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101070014 (OpenWebSearch.EU, https://doi.org/10.3030/101070014