This is the open-source implementation for CompressGraph:
CompressGraph: Efficient Parallel Graph Analytics with Rule-Based Compression”, Zheng Chen, Feng Zhang, Jiawei Guan, Jidong Zhai, Xipeng Shen, Huanchen Zhang, Wentong Shu, Xiaoyong Du. SIGMOD/PODS '23: Proceedings of the 2023 International Conference on Management of Data. https://dl.acm.org/doi/10.1145/3588684
git clone https://github.com/ZhengChenCS/CompressGraph.git --recursive
cd CompressGraph
mkdir -p build
cd build
cmake .. -DLIGRA=ON -DGUNROCK=ON
make -jIf you just want to run application on CPU, you can set -DGUNROCK=OFF.
The initial input graph format should be in the adjacency graph format. For example, the SNAP format(edgelist) and the adjacency graph format for a sample graph are shown below.
SNAP format:
src dst
0 1
0 2
2 0
2 1
Adjacency Graph format:
AdjacencyGraph
3 <The number of vertices>
4 <The number of edges>
0 <o0>
2 <o1>
2 <o2>
1 <e0>
2 <e1>
0 <e2>
1 <e3>
The Compression Mudule accepts binary CSR(Compressed Sparse Row) graph data as input, which contains a vlist and a elist array.
We provide two programs for converting graph files from edgelist and adjacency graph format to CSR format.
User can invoke them as follows:
- Edgelist to CSR
edgelist2csr < <edgelist.txt>- Adjacency graph to CSR
adj2csr < <adjgraph.txt>The two programs will generate two output files in CSR format: csr_vlist.bin and csr_elist.bin in the current directory.
The dataset folder provides an example.
To compress a input graph, run:
compress <csr_vlist.bin> <csr_elist.bin>To filter the rule by the threshold(16 by default), run:
filter <csr_vlist.bin> <csr_elist.bin> <info.bin> 16The filter program will filter out rules that meet (freq - 1) * (len - 1) - 1 <= threshold;, where freq is the frequency of rules, len is the length of rules.
We also provide a filtering program filter_decmp that can filter out rules that do not meet the frequency and length requirements separately.
filter_decomp <csr_vlist.bin> <csr_elist.bin> <info.bin> <freq_threshold> <len_threshold>The filter_decomp will filter out rules that meet freq < freq_threshold || len < len_threshold.
We have implemented the CompressGraph analytic engine based on Ligra on CPU and Gunrock on GPU.
Before running the graph analytic program, we need to convert the CSR format to the format required by Ligra.
Additionaly, we generate some auxiliary structures, including order.bin and degree.bin(used in pagerank and hits).
$convert2ligra $csr_vlist $csr_elist > $output
$save_degree $csr_vlist
$gene_rule_order $csr_vlist $csr_elist $infoWe provide a script data_prepare.sh and data_prepare_gpu.sh to execute the data prepareation process in script directory.
Users can execute graph applications using the following approach:
./bfs_cpu -r 1 $file
./cc_cpu $file
./sssp_cpu -r 1 $file
./pagerank_cpu -maxiters 10 -i $info -d $degree -o $order $file
./topo_cpu -i $info $file
./hits_cpu -maxiters 10 -i $info -o $order $file./bfs_gpu $file 0
./cc_gpu $file
./sssp_gpu $file $info 0
./pagerank_gpu $file
./hits_gpu $file
./topo_gpu $file $infoWe provide scripts in script/cpu and script/gpu directory to execute these programs.
cd script
bash data_prepare.sh
cd cpu
bash bfs.sh
bash sssp.sh
bash cc.sh
bash pagerank.sh
bash topo.sh
bash hits.sh
cd gpu
bash bfs.sh
bash sssp.sh
bash cc.sh
bash pagerank.sh
bash topo.sh
bash hits.shIf you use our code, please cite our paper:
@article{chen2023compressgraph,
title={CompressGraph: Efficient Parallel Graph Analytics with Rule-Based Compression},
author={Chen, Zheng and Zhang, Feng and Guan, JiaWei and Zhai, Jidong and Shen, Xipeng and Zhang, Huanchen and Shu, Wentong and Du, Xiaoyong},
journal={Proceedings of the ACM on Management of Data},
volume={1},
number={1},
pages={1--31},
year={2023},
publisher={ACM New York, NY, USA}
}
This section details additional features and analyses added to the original CompressGraph framework. All scripts mentioned are located in the script directory or its subdirectories.
- Goal: Implements a parallel triangle counting algorithm that operates directly on the CompressGraph representation using the Ligra framework.
- Compilation: The
triangle_cpuexecutable is built automatically when compiling with-DLIGRA=ON. Ensure you have applied the<cstdint>header fix to thedeps/ligracode. - How to Run:
cd script/cpu bash triangle.sh - Output: Prints the total triangle count and the execution time to the console.
- Goal: Analyzes the trade-off between the compression rule filter threshold, the resulting graph size (compression ratio), and the performance of BFS.
- How to Run:
- Ensure the baseline
.binfiles exist indataset/cnr-2000/compress/by runningbash script/data_prepare.shonce. - Run the analysis script:
cd script ./analyze_threshold.sh
- Ensure the baseline
- Output: Prints a CSV table to the console showing
threshold,filtered_size_bytes,size_ratio, andbfs_time_seconds. This data can be used to plot the trade-off curve.
- Goal: Evaluates the effectiveness of CompressGraph's rule-based compression and analytics performance on graph structures different from the default web graph. Tests on a collaboration network (
ca-GrQc) and a road network (roadNet-CA). - Data Preparation:
- Download the datasets (run from the
datasetdirectory):# Collaboration Network wget [https://snap.stanford.edu/data/ca-GrQc.txt.gz](https://snap.stanford.edu/data/ca-GrQc.txt.gz) gunzip ca-GrQc.txt.gz grep -v "^#" ca-GrQc.txt > ca-GrQc.edgelist # Road Network wget [https://snap.stanford.edu/data/roadNet-CA.txt.gz](https://snap.stanford.edu/data/roadNet-CA.txt.gz) gunzip roadNet-CA.txt.gz grep -v "^#" roadNet-CA.txt > roadNet-CA.edgelist
- Download the datasets (run from the
- How to Run:
Use the
run_full_pipeline.shscript, providing the base name of the graph edgelist file in thedatasetdirectory. Run from thescriptdirectory:cd script ./run_full_pipeline.sh ca-GrQc ./run_full_pipeline.sh roadNet-CA - Output: Creates a results directory for each graph (e.g.,
ca-GrQc_results,roadNet-CA_results) containing intermediate files, compression stats (compress_stats.txt), and prints analytics output to the console. Compare the compression ratios and run times against thecnr-2000graph. Note: PageRank may fail on these graphs.