Graph-Based Document Neighbourhood Visualization

This repository is based on the gar_vis which is initially built by Martijn Smits during his master's thesis at TU Delft.

Note: This project was developed using Cursor assisted by the Sonnet 4.5 model.

A Flask-based web application for visualizing and analyzing document neighbourhoods in information retrieval systems. This tool provides an interactive interface to explore corpus graphs, relevance patterns, and neighbourhood structures.

Features

• 🔍 Interactive Heatmap Visualization - Zoom, pan, and hover to explore document neighbourhoods
• 📊 Comprehensive Statistics - View recall metrics and relevance distribution
• 🎨 Color-Coded Relevance - Instantly identify relevant, new, and duplicate documents
• ⚡ Fast Navigation - Browse through multiple queries with keyboard shortcuts
• 📱 Remote Access - Access via SSH port forwarding from any device
• 💾 HDF5 Support - Efficiently handle large-scale corpus graphs

Document Types

The visualization uses a sophisticated labeling scheme to identify different types of relevant documents:

• Not Relevant (white) - Document is not relevant to the query
• Relevant Document (lightgray) - Relevant document present in the original ranking (column 0)
• Duplicate Relevant Document (lightcoral) - Relevant document NOT in original ranking, appearing multiple times in neighbours
• New Relevant Document (lime) ⭐ - Relevant document NOT in original ranking, first occurrence in neighbourhood

Installation

1. Clone the Repository

git clone https://github.com/Mandeep-Rathee/graph-visualization
cd graph-visualization

2. Create Virtual Environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install Dependencies

pip install -r requirements.txt

Usage

Create h5 file

python create_h5_file.py

Local Usage

Run the Flask app with your HDF5 neighbourhood file:

python app.py --h5_file /path/to/your/neighbourhoods.h5 --port 5000

Then open your browser to:

http://localhost:5000

Remote Server Usage

If running on a remote server, use SSH port forwarding:

Step 1: Start the server on the remote machine

python app.py --h5_file /path/to/your/neighbourhoods.h5 --port 5000

Step 2: On your local machine, create an SSH tunnel

ssh -L 5000:localhost:5000 username@remote-server.com

Step 3: Open in your browser

http://localhost:5000

Command-Line Arguments

python app.py [OPTIONS]

Option	Default	Description
--h5_file	./neighbourhoods/biology.h5	Path to HDF5 file containing neighbourhood data
--min_rel	1	Minimum relevance label to consider a document relevant
--port	5000	Port to run the Flask server on
--debug	False	Run in debug mode with auto-reload

Example Commands

# Basic usage
python app.py --h5_file ./data/msmarco.h5

# Custom port
python app.py --h5_file ./data/msmarco.h5 --port 8080

# Debug mode with higher relevance threshold
python app.py --h5_file ./data/msmarco.h5 --min_rel 2 --debug

HDF5 File Format

The application expects an HDF5 file with the following structure:

file.h5
├── dataset/
│   ├── topics_qid       # Query IDs (bytes/string array)
│   ├── topics_query     # Query text (bytes/string array)
│   ├── qrels_qid        # Qrels query IDs (bytes/string array)
│   ├── qrels_docno      # Qrels document IDs (bytes/string array)
│   └── qrels_label      # Relevance labels (integer array)
└── neighbourhoods/
    ├── {qid_1}          # 2D array of document IDs (rows × columns)
    ├── {qid_2}          # Each row represents a document
    └── ...              # Each column represents its neighbours

Creating HDF5 Files

If you're using PyTerrier Adaptive, you can create compatible HDF5 files using the gar_vis.py script from the parent project:

from pyterrier_adaptive import CorpusGraph
import pyterrier as pt
import h5py

# Load your corpus graph and dataset
corpus_graph = CorpusGraph.from_hf("macavaney/msmarco-passage.corpusgraph.bm25.128")
dataset = pt.get_dataset("irds:msmarco-passage/trec-dl-2019/judged")

# Create neighbourhood visualization file
# See gar_vis.py for full implementation

Statistics Explained

Recall Metrics

• Original Recall: Percentage of relevant documents found in the original ranking (column 0)
• Neighbour Recall: Percentage of relevant documents found in the neighbour columns (columns 1+)
• Total Recall: Percentage of relevant documents found across original + neighbours

Relevance Distribution

The visualization shows how relevant documents are distributed across:

• The original ranking (column 0)
• Immediate neighbours (columns 1-5)
• Extended neighbours (columns 6+)

This helps identify:

• Whether the original retrieval missed relevant documents
• Whether graph traversal successfully discovers new relevant documents
• Patterns in where relevant documents cluster

Architecture

Backend (Flask)

• app.py - Main Flask application
• REST API endpoints for queries and neighbourhood data
• Efficient HDF5 data loading with caching

Frontend (HTML/JavaScript)

• templates/viewer.html - Single-page application
• Plotly.js for interactive heatmap visualization
• Vanilla JavaScript for responsive UI

Data Processing

• Manhattan distance-based labeling for "first occurrence" detection
• Efficient numpy operations for large-scale matrices
• UTF-8 decoding for proper string handling

Troubleshooting

Port Already in Use

# Find the process using the port
lsof -i :5000
# Kill it
kill -9 <PID>

HDF5 File Not Found

Ensure the path to your HDF5 file is correct:

python app.py --h5_file $(pwd)/neighbourhoods/your_file.h5

Empty Visualization

• Check that your HDF5 file has data in the neighbourhoods/ group
• Verify that qrels exist for the query you're viewing
• Try adjusting --min_rel threshold

UTF-8 Encoding Issues

If you see garbled text, ensure your HDF5 file uses UTF-8 encoding for string data.

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

Development Setup

# Install development dependencies
pip install -r requirements.txt

# Run in debug mode
python app.py --debug