When I first examined and downloaded the files of dataset from the SciSciNet in HuggingFace, I transferred them from .parquet to .csv to help me beter convert them into .json file.

Then I used Python scripts to convert .csv files into .json that can be display in the website. Only the fields needed for visualization, such as paper IDs, publication years, author IDs are extracted. I also removed duplicated records, filtered papers by year, and excluded extremely large papers or author groups including almost every reputational universities in the world that are irrevalent, more importantly, would make the network too dense and crash.

Since there is a maximum size for file uploaded into Github also those files are too large to load all at once in the browser, I divided each large json file into several small files. I organized the data by year and by visualization purpose. For example, I tried to create a smaller overview file such as citation_network_top500.json for the initial visualization, and then designed the website to load additional data only when the user clicks a “Load More Data” button. This helped reduce the initial loading time and made the website more responsive.

For the visualization part, due to the limited time period, I only implement the requiement from specification to maintain the basic function without extra graphic interation. The citation network represents papers as nodes and citation relationships as edges, while the author collaboration network represents authors as nodes and co-authorship relationships as edges. I also added interactive functions such as draggable nodes, hover tooltips, zooming, filtering, and detail-on-demand so users can explore the data more actively instead of only viewing a static graph.

One major challenge I encountered was scalability. When the network included more than around 2,000 nodes, the framerate dropped significantly, making reader harder to maintain feeling comfortable. So I decided to turn visulization of networks which contain more than 2000 nodes into static pictures after rendering for the first few seconds. Also I applied a "load more data" button in each network section to prevent sudden stuck and stopped.

However, despite many attempts i have tried, I couldn't find a practical way to implement clear visualization like edge bundling technique to display the node clusters more distinguishable. When the filter was applied to more than 2,000 nodes, the nodes and edges still appeared crowded together. The website uses progressive data loading. The first screen provides a high-level overview of the citation network or author collaboration network. When users interact with the visualization, such as clicking a node, selecting a year, or zooming into a cluster, the frontend can request more related data from a backend so the network expands dynamically based on user interest instead of overwhelming the browser at the start.