This project focuses on intermediate-level data visualization and preprocessing using Python. It demonstrates how to read multiple Excel datasets, extract useful information, preprocess the data, save reusable datasets, and create clean visualizations for analysis.
This repository is a continuation of my Low Level Data Visualization project, where I covered the fundamentals of data visualization. Here, I build upon those concepts by introducing data preprocessing, combining datasets, exporting processed files, and creating publication-ready visualizations. In the future, I will also add a High Level Data Visualization project to complete the learning journey.
The workflow begins by importing the required libraries, including Pandas for data manipulation, Matplotlib for visualization, Pathlib for file management, and a custom plotting utility for consistent figure styling.
Two Excel datasets (fsi-2020.xlsx and fsi-2021.xlsx) are loaded from the raw data directory. Instead of using the complete datasets, only the required columns are selected to reduce unnecessary information and make the data easier to work with.
For the 2020 dataset, the script extracts the Change from Previous Year column, while for the 2021 dataset, it extracts the X1: External Intervention column. These columns are merged with the selected base data and then renamed to maintain a consistent structure across both datasets.
The processed datasets are saved as Pickle files, making them faster to reload during future analysis without repeatedly processing the original Excel files.
Next, both processed datasets are loaded back into memory and combined into a single DataFrame. This combined dataset is then visualized using a customized line chart that highlights the Improvement values.
The visualization includes customized axis labels, grid lines, legends, figure size adjustments, and predefined y-axis limits for better readability. Finally, the generated figure is automatically exported with sequential filenames, making it easy to manage multiple plots without overwriting previous results.
- Reads multiple Excel datasets using Pandas
- Extracts only the required columns from each dataset
- Merges additional analytical columns into the selected data
- Standardizes column names across different datasets
- Saves processed datasets as Pickle files
- Reloads processed data for efficient analysis
- Combines multiple processed datasets into a single DataFrame
- Creates customized line charts using Matplotlib
- Automatically exports figures with sequential filenames
- Organized and reusable workflow for intermediate-level data visualization
⬇️
⬇️
⬇️
⬇️
⬇️
⬇️
⬇️
⬇️
⬇️
After successful execution, the script generates:
- processed_data_2020.pkl – Processed dataset for 2020
- processed_data_2021.pkl – Processed dataset for 2021
- Automatically exported figures (
figure_1.png,figure_2.png, ...) - A customized line chart showing Improvement values
- Reusable processed datasets for further analysis and visualization
This project represents the Intermediate Level stage of my Data Visualization learning series. It extends the concepts introduced in my Low Level Data Visualization project by adding data preprocessing, dataset transformation, reusable storage using Pickle, automated figure exporting, and cleaner visualization techniques.
The next step in this series will be High Level Data Visualization, where I will explore advanced visualization techniques, interactive dashboards, statistical graphics, and more sophisticated analytical workflows.