Company: Codtech IT Solutions Private Limited
Name: Sharma Abhishek
Intern ID: CTIS9913
Domain: Data Science
Duration: 6 Weeks
ETL Pipeline using Python, Pandas, and Scikit-learn
This project focuses on building an ETL (Extract, Transform, Load) pipeline using Python. ETL is an important process in data preprocessing and machine learning because it helps convert raw data into a clean and usable format. The main objective of this task was to understand how data is collected, cleaned, transformed, and prepared for further analysis or machine learning applications.
The project was developed using Python libraries such as Pandas, NumPy, Scikit-learn, and Joblib. A customer dataset in CSV format was used as the input file. The complete workflow was automated using a Python script to make preprocessing efficient and reusable.
The project was completed in multiple stages following the ETL workflow.
The first step involved loading the raw customer dataset from a CSV file using Pandas. This stage helped in understanding the structure of the dataset and identifying different data types and missing values.
The transformation stage included preprocessing operations such as handling missing values, encoding categorical variables, and scaling numerical features. Scikit-learn preprocessing tools and pipeline concepts were used to automate these steps.
A preprocessing pipeline was created so that the same transformation process could be applied consistently without repeating code.
After preprocessing, the dataset was divided into training and testing sets. This step is important because it allows machine learning models to be trained and evaluated properly.
The cleaned and transformed datasets were exported into separate CSV files. Metadata and preprocessing objects were also saved using Joblib for future reuse.
- Python
- Pandas
- NumPy
- Scikit-learn
- Joblib
Task1/
│
├── etl_pipeline.py
├── requirements.txt
├── README.md
├── .gitignore
├── data/
│ └── raw/
│ └── customer_data.csv
└── outputs/
The pipeline generates the following files:
- cleaned_data.csv
- X_train_transformed.csv
- X_test_transformed.csv
- y_train.csv
- y_test.csv
- metadata.json
- preprocessor.joblib
Through this project, I gained practical knowledge of ETL workflows, data preprocessing, and automation using Scikit-learn pipelines. I learned how to handle missing values, transform datasets, and organize preprocessing steps efficiently. This task also improved my understanding of reusable machine learning workflows and real-world data preparation techniques.