Data Science Projects

Introduction

The main goal is to apply Data Science Methodology and mimic the business scenario I may face. You can observe stages of Data Science Methodology shown in the below image.

To achieve my main goal, I will act in dual roles: the Business Stakeholder who is expecting a successful solution to their business need, and the Data Scientist who is capable of delivering the solution by successfully passing each stage mentioned in the Data Science Methodology.

Below, you can find projects where I am mentioning the problem statement given by stakeholders, and the proposed solution is presented in the associated notebook for the project.

Projects

Computer Vision with Tiny_VIRAT

Goals

• To gain hands on experience and knowledge in the computer vision field.
• To understand data preparation and EDA for video data.

Problem Statement

We are a Surveillance and CCTV company. We have a standard CCTV system, and we want to upgrade it into smart CCTVs. We want a solution that enables our cameras to detect human patterns, count objects, provide analysis within a 24-hour cooldown, and correctly detect the proper alert action for the system.

Proposed Solution

Supply Chains Domain Knowledge

Goals

• To gain knowledge by completing Leverage Data Science for a More Agile Supply Chain Specialization.

• To apply Data Science, AI, and Automation ideas by implementing hands on project that mimic a real case business scenario in the level of supply chains management.

Version Control

It is very important to track dependencies from the very beginning of the project. Here I am using poetry to manage my dependencies.

Following are the steps I have committed on my local machine before I start coding.

• Ensure poetry is installed. You can check that in the VS Code Terminal after specifying the directory by typing poetry --version

• Type poetry init to create the pyproject.toml file. This step is done once. Follow the guide generated by the command.

• To activate the virtual environment, type poetry shell. The first time, it will create a .venv folder in the directory, and since we are committing our local code to GitHub, it is recommended to add the .venv folder to .gitignore. The next step fulfills that, and we need to do it once.

  • echo ".venv/" >> .gitignore — this command will create .gitignore if not created earlier during the first time we initiated the repository in GitHub. It will add the .venv folder to .gitignore.

• Now, we are ready to add needed packages using poetry. Use poetry add package

• It is recommended that whenever we add a new package, we track and share the requirements.txt file. Use the below command always whenever you add a new package and before committing:

  • poetry export -f requirements.txt --output requirements.txt --without-hashes Automation Tip

• The following command will make the kernel detects out .venv file python -m ipykernel install --user --name=poetry-env --display-name "Python (poetry)"

Automation

I was thinking if I can create files with ps1 extension, (Since I'm using Windows Power Shell), to automate my git commands. Below is the step by step:

• Manually create a new file in the directory ensure using meaning name with ps1 extension.
• Add the wanted command inside the file
• Save the file, if you want to run it use: .\meaning-name.ps1
• In case of having a security error use the below command, which you may need this step only once:
  • Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Example: I want to automate the process of tracking requirements.txt file as follows:

• I have created manually a new file named update-reqs.ps1 in Computer Vision with CCTV directory.
• I have added the below script:

# This script updates requirements.txt from Poetry, commits the changes to Git, and pushes to the main branch.
poetry export -f requirements.txt --output requirements.txt --without-hashes
git add requirements.txt
git commit -m "Update requirements.txt"
git push origin main
Write-Host "requirements.txt updated and committed to Git!"

• Finally run .\update-reqs.ps1 on your terminal.

List of Power Shell files used for Automation

• update-reqs.ps1: Used to update, commit, and push requirements.txt file with updated packages. You run it anytime needed when you add new packages using poetry.

• setup-venv-kernel.ps1: Used to create a kernel for the virtual environment created by poetry. Use it once in order to start coding with your notebook. Refresh VS Code, then navigate to select kernel where VS Code automatically will recommend you the kernel.

• create_hf_dataset.ps1: It is associated with create_hf_dataset_script.py. Used to create a new Hugging Face dataset repository, where I have designed it to have the following directories: raw, train, test, valid, references.

• upload_mp4_to_raw.ps1: It is associated with upload_mp4_to_raw_script.py. Used to upload mp4 files that are stored in a nested structure, where a recursive for loop is being used to extract those mp4 files. The process will push extracted mp4 files from my local to HF repository.