Mark duplicates in `dataset_frame.csv` generated by `blech_data_sentry.py`

[abuzarmahmood]

Use the basename of the data_dir to detect duplicates

[abuzarmahmood]

The issue involves marking duplicates in the dataset_frame.csv file generated by the blech_data_sentry.py script. The duplicates should be detected using the basename of the data_dir.

Suggestions and Changes Made:

1. File to Modify:

   • File: /home/exouser/Desktop/blech_github_bot/src/repos/katzlabbrandeis/blech_data_transfer/src/blech_data_sentry.py
   • Description: This script scans the file system for datasets and checks if they have accompanying metadata. It generates a CSV file (dataset_frame.csv) for tracking purposes. The function create_dataset_frame is responsible for creating the dataframe from the info file paths.

2. Plan to Address the Issue:

   • Modify the create_dataset_frame function to include logic that checks for duplicates using the basename of the data_dir and marks them accordingly in the dataframe.

3. Specific Changes:

   • Add Duplicate Detection Logic:
     • Line: After line 151 (end of create_dataset_frame function)
     • Description of Change: Add logic to detect duplicates based on the basename of the data_dir and mark them in the dataframe.
     • Code Snippet of Edits:

       def create_dataset_frame(info_file_paths, server_path):
           """Create a dataframe from the info file paths"""
           # Get directories with info.rhd files
           root_dir = [x.split(server_path)[1].split('/')[1] for x in info_file_paths] 
           data_dirs = [os.path.dirname(f) for f in info_file_paths]
       
           # Create dataframe 
           dataset_frame = pd.DataFrame(
                   dict(
                       root_dir=root_dir,
                       data_dir=[os.path.relpath(d, server_path) for d in data_dirs],
                       )
                   )
           
           # Detect duplicates based on basename of data_dir
           dataset_frame['basename'] = dataset_frame['data_dir'].apply(lambda x: os.path.basename(x))
           dataset_frame['is_duplicate'] = dataset_frame.duplicated(subset='basename', keep=False)
       
           print(f'Found {len(dataset_frame)} datasets')
           return dataset_frame, data_dirs

This change will add a new column is_duplicate to the dataframe, which will be True for rows that have duplicate basenames in the data_dir column. This will help in identifying and marking duplicates in the dataset_frame.csv.

---

This response was automatically generated by blech_bot using model gpt-4o