Functionality to run autosort remotely to generate dataset quality metrics

[abuzarmahmood]

Perhaps on jetstream when a new dataset is uploaded
https://docs.fabfile.org/en/stable/getting-started.html#run-commands-via-connections-and-run

[abuzarmahmood]

Can copy back all data that is not .dat, .npy, or .h5, and MOST .png files (perhaps psths and units plots can be kept)
All this is about < 100M but would be very useful information when assessing a dataset.

[abuzarmahmood]

The issue involves implementing functionality to run autosort remotely to generate dataset quality metrics, potentially using Jetstream when a new dataset is uploaded. The suggestion is to use Fabric for remote execution and file transfer via SSH. Additionally, the user wants to exclude certain file types (.dat, .npy, .h5, and most .png files) from being copied back, which are about < 100M in size, to assess dataset quality.

Suggested Changes:

1. Integrate Fabric for Remote Execution

   • File: /src/blech_data_transfer.py
   • Description: Add Fabric to the imports and create a function to execute autosort remotely.
   • Code snippet:

     from fabric import Connection
     
     def run_autosort_remotely(server_address, dataset_path):
         conn = Connection(host=server_address)
         result = conn.run(f'autosort {dataset_path}', hide=True)
         print(result.stdout)

2. Modify Data Transfer to Exclude Specific File Types

   • File: /src/blech_data_transfer.py
   • Line: 246-247
   • Description: Update the file list to exclude .dat, .npy, .h5, and most .png files.
   • Code snippet:

     file_list = [f for f in file_list if os.path.isfile(f) and not f.endswith(('.dat', '.npy', '.h5', '.png'))]

3. Ensure Dataset Quality Metrics are Generated

   • File: /src/dataset_handler.py
   • Description: Add functionality to generate and store dataset quality metrics after autosort is run.
   • Code snippet:

     def generate_quality_metrics(dataset_path):
         # Logic to generate and store quality metrics
         pass

4. Update Tests for New Functionality

   • File: /tests/test_blech_data_transfer.py
   • Description: Add tests to ensure that the remote execution and file exclusion logic works as expected.
   • Code snippet:

     def test_run_autosort_remotely():
         # Mock connection and test remote execution
         pass
     
     def test_file_exclusion():
         # Test that excluded file types are not transferred
         pass

These changes will integrate remote execution capabilities using Fabric, modify the data transfer process to exclude specific file types, and ensure that dataset quality metrics are generated and tested.

---

This response was automatically generated by blech_bot using model gpt-4o

[abuzarmahmood]

Look at https://github.com/katzlabbrandeis/blech_clust/blob/master/blech_autosort.sh to use for running autosort on remote machine

[abuzarmahmood]

It seems that I am unable to access the blech_autosort.sh script directly from the repository. However, I can still suggest improvements based on the feedback provided. Here's an updated response that incorporates the suggestion to use the blech_autosort.sh script for running autosort on a remote machine:

Suggested Changes:

1. Integrate Fabric for Remote Execution Using blech_autosort.sh

   • File: /src/blech_data_transfer.py
   • Description: Add Fabric to the imports and create a function to execute blech_autosort.sh remotely. This script should be used to handle the autosort process on the remote machine.
   • Code snippet:

     from fabric import Connection
     
     def run_autosort_remotely(server_address, dataset_path):
         conn = Connection(host=server_address)
         result = conn.run(f'bash blech_autosort.sh {dataset_path}', hide=True)
         print(result.stdout)

2. Modify Data Transfer to Exclude Specific File Types

   • File: /src/blech_data_transfer.py
   • Line: 246-247
   • Description: Update the file list to exclude .dat, .npy, .h5, and most .png files.
   • Code snippet:

     file_list = [f for f in file_list if os.path.isfile(f) and not f.endswith(('.dat', '.npy', '.h5', '.png'))]

3. Ensure Dataset Quality Metrics are Generated

   • File: /src/dataset_handler.py
   • Description: Add functionality to generate and store dataset quality metrics after autosort is run.
   • Code snippet:

     def generate_quality_metrics(dataset_path):
         # Logic to generate and store quality metrics
         pass

4. Update Tests for New Functionality

   • File: /tests/test_blech_data_transfer.py
   • Description: Add tests to ensure that the remote execution and file exclusion logic works as expected.
   • Code snippet:

     def test_run_autosort_remotely():
         # Mock connection and test remote execution
         pass
     
     def test_file_exclusion():
         # Test that excluded file types are not transferred
         pass

These changes will integrate remote execution capabilities using Fabric and the blech_autosort.sh script, modify the data transfer process to exclude specific file types, and ensure that dataset quality metrics are generated and tested.

---

This response was automatically generated by blech_bot using model gpt-4o