Skip to content

Add Parallelization via MPI for Sec 33 source and GridMapper#3

Open
Linked-Liszt wants to merge 20 commits into
AdvancedPhotonSource:masterfrom
Linked-Liszt:parallelize_loader
Open

Add Parallelization via MPI for Sec 33 source and GridMapper#3
Linked-Liszt wants to merge 20 commits into
AdvancedPhotonSource:masterfrom
Linked-Liszt:parallelize_loader

Conversation

@Linked-Liszt

@Linked-Liszt Linked-Liszt commented Sep 28, 2022

Copy link
Copy Markdown
Member

Overview:
This change adds new library files, scripts, and docs to parallelize the sector 33 grid mapping workflow. In short, the data is split by scan, and loaded with each process dynamically grabbing available scans. At the end of each operation, the data is merged similar to a merge sort operation and data is either re-shared or written to disk.

Relevant scripts have been added to /Scripts and documentation has been added to the existing sphinx project.

Implementation Notes:
Since parallelization brings many sharp edges, especially when it comes to loading data, the library files have been isolated into their own files. It would be possible to combine them with the main files using conditionals if it would be cleaner.

Future Work

  • This same technique could be likely be applied to the powder-mapping workflow.
  • Gui integration might be possible through the use of subprocess but would likely require significant changes to how the runs are launched and monitored.
  • Further parallelization could be done on the sub-scan memory splits. However, this may require duplicate loading.
  • Install process could be automated through a conda package

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant