From b6c6b3c027fcb4b39631e07364735c8dec19dd32 Mon Sep 17 00:00:00 2001 From: Ariel Rokem Date: Tue, 22 Aug 2023 15:10:49 -0700 Subject: [PATCH 1/2] Adds a notebook with a rather complete baby_afq tutorial. --- .pre-commit-config.yaml | 9 + examples/tutorial_examples/baby_afq.ipynb | 343 ++++++++++++++++++++++ 2 files changed, 352 insertions(+) create mode 100644 .pre-commit-config.yaml create mode 100644 examples/tutorial_examples/baby_afq.ipynb diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 000000000..1abfefe00 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,9 @@ +repos: + - repo: local + hooks: + - id: jupyter-nb-clear-output + name: jupyter-nb-clear-output + files: \.ipynb$ + stages: [commit] + language: system + entry: jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace \ No newline at end of file diff --git a/examples/tutorial_examples/baby_afq.ipynb b/examples/tutorial_examples/baby_afq.ipynb new file mode 100644 index 000000000..426b77c07 --- /dev/null +++ b/examples/tutorial_examples/baby_afq.ipynb @@ -0,0 +1,343 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "id": "7dd23004", + "metadata": {}, + "outputs": [], + "source": [ + "import warnings\n", + "warnings.filterwarnings(\"ignore\")" + ] + }, + { + "cell_type": "markdown", + "id": "4a1973ed", + "metadata": {}, + "source": [ + "# pyAFQ: Automated Fiber Quantification ... in Python" + ] + }, + { + "cell_type": "markdown", + "id": "44483d42", + "metadata": {}, + "source": [ + "## This notebook is available at: https://github.com/36000/babyAFQ_example\n" + ] + }, + { + "cell_type": "markdown", + "id": "2631a596", + "metadata": {}, + "source": [ + "## Section 1: Four benefits of using pyAFQ compared to matlab AFQ:" + ] + }, + { + "cell_type": "markdown", + "id": "88b0b126", + "metadata": {}, + "source": [ + "### 1. Free and open source\n", + " 1. No need for a MATLAB license" + ] + }, + { + "cell_type": "markdown", + "id": "3a345b2a", + "metadata": {}, + "source": [ + "### 2. Easier to run on cloud services\n", + " 1. For example, Amazon Web Services (AWS) or Microsoft Azure\n", + " 2. pyAFQ's only dependencies are other python packages which are installed automatically using \"Package Installer for Python (pip)\"\n", + "![](install_diagram.png)" + ] + }, + { + "cell_type": "markdown", + "id": "6cbd99d3", + "metadata": {}, + "source": [ + "### 3. Easier to run on BIDS datasets\n", + "1. The Brain Imaging Data Structure (BIDS) is a way of organizing neuroimaging and behavioral data\n", + "2. BIDS is a community effort\n", + "3. There are many [benefits to BIDS](https://bids.neuroimaging.io/benefits.html):\n", + " 1. It will be easy for another researcher to work on your data\n", + " 1. For example, by using BIDS you will save time trying to understand and reuse data acquired by a graduate student or postdoc that has already left the lab.\n", + " \n", + " 2. There is a growing number of data analysis software that understand BIDS data, like pyAFQ\n", + " \n", + " 3. Databases such as OpenNeuro.org, LORIS, COINS, XNAT, SciTran, and others will accept and export datasets organized according to BIDS.\n", + " 1. If you ever plan to share your data publicly (some journals require this) you can speed up the curation process by using BIDS.\n", + " \n", + " 4. There are validation tools (also available online) that can check your dataset integrity and let you easily spot missing values.\n", + " \n", + "![BIDS diagram](bids_diagram.png)\n", + "\n", + "Image source: http://bids.neuroimaging.io/" + ] + }, + { + "cell_type": "markdown", + "id": "f8cdb35c", + "metadata": {}, + "source": [ + "### 4. Easier to customize methods and integrate with other pipelines\n", + " 1. pyAFQ provides tools to import tractography or mori group files from the original Matlab based AFQ to pyAFQ\n", + " 2. pyAFQ can use tractography from other pipelines like MRtrix3 or Qsiprep which implement other algorithms\n", + " like SIFT and ACT\n", + "\n", + "#### For babyAFQ, Use MRtrix3 for the tractography: https://www.mrtrix.org/" + ] + }, + { + "cell_type": "markdown", + "id": "5f834933", + "metadata": {}, + "source": [ + "## Section 2: Baby pyAFQ code examples" + ] + }, + { + "cell_type": "markdown", + "id": "96c2ed52", + "metadata": {}, + "source": [ + "### Step 1: Organizing the data\n", + "Put your dataset in BIDS. Here is an example dataset with one subject:\n", + "![Example BIDS](bids_example.png)" + ] + }, + { + "cell_type": "markdown", + "id": "a7204feb", + "metadata": {}, + "source": [ + "## This data is available at: https://tinyurl.com/35tptjz9" + ] + }, + { + "cell_type": "markdown", + "id": "5cb26a48", + "metadata": {}, + "source": [ + "### Step 2: Initialize a GroupAFQ object\n", + "Creates a GroupAFQ object, that encapsulates tractometry. This object can be used to manage the entire AFQ pipeline, including:\n", + "\n", + " Tractography\n", + "\n", + " Registration\n", + "\n", + " Segmentation / Bundle Recognition\n", + "\n", + " Cleaning\n", + "\n", + " Profiling\n", + "\n", + " Visualization" + ] + }, + { + "cell_type": "markdown", + "id": "cbe77e4a", + "metadata": {}, + "source": [ + "In this case, MRtrix has already done the tractography (visualized using [MI-Brain](https://github.com/imeka/mi-brain)): ![mrtrix_track](3d_screenshot.png)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "efe6998b", + "metadata": {}, + "outputs": [], + "source": [ + "from AFQ.api.group import GroupAFQ\n", + "import AFQ.api.bundle_dict as abd\n", + "import AFQ.data.fetch as afd\n", + "from AFQ.definitions.image import LabelledImageFile\n", + "import plotly\n", + "\n", + "myafq = GroupAFQ(\n", + " # ==== BIDS parameters ====\n", + " bids_path=\"example_bids_subject\",\n", + " preproc_pipeline=\"vistasoft\",\n", + " # ===== Registration parameters ====\n", + " reg_template_spec=afd.read_pediatric_templates()[\"UNCNeo-withCerebellum-for-babyAFQ\"],\n", + " reg_subject_spec=\"b0\",\n", + " # ==== Bundle parameters ====\n", + " bundle_info=abd.PediatricBundleDict(),\n", + " # ==== Tracking parameters ====\n", + " import_tract={\n", + " \"suffix\": \"tractography\", \"scope\": \"mrtrix\"},\n", + " # ==== Segmentation parameters ====\n", + " segmentation_params={\n", + " \"filter_by_endpoints\": False},\n", + " clean_params={\n", + " 'distance_threshold': 4},\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "43d28f0b", + "metadata": {}, + "source": [ + "### Step 3: Run pyAFQ\n", + "After defining your pyAFQ API object, you can ask for the output of any step of the pipeline. It is common for users to just call export_all (for example, myafq.export_all())." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4f9ad529", + "metadata": {}, + "outputs": [], + "source": [ + "myafq.export(\"template_xform\")" + ] + }, + { + "cell_type": "markdown", + "id": "9eeeaa8c", + "metadata": {}, + "source": [ + "On the left, the babyAFQ template from MNI space has been transformed into the subject's native space. On the right, the subject's b0 is shown for reference.\n", + "![warped_template](warped_template.png)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ca8d87f1", + "metadata": {}, + "outputs": [], + "source": [ + "bundle_html = myafq.export(\"all_bundles_figure\")\n", + "plotly.io.show(bundle_html[\"01\"][0])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "96bf335a", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "id": "091db304", + "metadata": {}, + "source": [ + "## Section 3: Two other features" + ] + }, + { + "cell_type": "markdown", + "id": "a0ad2b83", + "metadata": {}, + "source": [ + "### 1. Outputs can be used as inputs to AFQ-Insight\n", + "[AFQ-Insight](https://github.com/richford/AFQ-Insight) is an open source, python based statistical learning library for tractometry. Here are some features:\n", + "1. Format tract profiles for the popular [scikit-learn](https://scikit-learn.org/stable/) machine learning python library\n", + "2. Harmonize tract profiles across multiple sites using combat\n", + "3. Make regression and classification pipelines tailored to tract profiles\n", + "4. Use neural networks on tract profiles for prediction" + ] + }, + { + "cell_type": "markdown", + "id": "3a833619", + "metadata": {}, + "source": [ + "### 2. Using the pyAFQ configuration file\n", + "pyAFQ provides a [command-line interface (CLI)](https://yeatmanlab.github.io/pyAFQ/usage/index.html). After installing the software, and organizing the data, run:\n", + "\n", + " pyAFQ /path/to/config.toml\n", + "\n", + "At minimum, the file should contain the BIDS path:\n", + "\n", + " [BIDS_PARAMS]\n", + " bids_path = \"path/to/study\"\n", + "\n", + "But additional configuration options can be provided. Here is an example configuration file to run babyAFQ:\n", + "\n", + " [BIDS_PARAMS]\n", + " bids_path = \"example_bids_subject\"\n", + " preproc_pipeline = \"vistasoft\"\n", + "\n", + " [DATA]\n", + " reg_template_spec = \"pediatric\"\n", + " reg_subject_spec = \"b0\"\n", + " bundle_info=\"PediatricBundleDict()\"\n", + "\n", + " [TRACTOGRAPHY_PARAMS]\n", + " import_tract = \"{'suffix': 'tractography', 'scope': 'mrtrix'}\"\n", + "\n", + " [SEGMENTATION_PARAMS]\n", + " filter_by_endpoints = false\n", + " \n", + " [CLEANING_PARAMS]\n", + " distance_threshold = 4" + ] + }, + { + "cell_type": "markdown", + "id": "334881f8", + "metadata": {}, + "source": [ + "## Section 4: Summary\n", + " 1. pyAFQ is free and open source (easy to run on the cloud)\n", + "\n", + " 2. pyAFQ is easy to install and does not require knowledge of python to use\n", + "\n", + " 3. pyAFQ uses BIDS datasets\n", + "\n", + " 4. pyAFQ is customizable and integrates with other pipelines like MRtrix\n", + " \n", + "Documentation: https://yeatmanlab.github.io/pyAFQ/
\n", + "Code: https://github.com/yeatmanlab/pyAFQ
" + ] + }, + { + "cell_type": "markdown", + "id": "4db85fe0", + "metadata": {}, + "source": [ + "# Thank you for listening!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6d1cbb6d", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 54839464d56ed0640dc0d3e643cc658cbdb8f155 Mon Sep 17 00:00:00 2001 From: Ariel Rokem Date: Tue, 22 Aug 2023 20:40:52 -0700 Subject: [PATCH 2/2] Notebook works top to bottom --- examples/tutorial_examples/baby_afq.ipynb | 206 +++------------------- 1 file changed, 24 insertions(+), 182 deletions(-) diff --git a/examples/tutorial_examples/baby_afq.ipynb b/examples/tutorial_examples/baby_afq.ipynb index 426b77c07..b2475ebdb 100644 --- a/examples/tutorial_examples/baby_afq.ipynb +++ b/examples/tutorial_examples/baby_afq.ipynb @@ -1,113 +1,13 @@ { "cells": [ - { - "cell_type": "code", - "execution_count": null, - "id": "7dd23004", - "metadata": {}, - "outputs": [], - "source": [ - "import warnings\n", - "warnings.filterwarnings(\"ignore\")" - ] - }, { "cell_type": "markdown", "id": "4a1973ed", "metadata": {}, "source": [ - "# pyAFQ: Automated Fiber Quantification ... in Python" - ] - }, - { - "cell_type": "markdown", - "id": "44483d42", - "metadata": {}, - "source": [ - "## This notebook is available at: https://github.com/36000/babyAFQ_example\n" - ] - }, - { - "cell_type": "markdown", - "id": "2631a596", - "metadata": {}, - "source": [ - "## Section 1: Four benefits of using pyAFQ compared to matlab AFQ:" - ] - }, - { - "cell_type": "markdown", - "id": "88b0b126", - "metadata": {}, - "source": [ - "### 1. Free and open source\n", - " 1. No need for a MATLAB license" - ] - }, - { - "cell_type": "markdown", - "id": "3a345b2a", - "metadata": {}, - "source": [ - "### 2. Easier to run on cloud services\n", - " 1. For example, Amazon Web Services (AWS) or Microsoft Azure\n", - " 2. pyAFQ's only dependencies are other python packages which are installed automatically using \"Package Installer for Python (pip)\"\n", - "![](install_diagram.png)" - ] - }, - { - "cell_type": "markdown", - "id": "6cbd99d3", - "metadata": {}, - "source": [ - "### 3. Easier to run on BIDS datasets\n", - "1. The Brain Imaging Data Structure (BIDS) is a way of organizing neuroimaging and behavioral data\n", - "2. BIDS is a community effort\n", - "3. There are many [benefits to BIDS](https://bids.neuroimaging.io/benefits.html):\n", - " 1. It will be easy for another researcher to work on your data\n", - " 1. For example, by using BIDS you will save time trying to understand and reuse data acquired by a graduate student or postdoc that has already left the lab.\n", - " \n", - " 2. There is a growing number of data analysis software that understand BIDS data, like pyAFQ\n", - " \n", - " 3. Databases such as OpenNeuro.org, LORIS, COINS, XNAT, SciTran, and others will accept and export datasets organized according to BIDS.\n", - " 1. If you ever plan to share your data publicly (some journals require this) you can speed up the curation process by using BIDS.\n", - " \n", - " 4. There are validation tools (also available online) that can check your dataset integrity and let you easily spot missing values.\n", - " \n", - "![BIDS diagram](bids_diagram.png)\n", + "# Tutorial: run pyAFQ on data from infants\n", "\n", - "Image source: http://bids.neuroimaging.io/" - ] - }, - { - "cell_type": "markdown", - "id": "f8cdb35c", - "metadata": {}, - "source": [ - "### 4. Easier to customize methods and integrate with other pipelines\n", - " 1. pyAFQ provides tools to import tractography or mori group files from the original Matlab based AFQ to pyAFQ\n", - " 2. pyAFQ can use tractography from other pipelines like MRtrix3 or Qsiprep which implement other algorithms\n", - " like SIFT and ACT\n", - "\n", - "#### For babyAFQ, Use MRtrix3 for the tractography: https://www.mrtrix.org/" - ] - }, - { - "cell_type": "markdown", - "id": "5f834933", - "metadata": {}, - "source": [ - "## Section 2: Baby pyAFQ code examples" - ] - }, - { - "cell_type": "markdown", - "id": "96c2ed52", - "metadata": {}, - "source": [ - "### Step 1: Organizing the data\n", - "Put your dataset in BIDS. Here is an example dataset with one subject:\n", - "![Example BIDS](bids_example.png)" + "Infant brains are profoundly different from grownup brains." ] }, { @@ -115,37 +15,26 @@ "id": "a7204feb", "metadata": {}, "source": [ - "## This data is available at: https://tinyurl.com/35tptjz9" + "## Get some baby diffusion data: \n", + "\n", + "The data for this example, provided by [Kalanit Grill Spector's Stanford Vision and Perception Neuroscience Lab](https://vpnl.stanford.edu/) is available to download [on Figshare](https://figshare.com/articles/dataset/Example_babyAFQ_BIDS_subject/21440739). You can download it from there and unzip it into `~/AFQ_Data/baby_example/` (Note that this is 2.69GB of data, so it can take a while to download). This data has been previously published in Grotheer et al. (2022).\n" ] }, { "cell_type": "markdown", - "id": "5cb26a48", + "id": "cbe77e4a", "metadata": {}, "source": [ - "### Step 2: Initialize a GroupAFQ object\n", - "Creates a GroupAFQ object, that encapsulates tractometry. This object can be used to manage the entire AFQ pipeline, including:\n", - "\n", - " Tractography\n", - "\n", - " Registration\n", - "\n", - " Segmentation / Bundle Recognition\n", - "\n", - " Cleaning\n", - "\n", - " Profiling\n", - "\n", - " Visualization" + "In this case, a tractography has already been run with the excellent [MRtrix](https://www.mrtrix.org/) software. When you first run the following code, it will download the infant templates into your `~/AFQ_data/pediatric_templates` folder, organizing them there in the way that pyAFQ expects to find them." ] }, { - "cell_type": "markdown", - "id": "cbe77e4a", + "cell_type": "code", + "execution_count": null, + "id": "cdb5a2d1", "metadata": {}, - "source": [ - "In this case, MRtrix has already done the tractography (visualized using [MI-Brain](https://github.com/imeka/mi-brain)): ![mrtrix_track](3d_screenshot.png)" - ] + "outputs": [], + "source": [] }, { "cell_type": "code", @@ -157,12 +46,10 @@ "from AFQ.api.group import GroupAFQ\n", "import AFQ.api.bundle_dict as abd\n", "import AFQ.data.fetch as afd\n", - "from AFQ.definitions.image import LabelledImageFile\n", - "import plotly\n", - "\n", + "import os.path as op\n", "myafq = GroupAFQ(\n", " # ==== BIDS parameters ====\n", - " bids_path=\"example_bids_subject\",\n", + " bids_path=op.join(op.expanduser('~'), \"AFQ_data/example_bids_subject/\"),\n", " preproc_pipeline=\"vistasoft\",\n", " # ===== Registration parameters ====\n", " reg_template_spec=afd.read_pediatric_templates()[\"UNCNeo-withCerebellum-for-babyAFQ\"],\n", @@ -177,7 +64,7 @@ " \"filter_by_endpoints\": False},\n", " clean_params={\n", " 'distance_threshold': 4},\n", - ")" + ")\n" ] }, { @@ -215,45 +102,18 @@ "metadata": {}, "outputs": [], "source": [ + "import plotly\n", "bundle_html = myafq.export(\"all_bundles_figure\")\n", "plotly.io.show(bundle_html[\"01\"][0])" ] }, - { - "cell_type": "code", - "execution_count": null, - "id": "96bf335a", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "id": "091db304", - "metadata": {}, - "source": [ - "## Section 3: Two other features" - ] - }, - { - "cell_type": "markdown", - "id": "a0ad2b83", - "metadata": {}, - "source": [ - "### 1. Outputs can be used as inputs to AFQ-Insight\n", - "[AFQ-Insight](https://github.com/richford/AFQ-Insight) is an open source, python based statistical learning library for tractometry. Here are some features:\n", - "1. Format tract profiles for the popular [scikit-learn](https://scikit-learn.org/stable/) machine learning python library\n", - "2. Harmonize tract profiles across multiple sites using combat\n", - "3. Make regression and classification pipelines tailored to tract profiles\n", - "4. Use neural networks on tract profiles for prediction" - ] - }, { "cell_type": "markdown", "id": "3a833619", "metadata": {}, "source": [ - "### 2. Using the pyAFQ configuration file\n", + "### Another way to run this: Using the pyAFQ configuration file\n", + "\n", "pyAFQ provides a [command-line interface (CLI)](https://yeatmanlab.github.io/pyAFQ/usage/index.html). After installing the software, and organizing the data, run:\n", "\n", " pyAFQ /path/to/config.toml\n", @@ -266,7 +126,7 @@ "But additional configuration options can be provided. Here is an example configuration file to run babyAFQ:\n", "\n", " [BIDS_PARAMS]\n", - " bids_path = \"example_bids_subject\"\n", + " bids_path = \"/path/to/AFQ_data/example_bids_subject\"\n", " preproc_pipeline = \"vistasoft\"\n", "\n", " [DATA]\n", @@ -286,36 +146,18 @@ }, { "cell_type": "markdown", - "id": "334881f8", + "id": "6d1cbb6d", "metadata": {}, "source": [ - "## Section 4: Summary\n", - " 1. pyAFQ is free and open source (easy to run on the cloud)\n", - "\n", - " 2. pyAFQ is easy to install and does not require knowledge of python to use\n", - "\n", - " 3. pyAFQ uses BIDS datasets\n", + "# References \n", "\n", - " 4. pyAFQ is customizable and integrates with other pipelines like MRtrix\n", - " \n", - "Documentation: https://yeatmanlab.github.io/pyAFQ/
\n", - "Code: https://github.com/yeatmanlab/pyAFQ
" + "1. Grotheer M, Rosenke M, Wu H, Kular H, Querdasi FR, Natu VS, Yeatman JD, Grill-Spector K. White matter myelination during early infancy is linked to spatial gradients and myelin content at birth. Nat Commun. 2022 Feb 22;13(1):997. doi: 10.1038/s41467-022-28326-4. PMID: 35194018; PMCID: PMC8863985." ] }, { "cell_type": "markdown", - "id": "4db85fe0", + "id": "d8ca78c8", "metadata": {}, - "source": [ - "# Thank you for listening!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "6d1cbb6d", - "metadata": {}, - "outputs": [], "source": [] } ], @@ -335,7 +177,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.10.8" } }, "nbformat": 4,