Brain Visualisation

Based on finetuned BioBERT, for visualising different brain regions from text description.

Different NLP approaches are stored in the helpers/experiments directory.

Setup to run the full visualisation (Node.js, conda/mamba required, cuda for faster execution)

1. Clone or download the project.
2. Start with this command in the root directory.

   npm install

   This will install the necessary libraries, depending on the existing version of node installed some errors need to be resoled.
3. Create a file local.config.json based on template_local.config.json. This defines which model will be used to predict the brain regions. Update config.json if the typically used localhost:3000 is not available.
4. Create conda env for running the model.

   conda env create -f environment.yml

5. Update local.config.json with the exact conda_model and bert_model paths.
6. Start the server with visualisations.

   npm start

   This will start the visualisation server at http://localhost:3000 in your browser.

The available functionalities are listed below.

• Uploading pdf/html file.
• Opening the file in a browser.
• Highlighting the text connects to the model, that read the text, and generates the image. The image is also saved in the output directory.

Data

The uploaded data includes the following elements.

• train folder with the dataset used to train the model
• test folder with the dataset used to test the model
• validation dataset folder with the dataset used to validate the model
• AAL3v1 files (.nii|.txt) with the data extracted from the AAL brain atlas
• AAL3v1 structured dictionary (AAL3v1.json) with the data extracted from the AAL brain atlas
• AAL dictionary from the relevant paper (paper_ALL3.json) with the data extracted from the AAL brain atlas
• Uberon anatomy ontology, filtered for brain entities (brain-view.json)
• Structured Uberon brain regions with names and definitions (brain_uberon_structured.json)
• Downloaded PubMED abstracts (abstracts.csv)
• Annotated PubMED abstracts, with either B-Brain or O annotations (annotated_abstracts.csv)
• Annotated brain regions connl file for NER finetuning with either B-Brain or O annotations (annotated_brain_regions.connl)

Data preprocessing

The main question is what brain regions should be used for text extraction and later visualisation. There are the major parts (cerebrum, brainstem, cerebellum), lobes (frontal, temporal, parietal, occipital), individual structures (hippocampus, amygdala). Each region can be subdivided into smaller parts, there are different networks (DMN) and systems.

Each atlas for visualisations has a different naming system and split, with AAL3v1 and schaeffer being the most commonly used for brain regions. Most of the experiments in the repository have focues on either structures of the brain or aal atlas regions.

During the data preprocessing part, new dictionaries and connections were generated, and stored in the data folder.

Experiments

The helper\experiments folder contains Jupyter notebooks with experiments and different approaches to the given problem.

brain_text_detection.ipynb: the simplest approach

ner_tutorial.ipynb based on medical NER tutorial

The brain regions used in the notebook are the most common ones, they do not come from any atlas or dictionary. First, the model was trained to recognize whether the text is brain related or not, based on PubMED abstracts fetched for the list of regions.

Epoch	Training Loss	Validation Loss	Precision	Recall	F1
1	No log	0.282965	0.201717	0.256831	0.225962
2	No log	0.154976	0.426724	0.540984	0.477108
3	No log	0.122615	0.542373	0.699454	0.610979
4	No log	0.100538	0.666667	0.765027	0.712468
5	No log	0.095848	0.669725	0.797814	0.728180

brain2qwerty and how it connects with hierarchy of brain regions

Final Models

Normally the embeddings and safetensors would not be uploaded, and ignored, but they were uploaded using LFS for convenience.

Results

The simple version of biobert, with embeddings based on AAL3v1 regions of the brain but with no extra pre-training works well only when the text is closely related to the name of the region. The probability of each region is listed in the brackets. The more complex version of biobert, trained on annotated PubMed abstracts. The dictionary comes from the AAL3v1 atlas with 170 unique brain regions.

The most complex model was trained on hierarchical embeddings. AAL atlas has brain regions that are extremely common (hippocampus) and others that are quite rare. The goal is to highlight the most relevant regions.

References

1. Medium NER tutorial link
2. BioBERT-CRF paper link
3. How to choose tools for unique BioNER need link
4. BioSERPBERT repository of neuroscience representation of brain region text mining link
5. Connectivity search engine with different connected embeddings link
6. Data fetching tutorial link
7. Token classification link
8. Brain2Qwerty link
9. Noninvasive decoding of typed sentences from human brain activity link