This project is inspired by the publication "Neurotransmitter Classification from Electron Microscopy Images at Synaptic Sites in Drosophila"
Utilizing the same dataset, presented here is a simple web-app that performs in-browser similarity searches among ~100k 2D images of synapses from a fruitfly brain. It is designed to allow the user to explore similar-looking synapses from other neuronal subtypes.
Does it work? Kinda! Unlike some previous image similarity work Ive done (Facial rec, satellite imagery), my brain finds it much harder to evaluate whether or not the similarities are sensible. This is also the first time Ive used the in-browser PGVector database, as opposed to server-side platforms like Milvus or Annoy. I may revisit this project in the future with a larger scale and a server-side database.
Pointlike synapses (T-bars?)
Larger semicircular synapses
Synapses with big-honkin mito!
Although the live demo uses an off-the-shelf imagenet vision transformer (XCIT), an attempt was made to train a classifier. I obtained similar performance to the paper, however I found that it did not seemingly perform as well for simularity searches. I hypothesize this is because the neuronal subtypes become isolated in the embedding space.
This was a quick weekend project, so I decided to train a 2D classifier instead of a 3D convnet. I did however exploit the 3 RGB channels of pretrained networks to look at a stack of 3 depthwise tiles for each synapse. The images are all from the Full Adult Fly Brain "FAFB". Since the volumes don't appear to be isotropic in the Z axis, making use of existing RGB channels of pretrained networks might find some suitable depthwise patterns.
