Papers

ML Reference

Paper	Author	Info.
Attention is All You Need	Vaswani+17	Transformer
An Image is Worth 16x16 Words	Dosovitskiy+20
Linformer: Self-Attention with Linear Complexity	Wang+20	Linformer
Emerging Properties in Self-Supervised Vision Transformers	Caron+21
Do Vision Transformers See Like Convolutional Neural Networks?	Raghu+21	ViT v.s. CNN
Training data-efficient image transformers & distillation through attention	Touvron+21

Blog	Info.
The Illustrated Transformer	Transformer
The transformer family
How Facebook uses super-efficient AI models to detect hate speech	Transformer v.s. Linformer

Astrophysics Reference

• Galaxy Morphology Classification using EfficientNet Architectures -- Kalvankar+20

  • Training data: Kaggle galaxy challenge (61,578 galaxies).

  • 7 classes : completely round smooth (Er), in-between smooth (Ei), cigar-shaped smooth (Ec)
    edge-on
    barred spiral, unbarred spiral
    irregular (merger, disturbed morphology)

  • Clean galaxy sample: assign specific morphological catagories based on debiased vote fractions (Table 2 of Kalvankar+20).

  • Every class contains 8107, 7782, 578, 3780, 827, 3307 and 1560 samples respectively. The dataset reduces to 25,941 images that are divided in a train, test ratio of 9:1. Thus, we end up with 23,352 training images and 2589 testing images.

• Galaxy Morphology Classification with Deep Convolutional Neural Networks -- Dai & Tong 18

Quick link

• Overleaf