[TOC]
From one-hot vector to word embedding
Embedding + NN
- efficient on large datasets
- State-of-the-art on several classification when adding bigrams/trigrams
- Little gains from depth
Unsupervised / self-supervised learning of word representations
Unlabelled text data is almost infinite
Self-supervised training : words are characterised by the company that they keep
- CBoW (continuous Bag-of-Word): context
$\rightarrow$ word- Large impact of context size
- Skip Gram: word
$\rightarrow$ context- Widely used in practice
- Negative Sampling is used as a cheaper alternative to full softmax
Other popular method: GloVe
- If little training data, use pre-trained self-supervised embeddings
- If large training data with labels, directly learn task-specific embedding with methods such as fastText in supervised mode
- Methods use Bag-of-words ignore the order in word sequence
- Depth & non-linear activations on hidden layers are not that useful for BoW text classification
Conditional Language models: use conditional probalility (usually a sequence of probability) to build the model
- Similar idea as LSTM, no systematic difference
- In practice, more recent, people tend to use LSTM more
- Gradient messages close to
$0$ can shrink be$0$ - gradient messages larger than
$1$ can explode - LSTM / GRU mitigate that in rnns