Activation Functions

Table of Contents

1. Sigmoid Activation Function
2. Softmax Activation Function
3. Tanh Activation Function
4. ReLU Activation Function
5. Leaky ReLU Activation Function
6. Derivatives
7. Equations and Graphs

-- More Coming Soon --

1. Sigmoid Activation Function

• The Sigmoid Function curve looks like an S-shape
• Range: [0, 1]
• Application: binary classification

2. Softmax Activation Function

• The softmax function is a more generalized logistic activation function
• Application: multiclass classification.

3. tanh Activation Function

• tanh is also like the logistic sigmoid function (s-shaped) but better.
• Range: [-1, 1]
• Application: binary classification

4. ReLU Activation Function

• ReLU stands for Rectified Linear Unit
• The issue with it is that all the negative values become zero immediately which decreases the ability of the model to fit or train from the data properly
• Range: [0, infinity)
• Application: Convolutional Neural Network

5. Leaky ReLU Activation Function

• It is an attempt to solve the dying ReLU problem. The leak helps to increase the range of the ReLU function.
• The equation for negative y becomes f(y) = ay where a is 0.01
• When a is not 0.01 then it is called Randomized ReLU
• Range: (-infinity to infinity)

Derivatives

Equations and Graphs