Activation functions are mathematical equations that determine the output of a neural network model. They are attached to each neuron in the network and determine whether it should be activated or not, based on whether the neuron's input is relevant for the model's prediction. They introduce non-linearity into the network, allowing it to learn complex patterns and relationships in the data. Common activation functions include Sigmoid, Tanh, ReLU (Rectified Linear Unit), and Softmax.
The concept is inspired by the biological action potential in the brain, where a neuron fires only when the input signal exceeds a certain threshold. The McCulloch-Pitts neuron (1943) used a step function, which was a precursor to modern activation functions.
Activation functions are a critical component of every deep learning model. The choice of activation function can significantly impact the training speed and the final performance of the network. ReLU has become the default choice for hidden layers in many architectures due to its computational efficiency and ability to mitigate the vanishing gradient problem.