The most popular choice, due to both simplicity of implementation and its good performance on a variety of predictive tasks, is the rectified linear unit (ReLU). RELU provides a very simple nonlinear...

program in node

Extracted text: The most popular choice, due to both simplicity of implementation and its good performance on a variety of predictive tasks, is the rectified linear unit (ReLU). RELU provides a very simple nonlinear transformation. Given an element r, the function is defined as the maximum of that element and 0: RELU(x) = max(r, 0). (4.1.4) Informally, the RelLU function retains only positive elements and discards all negative elements by setting the corresponding activations to 0. To gain some intuition, we can plot the function. As you can see, the activation function is piecewise linear.