Contributed by Louis-Guillaume Gagnon.
- Using the definition of the derivative:
show that the derivative of the rectified linear unit ($relu(x) = max(0, x)$) is given by the Heaviside step function:
- Give an alternative definition of relu using . Can you think of a second one?
- Give an asymptotic expression for using the sigmoid function, .
- Using the same technique as in (a), show that the derivative of is given by the dirac delta function , defined in section 3.9.5 of the Deep Learning textbook.