Q5 – Softmax and Sigmoid

Contributed by  Antoine Lefebvre-Brossard

  1. Show that a 2-class softmax function can be rewritten a a sigmoid function. In
    other words that \text{softmax}(\boldsymbol{x}) = \sigma(z) where the softmax function is defined by  \text{softmax}(\boldsymbol{x}) = (\frac{e^{x_1}}{e^{x_1}+e^{x_2}}, \frac{e^{x_2}}{e^{x_1}+e^{x_2}})^T and the sigmoid function is defined by \sigma(z) = \frac{1}{1 + e^{-z}}
  2. Use the previous result to show that it’s possible to write a k-class softmax
    function as a function of k-1 variables.
Advertisements

4 thoughts on “Q5 – Softmax and Sigmoid

    • 1. We have to divide the numerator by exp (x1) and the denominator by exp (x1) for the first coordinate. And then we do the same thing by exp (x2) for the 2nd coordinate. We obtain z1 = x1-x2 and z2 = x2-x1. Thus for x = (x1, x2) and z = (z1 = x1-x2, z2 = x2-x1) softmax (x) = sigma(z)

      Like

  1. 2. We can notice in answer (1.) that z1 = -z2. So we can eliminate the variable x1 (or x2) because we have x1-x2 = z and x2-x1 = -z. This gives us x1 = z + x2.

    Suppose k = 3:
    If we then make the divisions which we have made in question (1.) over all 3 coordinates produced by softmax, we will have in the denominator for the three coordinates

    1 + exp (x2-x1) + exp (x3-x1),
    1 + exp (x1-x2) + exp (x3-x2),
    1 + exp (x1-x3) + exp (x2-x3)

    Let z1 = x1-x2 and z2 = x3-x1 then z1 + z2 = x3-x2. So the denominators of the three coordinates respectively will be:

    1 + exp (x2-x1) + exp (x3-x1) = 1+ exp (-z1) + exp (z2)
    1 + exp (x1-x2) + exp (x3-x2) = 1 + exp (z1) + (z1 + z2)
    1 + exp (-1-x3) + exp (x2-x3) = 1 +

    We see that we have k-1 variables z1 and z2 instead of x1, x2 and x3 (here k = 3). The same operations can be performed for k> 3.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s