# Q9 – Variational Lower Bound

Contributed by Matthew Zak.

1. Using the property of a concave function: $\log(\sum_i\alpha_ix_i)\geq\alpha_i\log(x_i)$ show that $\log(p(x))\geq\sum_hg(h \mid x)\log(p(x,h))-\sum_hq(h\mid x)\log(q(h\mid x))$ where $p(x,y)$ describes the true conditional probability distribution over x using a latent variable $h$ ($p(x)$ equals $\sum_hp(x, h)$) and $q(h\mid x)$ is an approximation of $p(h\mid x)$.
2. Show that if $q(h\mid x) = p(h\mid x)$ and $p(x,h) = p(h \mid x)*p(x)$ then the variational lower bound (given by second equation) equals a logarithm of true conditional distribution $\log(p(x))$
3. Knowing that the difference between variational lower bound and the data distribution is given by $KL_(q||p)=\sum_h q(h\mid x)\log(\frac{q(h\mid x)}{p(h\mid x)})$, prove the limit of the KL divergence is $0$ as $q(h\mid x)$ approaches $p(h\mid x)$.

## 3 thoughts on “Q9 – Variational Lower Bound”

1. I have a small doubt. Should the property of the concave function for the log function be $log(\sum_i \alpha_i x_i) \geq \sum_i \alpha_i log(x_i)$?
Also, are we supposed to post our answers here for discussions?
Thanks!

Like

• Stéphanie Larocque says:

Yes, there is a sum on the right term as well.
And I think that posting your answers is one of the purposes of this Q/A stuff

Like

2. This is my answer for Q1.

As $p(x) = \sum_h p(x,h)$, we can write $\text{log} p(x) = \text{log} \sum_h$ $p(x, h) = \text{log} \sum_h \frac{q(h|x) p(x,h)}{q(h|x)}$.
As $\sum_h q(h|x) = 1$ and $\text{log}x$ is a concave function, it is possible to use the property of the concave function (Jensen’s inequality). Thus,
$\text{log}p(x) \geq \sum_h q(h|x) \text{log} (\frac{p(x,h)}{q(h|x)})$,
$\text{log}p(x) \geq \sum_h q(h|x) [\text{log}p(x,h) - \text{log}q(h|x)]$ ,
$\text{log}p(x) \geq \sum_h$ $q(h|x)\text{log}p(x,h) - \sum_h q(h|x)\text{log}q(h|x)$

Like