Supongamos el WLOG $b > a$. $c = b/a$ Tenemos
$$\begin{align} I &= \int_0^\pi \log(a^2 + b^2 - 2ab \cos x) \, dx \\ &=\int_0^\pi \log a^2 \, dx + \int_0^\pi \log(1 + (b/a)^2 - 2(b/a) \cos x) \, dx \\ &= 2\pi \log a + \int_0^\pi \log(1 + c^2 - 2c \cos x) \, dx \end{align}$$
Ahora podemos evaluar la segunda integral en el lado derecho como el límite de una suma de Riemann
$$\begin{align}J &= \int_0^\pi \log(1 + c^2 - 2c \cos x) \, dx \\ &= \lim_{n \to \infty} \frac{\pi}{n}\sum_{j=1}^n\log(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &= \lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=1}^n(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &=\lim_{n \to \infty} \frac{\pi}{n}\log(1-c)^2+\lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=2}^{n}(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &= \lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=1}^{n-1}(1 + c^2 - 2c\cos(\pi j/n))\end{align} .$$
A factoring, obtenemos
$$1 + c^2 - 2c\cos(\pi j/n) = [c - \exp(i \pi j /n)][c - \exp(-i \pi j /n)],$$
y
$$c^{2n} -1 = (c-1)(c+1)\prod_{j=1}^{n-1}[c - \exp(i \pi j /n)][c - \exp(-i \pi j /n)].$$
Recordando que $c = b/a > 1$, por lo tanto, se deduce que
$$\begin{align}J &= \lim_{n \to \infty} \frac{\pi}{n} \log\left(\frac{c^{2n}-1}{c^2-1} \right) \\ &= \pi \lim_{n \to \infty} \log\left(\frac{c^{2n}-1}{c^2-1} \right)^{1/n} \\ &= \pi \lim_{n \to \infty} \log\left[c^2 \left(\frac{1-c^{-2n}}{c^2-1} \right)^{1/n}\right] \\ &= \pi \log c^2 + \log \left[\lim_{n \to \infty} \left(\frac{1-c^{-2n}}{c^2-1} \right)^{1/n}\right]\\ &= \pi \log c^2\end{align}$$
Por lo tanto,
$$I = 2\pi \log a + \pi \log c^2 = 2\pi \log a + \pi \log (b/a)^2 \\ = 2\pi \log b $$