Premisa: esta auto-respuesta se ha añadido "tomando la inspiración" de las dos anteriores, especialmente la de @Travis.
Ya que, como de costumbre, es posible escribir:
$$
\begin{aligned}
A\cos x+B\sin x
& = \sqrt{A^2 + B^2}\left(\frac{A}{\sqrt{A^2+B^2}}\cos x + \frac{B}{\sqrt{A^2+B^2}}\sin x\right) \\
& = C\left(\cos\varphi\cos x + \sin\varphi\sin x\right) \\
& = C\cos(x-\varphi)
\end{aligned}
$$
de ello se sigue que:
$$
I(C)
:= \int_0^{2\pi}\arctan\left(\left(A\cos x+B\sin x\right)^2\right)\text{d}x
= \int_0^{2\pi}\arctan\left(\left(C\cos(x-\varphi)\right)^2\right)\text{d}x
$$
que la colocación de $y = x - \varphi$, gracias a la periodicidad de las integrando, equivalente a la escritura (1):
$$
I(C)
= \int_{0-\varphi}^{2\pi\varphi}\arctan\left(\left(C\cos y\right)^2\right)\text{d}y
= \int_0^{2\pi}\arctan\left(C^2\cos^2 y\right)\text{d}y\,.
$$
Entonces, la diferenciación bajo el signo integral, tenemos:
$$
\begin{aligned}
I'(C)
& = \int_0^{2\pi} \frac{2C\cos^2 y}{1+C^4\cos^4 y}\,\text{d}y \\
& = \int_0^{\frac{\pi}{2}} \frac{8C\frac{1+\cos(2y)}{2}}{1+C^4\left(\frac{1+\cos(2y)}{2}\right)^2}\,\text{d}y \\
& = \int_0^{\pi} \frac{4C\left(1+\cos(z)\right)}{1+\frac{C^4}{4}\left(1+\cos(z)\right)^2}\,\frac{1}{2}\text{d}z \\
& = \int_0^{\infty} \frac{2C\left(1+\frac{1-t^2}{1+t^2}\right)}{1+\frac{C^4}{4}\left(1+\frac{1-t^2}{1+t^2}\right)^2}\,\frac{2}{1+t^2}\text{d}t \\
& = \int_0^{\infty} \frac{8C}{C^4+\left(1+t^2\right)^2}\text{d}t \\
& = \int_0^{\infty} \frac{8C}{\left(C^2 + \text{i}\left(1+t^2\right)\right)\left(C^2 - \text{i}\left(1+t^2\right)\right)}\text{d}t \\
& = \frac{4\text{i}}{C}\left(\int_0^{\infty} \frac{1}{1+\text{i}C^2+t^2}\text{d}t - \int_0^{\infty} \frac{1}{1-\text{i}C^2+t^2}\text{d}t\right) \\
& = \frac{4\text{i}}{C}\left(\frac{1}{\sqrt{1+\text{i}C^2}}\int_0^{\infty} \frac{\frac{1}{\sqrt{1+\text{i}C^2}}}{1+\left(\frac{t}{\sqrt{1+\text{i}C^2}}\right)^2}\text{d}t - \frac{1}{\sqrt{1-\text{i}C^2}}\int_0^{\infty} \frac{\frac{1}{\sqrt{1-\text{i}C^2}}}{1+\left(\frac{t}{\sqrt{1-\text{i}C^2}}\right)^2}\text{d}t\right) \\
& = \frac{4\text{i}}{C}\left(\frac{\arctan\left(\frac{t}{\sqrt{1+\text{i}C^2}}\right)}{\sqrt{1+\text{i}C^2}}-\frac{\arctan\left(\frac{t}{\sqrt{1-\text{i}C^2}}\right)}{\sqrt{1-\text{i}C^2}}\right)_{t=0}^{t=\infty} \\
& = \frac{4\text{i}}{C}\left(\frac{\frac{\pi}{2}}{\sqrt{1+\text{i}C^2}}-\frac{\frac{\pi}{2}}{\sqrt{1-\text{i}C^2}}\right) - \frac{4\text{i}}{C}\left(\frac{0}{\sqrt{1+\text{i}C^2}}-\frac{0}{\sqrt{1-\text{i}C^2}}\right) \\
& = \frac{2\pi}{C}\frac{\text{i}\left(\sqrt{1-\text{i}C^2}-\sqrt{1+\text{i}C^2}\right)}{\sqrt{1+C^4}} \\
\end{aligned}
$$
y observando que:
$$
\left(\frac{\text{i}\left(\sqrt{1-\text{i}C^2}-\sqrt{1+\text{i}C^2}\right)}{\sqrt{1+C^4}}\right)^4 = \left(\frac{-2\left(-1+\sqrt{1+C^4}\right)}{1+C^4}\right)^2
$$
obtenemos:
$$
I'(C)
= \frac{2\pi}{C}\sqrt{\frac{2\left(-1+\sqrt{1+C^4}\right)}{1+C^4}}\,,
$$
mientras, la integración, tenemos (2):
$$
\begin{aligned}
I(C)
& = 2\pi\int \frac{2}{C\sqrt{1+C^4}}\sqrt{\frac{-1+\sqrt{1+C^4}}{2}}\,\text{d}C \\
& = 2\pi\int \frac{\frac{2}{C\sqrt{1+C^4}}\sqrt{\frac{-1+\sqrt{1+C^4}}{2}}\left(1+\frac{-1+\sqrt{1+C^4}}{2}\right)}{1+\frac{-1+\sqrt{1+C^4}}{2}}\,\text{d}C \\
& = 2\pi\int \frac{\frac{2}{C\sqrt{1+C^4}}\,\frac{C^4}{4\sqrt{\frac{-1+\sqrt{1+C^4}}{2}}}}{1+\frac{-1+\sqrt{1+C^4}}{2}}\,\text{d}C \\
& = 2\pi\int \frac{\frac{1}{2\sqrt{\frac{-1+\sqrt{1+C^4}}{2}}}\,\frac{C^3}{\sqrt{1+C^4}}}{1+\frac{-1+\sqrt{1+C^4}}{2}}\,\text{d}C \\
& = 2\pi\arctan\left(\sqrt{\frac{-1+\sqrt{1+C^4}}{2}}\,\right) + k\,.
\end{aligned}
$$
Dado que a partir de (1) tenemos $I(0) = 0$ y de (2) tenemos $I(0) = k$, podemos deducir que $k = 0$ y por tanto:
$$
\int_0^{2\pi}\arctan\left(C^2\cos^2 y\right)\text{d}y
= 2\pi\arctan\left(\sqrt{\frac{-1+\sqrt{1+C^4}}{2}}\,\right)
$$
es decir:
$$
\int_0^{2\pi}\arctan\left(\left(A\cos x+B\sin x\right)^2\right)\text{d}x
= 2\pi\arctan\left(\sqrt{\frac{-1+\sqrt{1+\left(A^2+B^2\right)^2}}{2}}\,\right),
$$
como queríamos demostrar.