Aquí uso la función Beta para evaluar esta integral y el cálculo es un poco tedioso. Como Omran Kouba hizo,
\begin{eqnarray}
I&=&\int_{0}^{\Large\frac{\pi}{2}} \ln^2 {(\sin x)}\cdot\ln {(\cos x)}\tan xdx.
\end{eqnarray}
Vamos
$$ J(a,b)=\int_0^{\frac{\pi}{2}}\sin^ax\cos^bx\tan xdx. $$
Entonces claramente
$$ \frac{\partial^3 J(0,0)}{\partial a^2\partial b}=I.$$
Ahora, por el cambio de las variables de $\sin x\to t$ y, a continuación,$t^2\to u$, es fácil ver
\begin{eqnarray}
J(a,b)=\frac{\Gamma(1+\frac{a}{2})\Gamma(\frac{b}{2})}{2\Gamma(1+\frac{a+b}{2})}.
\end{eqnarray}
Observando $\Gamma'(x)=\Gamma(x)\psi_0(x), \Gamma''(x)=\Gamma(x)\psi_0^2(x)+\Gamma(x)\psi_1(a)$, tenemos
\begin{eqnarray} \frac{\partial^2 J(a,b)}{\partial a^2}&=&\frac{\Gamma(1+\frac{a}{2})\Gamma(\frac{b}{2})}{8\Gamma(1+\frac{a+b}{2})}[\psi_0^2(1+\frac{a}{2})-2\psi_0(1+\frac{a}{2})\psi_0(1+\frac{a+b}{2})\\
&&+\psi_0^2(1+\frac{a+b}{2})+\psi_1(1+\frac{a}{2})-\psi_1(1+\frac{a+b}{2})].
\end{eqnarray}
El uso de
$$ \Gamma(1)=1, \psi_0(1)=\gamma, \psi_1(1)=\frac{\pi^2}{6}, $$
obtenemos
$$ \frac{\partial^2 J(0,b)}{\partial a^2}=\frac{\Gamma(\frac{b}{2})}{48\Gamma(1+\frac{b}{2})}(6\gamma^2+\pi^2+12\gamma\psi_0(1+\frac{b}{2})+6\psi_0^2(1+\frac{b}{2})-6\psi_1(1+\frac{b}{2}))) $$
y por lo tanto
\begin{eqnarray}
\frac{\partial^3 J(0,b)}{\partial a^2\partial b}&=&\frac{\Gamma(\frac{b}{2})}{96\Gamma(1+\frac{b}{2})}[-6\psi_0^3(1+\frac{b}{2})+6\psi_0^2(1+\frac{b}{2})(-2\gamma+\psi_0(1+\frac{b}{2}))\\
&&+\psi_0(\frac{b}{2})(6\gamma^2+\pi^2-6\psi_1(1+\frac{b}{2}))+12\gamma\psi_1(1+\frac{b}{2})+\gamma\psi_0(1+\frac{b}{2})(-6\gamma^2-\pi^2\\
&&+12\gamma\psi_0(\frac{b}{2})+18\gamma\psi_1(1+\frac{b}{2}))-6\gamma\psi_2(1+\frac{b}{2})).
\end{eqnarray}
Observando, por $z\in(0,1)$,
$$ \Gamma(1)=1, \Gamma(1-z)\Gamma(z)=\frac{\pi}{\sin(\pi z)} $$
tenemos
$$ \Gamma(\frac{b}{2})\approx\frac{\pi}{\sin(\frac{b\pi}{2})}, \psi_0(1+\frac{b}{2})=-\gamma+\frac{\pi^2b}{12}+O(b^2).$$
A partir de esto, hemos
\begin{eqnarray}
\psi_0(\frac{b}{2})&=&-\frac{2}{b}-\gamma+\frac{\pi^2b}{12}+O(b^2)\\
\psi_0(1+\frac{b}{2})&=&-\gamma+\frac{\pi^2b}{12}+O(b^2)\\
\psi_1(1+\frac{b}{2})&=&\frac{\pi^2}{6}+\frac{b}{2}\psi_2(1)+\frac{\pi^4b^2}{120}+O(b^3)\\
\psi_2(1+\frac{b}{2})&=&\psi_2(1)+\frac{\pi^4b}{30}+O(b^2),
\end{eqnarray}
y por lo tanto
\begin{eqnarray}
\frac{\partial^3 J(0,b)}{\partial a^2\partial b}&\approx&\frac{b\pi^3}{11520\sin(\frac{b\pi}{2})}(-2\pi^2+b^2\pi^4+60b\psi_2(1)).
\end{eqnarray}
Así
$$ \frac{\partial^3 J(0,0)}{\partial a^2\partial b}=\lim_{b\to0}\frac{\partial^3 J(0,b)}{\partial a^2\partial b}=\lim_{b\to0}\frac{b\pi^3}{11520\sin(\frac{b\pi}{2})}(-2\pi^2+b^2\pi^4+60b\psi_2(1))=-\frac{\pi^4}{2880}.$$