Hoy encontré una prueba de este problema. La prueba es bastante desordenada.
Ten en cuenta que como $A,U$ son simétricas, tenemos $$ \mathrm{tr}\left(AU\right)=\sum_{i,j=1}^{2}a^{ij}u_{ij}. $$ Dado que $A$ y $U$ son matrices simétricas, podemos diagonalizarlas, digamos $$ A=P\begin{bmatrix}\lambda_{1} & 0\\ 0 & \lambda_{2} \end{bmatrix}P^{T},\quad U=Q\begin{bmatrix}\varepsilon_{1} & 0\\ 0 & \varepsilon_{2} \end{bmatrix}Q^{T}, $$ donde $P$ y $Q$ son matrices ortogonales. Podemos asumir que $\lambda_{1}\ge\lambda_{2}$ y $\varepsilon_{1}\ge\varepsilon_{2}$. Entonces \begin{align*} \mathrm{tr}\left(AU\right) & =\mathrm{tr}\left(P\begin{bmatrix}\lambda_{1} & 0\\ 0 & \lambda_{2} \end{bmatrix}P^{T}Q\begin{bmatrix}\varepsilon_{1} & 0\\ 0 & \varepsilon_{2} \end{bmatrix}Q^{T}\right)\\ & =\mathrm{tr}\left(Q^{T}P\begin{bmatrix}\lambda_{1} & 0\\ 0 & \lambda_{2} \end{bmatrix}P^{T}Q\begin{bmatrix}\varepsilon_{1} & 0\\ 0 & \varepsilon_{2} \end{bmatrix}\right). \end{align*} Dado que $P,Q$ son matrices ortogonales, $P^{T}Q$ también son matrices ortogonales. Denotemos esto como $R=P^{T}Q$. Escribimos $$ R=\begin{bmatrix}a & b\\ c & d \end{bmatrix} $$ con $a^{2}+c^{2}=1$ y $b^{2}+d^{2}=1$, $a^{2}+b^{2}=1$, $c^{2}+d^{2}=1$. Entonces esto implica que $a^{2}=d^{2}$ y $b^{2}=c^{2}$. Así que \begin{align*} \mathrm{tr}\left(AU\right) & =a^{2}\lambda_{1}\varepsilon_{1}+b^{2}\lambda_{1}\varepsilon_{2}+c^{2}\lambda_{2}\varepsilon_{1}+d^{2}\lambda_{2}\varepsilon_{2}\\ & =a^{2}\left(\lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2}\right)+b^{2}\left(\lambda_{1}\varepsilon_{2}+\lambda_{2}\varepsilon_{1}\right). \end{align*>
Nota $$ \mathrm{tr}\left(AU\right)^{2}=a^{4}\left(\lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2}\right)^{2}+2a^{2}b^{2}\left(\lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2}\right)\left(\lambda_{1}\varepsilon_{2}+\lambda_{2}\varepsilon_{1}\right)+b^{4}\left(\lambda_{1}\varepsilon_{2}+\lambda_{2}\varepsilon_{1}\right)^{2} $$
Dividimos en tres casos
- (i) $\varepsilon_{1}\ge\varepsilon_{2}\ge0$
- (ii) $0\ge\varepsilon_{1}\ge\varepsilon_{2}$
- (iii) $\varepsilon_{1}\ge0\ge\varepsilon_{2}$
Estos son todos cálculos tediosos:
(i) Nota que \begin{align*} \lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2} & \ge\mu\left(\varepsilon_{1}+\varepsilon_{2}\right)\\ \lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2} & \ge\mu\left(\varepsilon_{1}+\varepsilon_{2}\right). \end{align*> Entonces \begin{align*} \mathrm{tr}\left(AU\right)^{2} & \ge\mu^{2}\left(\varepsilon_{1}+\varepsilon_{2}\right)^{2}\left[a^{4}+2a^{2}b^{2}+b^{4}\right]\\ & =\mu^{2}\left(\varepsilon_{1}^{2}+\varepsilon_{2}^{2}+2\varepsilon_{1}\varepsilon_{2}\right)\\ & \ge\mu^{2}\left(\frac{\mu^{2}}{\nu^{2}}\left(\varepsilon_{1}^{2}+\varepsilon_{2}^{2}\right)+2\varepsilon_{1}\varepsilon_{2}\right).
(ii) Nota que \begin{align*> 0\ge\mu\left(\varepsilon_{1}+\varepsilon_{2}\right) & \ge\lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2}\ge\nu\left(\varepsilon_{1}+\varepsilon_{2}\right)\\ 0\ge\mu\left(\varepsilon_{1}+\varepsilon_{2}\right) & \ge\lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2}\ge\nu\left(\varepsilon_{1}+\varepsilon_{2}\right). \end{align*> Entonces \begin{align*} \mathrm{tr}\left(AU\right)^{2} & \ge\mu^2\left(\varepsilon_{1}+\varepsilon_{2}\right)^2\left[a^4+2a^2b^2+b^4\right]\\ & =\mu^2\left(\varepsilon_{1}^2+\varepsilon_{2}^2+2\varepsilon_{1}\varepsilon_{2}\right)\\ & \ge\mu^2\left(\frac{\mu^{2}}{\nu^{2}}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right)+2\varepsilon_{1}\varepsilon_{2}\right).
(iii) Nota que $$ \nu\varepsilon_{2}+\mu\varepsilon_{1}\le\mathrm{tr}\left(AU\right)\le\nu\varepsilon_{1}+\mu\varepsilon_{2}. $$ Entonces debemos dividir en tres casos.
(1) $\mathrm{tr}\left(AU\right)=0$. Dado que $\nu\varepsilon_{2}+\mu\varepsilon_{1}\le0\le\mu\varepsilon_{2}+\nu\varepsilon_{1}$, tenemos $$ \left(\nu\varepsilon_{2}+\mu\varepsilon_{2}\right)\left(\mu\varepsilon_{2}+\nu\varepsilon_{1}\right)\le0. $$ Reescribimos esta desigualdad. Entonces $$ \mu\nu\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right)+\left(\mu^2+\nu^2\right)\varepsilon_{1}\varepsilon_{2}\le0. $$ Por lo tanto, \begin{align*> \varepsilon_{1}\varepsilon_{2} & \le-\frac{\mu\nu}{\mu^2+\nu^2}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right)\\ & =-\frac{1}{\frac{\mu}{\nu}+\frac{\nu}{\mu}}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right)\\ & \le-\frac{1}{2\frac{\nu^2}{\mu^2}}\left(\varepsilon_{1}^2+\varepsilon_{2}\right)\\ & =-\frac{\mu^2}{2\nu^2}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right).
Por lo tanto $$ \frac{\mu^2}{\nu^2}\left(\mu^2\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right)+2\nu^2\varepsilon_{1}\varepsilon_{2}\right)\le0\le\mathrm{tr}\left(AU\right)^2, $$ es decir, $$ \mathrm{tr}\left(AU\right)^2\ge\frac{\mu^4}{\nu^2}\mathrm{tr}\left(U^2\right)+2\det\left(U\right). $$
(2) Supongamos que $\mathrm{tr}\left(AU\right)\neq0$ y $\left|\nu\varepsilon_{2}+\mu\varepsilon_{1}\right|\le\left|\nu\varepsilon_{1}+\mu\varepsilon_{2}\right|$: \begin{align*> \frac{1}{2\mu^2}\mathrm{tr}\left(AU\right)^2 & \ge\frac{1}{2\mu^2}\left(\nu\varepsilon_{2}+\mu\varepsilon_{1}\right)^2\\ & \ge\frac{1}{2\nu^2}\left(\nu^2\varepsilon_{2}^2+\mu^2\varepsilon_{1}^2+2\mu\nu\varepsilon_{1}\varepsilon_{2}\right)\\ & \ge\frac{1}{2\nu^2}\left(\nu^2\varepsilon_{2}^2+\mu^2\varepsilon_{1}^2\right)+\varepsilon_{1}\varepsilon_{2}.
Nota que $$ \frac{1}{2\nu^2}\left(\nu^2\varepsilon_{2}^2+\mu^2\varepsilon_{1}^2\right)\ge\frac{\mu^2}{2\nu^2}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right). $$ Por lo tanto, en este caso, obtenemos la desigualdad deseada.
(3) Supongamos que $\mathrm{tr}\left(AU\right)\neq0$ y $\left|\nu\varepsilon_{2}+\mu\varepsilon_{1}\right|>\left|\nu\varepsilon_{1}+\mu\varepsilon_{2}\right|$: \begin{align*> \frac{1}{2\mu^2}\mathrm{tr}\left(AU\right)^2 & \ge\frac{1}{2\mu^2}\left(\nu\varepsilon_{1}+\mu\varepsilon_{2}\right)^2\\ & \ge\frac{1}{2\nu^2}\left(\nu^2\varepsilon_{1}^2+\mu^2\varepsilon_{2}^2+2\mu\nu\varepsilon_{1}\varepsilon_{2}\right)\\ & =\frac{1}{2\nu^2}\left(\nu^2\varepsilon_{1}^2+\mu^2\varepsilon_{2}^2\right)+\varepsilon_{1}\varepsilon_{2}.
Nota que $$ \frac{1}{2\nu^2}\left(\nu^2\varepsilon_{1}^2+\mu^2\varepsilon_{2}^2\right)\ge\frac{\mu^2}{2\nu^2}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right). $$
Por lo tanto, en este caso, obtenemos la desigualdad deseada.
Con todos estos tres casos, obtenemos la siguiente desigualdad:
$$ \left[a^2\left(\lambda_{1}\varepsilon_{1}+\lambda_{2}\varepsilon_{2}\right)+b^2\left(\lambda_{1}\varepsilon_{2}+\lambda_{2}\varepsilon_{1}\right)\right]^2\ge\mu^2\left(\frac{\mu^2}{\nu^2}\left(\varepsilon_{1}^2+\varepsilon_{2}^2\right)+2\varepsilon_{1}\varepsilon_{2}\right). $$ Eso es, \begin{align*> \mathrm{tr}\left(AU\right)^2 & \ge\mu^2\left(\frac{\mu^2}{\nu^2}\mathrm{tr}\left(U^2\right)+2\det U\right)\\ & =\frac{\mu^4}{\nu^2}\mathrm{tr}\left(U^2\right)+2\mu^2\det U.
Por lo tanto, $$\frac{1}{2\mu^2}\left(\sum_{i,j=1}^{2}a^{ij}u_{ij}\right)^2\ge\frac{\mu^2}{2\nu^2}\sum_{i,j=1}^{2}u_{ij}^2+\det U.$$ Esto completa la prueba.