Trace–determinant stability criterion

By definition of the characteristic polynomial, $\lambda^2-(\text{tr}\,A)\lambda+\det A=0$ has roots $\lambda_{1,2}=\dfrac{\text{tr}\,A\pm\sqrt{(\text{tr}\,A)^2-4\det A}}{2}$, and by Vieta's formulas $\lambda_1+\lambda_2=\text{tr}\,A$ and $\lambda_1\lambda_2=\det A$.

If $\det A<0$, then $\lambda_1\lambda_2<0$, so the eigenvalues are real with opposite signs (a saddle); solutions grow without bound along the positive-eigenvalue direction, so the origin is unstable.

If $\det A>0$ and $\text{tr}\,A<0$: when the eigenvalues are real, $\lambda_1\lambda_2>0$ means they share a sign, and $\lambda_1+\lambda_2<0$ forces that sign to be negative, so both $e^{\lambda_1t}\to 0$ and $e^{\lambda_2t}\to 0$; when the eigenvalues are complex conjugates $\alpha\pm i\beta$, $\text{tr}\,A=2\alpha<0$ gives $\alpha<0$, so the amplitude factor $e^{\alpha t}\to 0$ while $\cos(\beta t),\sin(\beta t)$ stay bounded. In every sub-case, every solution decays to $\mathbf{0}$: asymptotic stability.

Conversely if $\text{tr}\,A>0$ (with $\det A>0$, so real eigenvalues share the trace's sign, or complex with positive real part), at least one exponential factor grows, so solutions starting near but not at the origin move away: instability. This covers every sign combination, proving the equivalence.