Stability of dynamical systems

Near $x^{*}$ the nonlinear term is much smaller than the linear term, so if the linear part contracts every direction exponentially, that contraction dominates and drags the true trajectory back to $x^{*}$ as well.

Since every eigenvalue of $A$ has negative real part ($A$ is Hurwitz), the matrix equation $A^{T}P+PA=-I$ has a unique symmetric positive-definite solution $P$, given explicitly by the convergent integral $P=\int_0^{\infty}e^{A^{T}t}e^{At}\,dt$.

Define the candidate Lyapunov function $V(\xi)=\xi^{T}P\xi\ge0$ for the deviation $\xi=x-x^{*}$. Writing the nonlinear system as $\dot\xi=A\xi+g(\xi)$ with $g(\xi)=o(\|\xi\|)$ as $\xi\to0$ (the Taylor remainder), compute the time-derivative of $V$ along nonlinear trajectories.

Using $\dot\xi=A\xi+g(\xi)$: $\dot V=\dot\xi^{T}P\xi+\xi^{T}P\dot\xi=\xi^{T}(A^{T}P+PA)\xi+2\xi^{T}Pg(\xi)=-\|\xi\|^{2}+2\xi^{T}Pg(\xi)$. Since $g(\xi)=o(\|\xi\|)$, there is a neighborhood of $x^{*}$ where $2\xi^{T}Pg(\xi)\le\frac{1}{2}\|\xi\|^{2}$, so $\dot V\le-\frac{1}{2}\|\xi\|^{2}<0$ for $\xi\ne0$ in that neighborhood.

Because $V$ is positive-definite and $\dot V$ is negative-definite near $x^{*}$, Lyapunov's direct method (proved next) concludes trajectories starting in that neighborhood converge to $x^{*}$: asymptotic stability. For the unstable case, if $A$ has an eigenvalue with positive real part, a symmetric construction (Chetaev's instability theorem) exhibits a direction along which $V$ increases, showing trajectories are pushed away from $x^{*}$ no matter how close they start.

Think of $V$ as a generalized energy: if energy never increases along trajectories and is zero only at $x^{*}$, the system cannot wander away from $x^{*}$, and if energy strictly decreases it must eventually settle at the only point where it is zero.

Fix a small $\varepsilon>0$ small enough that the closed ball $\{\|x-x^{*}\|\le\varepsilon\}$ lies in the domain where $V$ is defined and positive away from $x^{*}$, and let $S_\varepsilon=\{\|x-x^{*}\|=\varepsilon\}$ be its boundary sphere. Since $V$ is continuous and strictly positive on the compact set $S_\varepsilon$, it attains a minimum $m=\min_{S_\varepsilon}V>0$.

By continuity of $V$ and $V(x^{*})=0$, choose $\delta\in(0,\varepsilon)$ small enough that $\|x(0)-x^{*}\|<\delta$ implies $V(x(0))<m$.

Because $\dot V\le0$ along trajectories, $V(x(t))\le V(x(0))<m$ for every $t\ge0$ that the solution exists. Every point of $S_\varepsilon$ has $V\ge m$, so the trajectory can never reach $S_\varepsilon$; by continuity of $t\mapsto x(t)$, it must remain strictly inside the ball of radius $\varepsilon$ for all $t\ge0$. This is exactly the definition of Lyapunov stability.

If in addition $\dot V(x)<0$ strictly for $x\ne x^{*}$ nearby, then $V(x(t))$ is strictly decreasing and bounded below by $0$, so it converges to some limit $c\ge0$. If $c>0$ were positive, the trajectory would stay in the compact annulus where $V\ge c>0$, on which $\dot V$ attains a strictly negative maximum, forcing $V$ to decrease past $c$ in finite time — a contradiction. Hence $c=0$, and since $V$ is positive-definite this forces $x(t)\to x^{*}$: asymptotic stability.