The Polar Decomposition

Let’s unpack the identity $O_t=(M_t{M}_t^T{)}^{-1/2}{M}_t=M_t(M_t^T{M}_t{)}^{-1/2}$ and understand why it holds, how it comes from polar decomposition / SVD, and what the pieces mean. I’ll start with some linear algebra background, then show the derivation, then explain intuitive meaning.

Background: Polar decomposition & SVD

1. Singular Value Decomposition (SVD)

• For any real matrix $M\in {\mathbb{R}}^{n\times m}$, there is a factorization

$$M=U\Sigma V^T$$

where

• $U\in {\mathbb{R}}^{n\times r}$, $V\in {\mathbb{R}}^{m\times r}$ have orthonormal columns (with $r=\text{rank}(M)$).
• $\Sigma \in {\mathbb{R}}^{r\times r}$ is diagonal with nonnegative singular values ${\sigma }_1,\dots ,{\sigma }_r$, sorted decreasingly.

2. Polar decomposition

• Polar decomposition says that any (real) matrix $M$ can be written as

$$M=O\cdot H$$

• where:
  • $O$ is an orthogonal (or semi-orthogonal) matrix (if $M$ isn’t square / full rank, some relaxed version).
  • $H$ is Symmetric Positive Definite Matrices (SPD Matrices).
• One canonical construction is: let

$$H=(M^{T}M{)}^{1/2}$$

• Then define

$$O=M{H}^{-1}$$

(assuming $M^{T}M$ is invertible, or using pseudo-inverse if rank deficient).

• Alternatively,

$$O=(M{M}^T{)}^{-1/2}M$$

are equivalent forms (they both give the same $O$), where the inverse or inverse square root is suitably defined.

Derivation of the identity

Starting from the SVD:

$$M=U\Sigma V^T$$

• $U$ is $n\times r$, $V$ is $m\times r$.
• $\Sigma$ is $r\times r$, diagonal, with positive singular values $\{{\sigma }_i\}$.

Compute $(M^{T}M{)}^{1/2}$

$$M^{T}M=(U\Sigma V^T{)}^T(U\Sigma V^T)=V\Sigma U^{T}U\Sigma V^T=V{\Sigma }^2{V}^T$$

because $U^{T}U=I_r$. So

$$(M^{T}M{)}^{1/2}=V\Sigma V^T$$

Then

$$M(M^{T}M{)}^{-1/2}=(U\Sigma V^T)\cdot (V\Sigma V^T{)}^{-1}$$

• $(V\Sigma V^T{)}^{-1}=V{\Sigma }^{-1}{V}^T$ since $V$ has orthonormal columns.
• So $M(M^{T}M{)}^{-1/2}=U\Sigma V^T\cdot V{\Sigma }^{-1}{V}^T=U\Sigma {\Sigma }^{-1}{V}^T=U{V}^T$

Similarly, starting with $(M{M}^T{)}^{-1/2}M$

$$M{M}^T=(U\Sigma V^T)(U\Sigma V^T{)}^T=U\Sigma V^{T}V\Sigma U^T=U{\Sigma }^2{U}^T$$

so $(M{M}^T{)}^{1/2}=U\Sigma U^T$ Then

$$(M{M}^T{)}^{-1/2}M=(U\Sigma U^T{)}^{-1}\cdot U\Sigma V^T=U{\Sigma }^{-1}{U}^T\cdot U\Sigma V^T=U{V}^T$$

since $U^{T}U=I_r$. Thus both expressions produce

$$O=U{V}^T$$

which is the orthogonal polar factor.

Summary: so the identity holds because of SVD properties

• Both

$$O=M(M^{T}M{)}^{-1/2}$$

and

$$O=(M{M}^T{)}^{-1/2}M$$

equal the same canonical “orthogonal part” $U{V}^T$ in the SVD $M=U\Sigma V^T$.

• The square root / inverse square root comes from extracting the singular values: $(M^{T}M)$ has eigenvalues ${\sigma }_i^2$, so $(M^{T}M{)}^{-1/2}$ has eigenvalues ${\sigma }_i^{-1}$. Multiplying by that removes the magnitudes (singular values), leaving the pure directions $U{V}^T$.

Intuition: what it means

• Decompose $M$ into “magnitude” + “direction”:
  • $\Sigma$ = magnitudes / singular values (how much scaling along different orthogonal input/output directions).
  • $U$, $V$ capture “directions” (row/column subspaces).
• $O=U{V}^T$ is like the “shape / rotation / directional” part, ignoring scaling. It’s an orthogonal matrix (or semi-orthogonal if $M$ isn’t square or full rank).
• The expressions $M(M^{T}M{)}^{-1/2}$ or $(M{M}^T{)}^{-1/2}M$ are implementations that “divide out” the scaling imposed by $\Sigma$ by applying the inverse of $\Sigma$ in whichever product yields a symmetric positive definite (or semidef.) matrix.
• In the context of Muon Optimizer AdaMuon - Adaptive Muon Optimizer Muon Blog:
  • $M_t$ holds the momentum (matrix).
  • Polar / orthogonal factor $O_t$ = remove scaling / magnitude part of $M_t$, leaving the structured directional information.
  • This helps ensure update steps follow coherent geometry, not dominated by large singular values (which might correspond to disproportionate scaling along some directions).

Why “both” forms are valid / when to use which

• $M(M^{T}M{)}^{-1/2}$ might be more convenient when $m$ is small (so $M^{T}M$ is a small matrix) – computing its inverse square root is cheaper.
• $(M{M}^T{)}^{-1/2}M$ might be better when $n$ is smaller.
• Both give the same $O$ in exact arithmetic; numerical approximations (like Newton–Schulz) approximate one of these forms more directly.

Example: 2x2 Matrix

Let’s work through a concrete example with a $2\times 2$ matrix. Let $M=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]$.

Step 1: Compute $M^{T}M$ and $M{M}^T$

• $M^{T}M=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]=\left[\begin{array}{cc}5& 4\\ 4& 5\end{array}\right]$
• $M{M}^T=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]=\left[\begin{array}{cc}5& 4\\ 4& 5\end{array}\right]$

Step 2: Compute $(M^{T}M{)}^{-1/2}$ and $(M{M}^T{)}^{-1/2}$ First, find the eigenvalues and eigenvectors of $M^{T}M$ (which is the same as for $M{M}^T$ in this symmetric case). The eigenvalues $\lambda$ satisfy $\det (\left[\begin{array}{cc}5-\lambda & 4\\ 4& 5-\lambda \end{array}\right])=(5-\lambda {)}^2-16=0$. So ${\lambda }_1=9$, ${\lambda }_2=1$.

Eigenvector for ${\lambda }_1=9$: $\left[\begin{array}{c}1\\ 1\end{array}\right]$ (normalized: $\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ 1\end{array}\right]$) Eigenvector for ${\lambda }_2=1$: $\left[\begin{array}{c}1\\ -1\end{array}\right]$ (normalized: $\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ -1\end{array}\right]$)

Thus, the eigendecomposition is $M^{T}M=Q\Lambda Q^T$, where $Q=\frac{1}{\sqrt{2}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]$ and $\Lambda =\left[\begin{array}{cc}9& 0\\ 0& 1\end{array}\right]$.

Then $(M^{T}M{)}^{-1/2}=Q{\Lambda }^{-1/2}{Q}^T=\frac{1}{\sqrt{2}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]\left[\begin{array}{cc}1/3& 0\\ 0& 1\end{array}\right]\frac{1}{\sqrt{2}}{\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]}^T$ $=\frac{1}{2}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]\left[\begin{array}{cc}1/3& 0\\ 0& 1\end{array}\right]\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]$ $=\frac{1}{2}\left[\begin{array}{cc}1/3& 1\\ 1/3& -1\end{array}\right]\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]$ $=\frac{1}{2}\left[\begin{array}{cc}1/3+1& 1/3-1\\ 1/3-1& 1/3+1\end{array}\right]=\frac{1}{2}\left[\begin{array}{cc}4/3& -2/3\\ -2/3& 4/3\end{array}\right]=\left[\begin{array}{cc}2/3& -1/3\\ -1/3& 2/3\end{array}\right]$

Since $M{M}^T=M^{T}M$ in this case, $(M{M}^T{)}^{-1/2}$ is the same matrix.

Step 3: Compute $O$ using both forms

First form: $O=(M{M}^T{)}^{-1/2}M=\left[\begin{array}{cc}2/3& -1/3\\ -1/3& 2/3\end{array}\right]\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]=\left[\begin{array}{cc}(4/3-1/3)& (2/3-2/3)\\ (-2/3+2/3)& (-1/3+4/3)\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

Second form: $O=M(M^{T}M{)}^{-1/2}=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]\left[\begin{array}{cc}2/3& -1/3\\ -1/3& 2/3\end{array}\right]=\left[\begin{array}{cc}(4/3-1/3)& (-2/3+2/3)\\ (2/3-2/3)& (-1/3+4/3)\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

Both forms yield the same orthogonal matrix $O=I$, the identity matrix. This makes sense because the original matrix $M$ is symmetric and positive definite, and its polar decomposition is $M=I\cdot H$, where $H=M$ itself. The operations successfully removed the “scaling” (the eigenvalues 3 and 1 of $M$) to leave the pure “rotation” (which is the identity in this case).