Inner Products

Introduction to Inner Products

An inner product is a mathematical concept that introduces the notions of length, distance, and angle into vector spaces. Inner products allow us to determine whether two vectors are orthogonal (perpendicular) to each other and generalize the concept of the dot product from ${\mathbb{R}}^n$ to more abstract vector spaces. With inner products, we can formally define notions like vector length (or magnitude) and the angle between vectors, which are fundamental in fields such as machine learning and physics.

The Dot Product in ${\mathbb{R}}^n$

The dot product (also known as the scalar product) is a specific type of inner product in the Euclidean space ${\mathbb{R}}^n$. For two vectors $x,y\in {\mathbb{R}}^n$, the dot product is defined as:

$$x^{\top }y=\sum _{i=1}^n{x}_i{y}_i$$

where $x=(x_1,x_2,\dots ,x_n)$ and $y=(y_1,y_2,\dots ,y_n)$. The dot product yields a scalar value and serves as a measure of the extent to which two vectors “point in the same direction.” In this book, we refer to this specific inner product as the dot product, although inner products in general can be more versatile.

General Definition of Inner Products

An inner product is a more generalized concept than the dot product, defined on any vector space, not just ${\mathbb{R}}^n$.

Properties of an Inner Product

For a function $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{R}$ to qualify as an inner product on a vector space $V$, it must satisfy the following properties:

1. Linearity in Each Argument (Bilinearity):

   • For all vectors $x,y,z\in V$ and scalars $\lambda ,\psi \in \mathbb{R}$, we have: $$⟨\lambda x+\psi y,z⟩=\lambda ⟨x,z⟩+\psi ⟨y,z⟩$$ $$⟨x,\lambda y+\psi z⟩=\lambda ⟨x,y⟩+\psi ⟨x,z⟩$$

   These conditions ensure that the inner product is linear in both arguments, meaning it respects vector addition and scalar multiplication.

2. Symmetry:

   • For all vectors $x,y\in V$, the inner product satisfies: $$⟨x,y⟩=⟨y,x⟩$$

   This property ensures that the order of arguments does not affect the result, which aligns with our intuitive sense of symmetry.

3. Positive Definiteness:

   • For all vectors $x\in V$, we have: $$⟨x,x⟩\ge 0$$ with equality if and only if $x=0$. This property implies that the inner product of a vector with itself is always non-negative, capturing the idea that the “length” or “magnitude” of a vector is zero only if the vector itself is zero.

A positive definite, symmetric bilinear mapping $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{R}$ defines an inner product space. If the vector space $V$ is equipped with an inner product, we call it an inner product space, denoted $(V,⟨\cdot ,\cdot ⟩)$.

Examples of Inner Products

1. Euclidean Inner Product (Dot Product)

In ${\mathbb{R}}^n$, the standard inner product is the dot product:

$$⟨x,y⟩=x^{\top }y=\sum _{i=1}^n{x}_i{y}_i$$

This dot product satisfies all properties of an inner product, making ${\mathbb{R}}^n$ with the dot product an example of an inner product space known as the Euclidean space.

2. A Non-Standard Inner Product in ${\mathbb{R}}^2$

Consider the vector space $V={\mathbb{R}}^2$ with the inner product defined by:

$$⟨x,y⟩:=x_1{y}_1-(x_1{y}_2+x_2{y}_1)+2x_2{y}_2$$

This example demonstrates that we can define inner products differently than the dot product, provided they satisfy the properties of bilinearity, symmetry, and positive definiteness. This inner product adheres to these properties, although it does not resemble the typical dot product.

Symmetric, Positive Definite Matrices and Inner Products

Symmetric Positive Definite Matrices (SPD Matrices) In machine learning, symmetric, positive definite matrices are essential, particularly in contexts like matrix decompositions and kernel methods. Such matrices help define inner products, distances, and even functions in higher-dimensional spaces.

Symmetric, Positive Definite Matrices

A matrix $A\in {\mathbb{R}}^{n\times n}$ is symmetric and positive definite if it satisfies two key conditions:

1. Symmetry: $A=A^{\top }$.
2. Positive Definiteness: For all non-zero vectors $x\in {\mathbb{R}}^n$, $$x^{\top }Ax>0$$

If $A$ satisfies $x^{\top }Ax\ge 0$ (instead of $>0$) for all $x\ne 0$, then $A$ is called positive semidefinite.

Example of Symmetric, Positive Definite Matrices

Consider the following matrices:

$$A_1=\left(\begin{array}{cc}9& 6\\ 6& 5\end{array}\right),A_2=\left(\begin{array}{cc}9& 6\\ 6& 3\end{array}\right)$$

• $A_1$ is positive definite, as for all $x\in {\mathbb{R}}^2\setminus \{0\}$, $x^{\top }{A}_{1}x>0$.
• $A_2$ is symmetric but not positive definite, as there exists $x\in {\mathbb{R}}^2$ such that $x^{\top }{A}_{2}x<0$.

Inner Products and Matrix Representation

In an $n$-dimensional vector space $V$ with an ordered basis $B=(b_1,b_2,\dots ,b_n)$, any vectors $x,y\in V$ can be expressed as linear combinations of the basis vectors:

$$x=\sum _{i=1}^n{\psi }_i{b}_i,y=\sum _{j=1}^n{\lambda }_j{b}_j$$

where ${\psi }_i,{\lambda }_j\in \mathbb{R}$.

The inner product $⟨x,y⟩$ with respect to the basis $B$ can be expressed as: $⟨\mathbf{x},\mathbf{y}⟩=⟨{\sum }_{i=1}^n{\psi }_i{\mathbf{b}}_i,{\sum }_{j=1}^n{\lambda }_j{\mathbf{b}}_j⟩={\sum }_{i=1}^n{\sum }_{j=1}^n{\psi }_i⟨{\mathbf{b}}_i,{\mathbf{b}}_j⟩{\lambda }_j={\hat{\mathbf{x}}}^{\top }\mathbf{A}\hat{\mathbf{y}}$ where $A_{ij}:=⟨b_i,b_j⟩$ and $\hat{x},\hat{y}$ are the coordinate representations of $x$ and $y$ with respect to $B$. Here, $A$ is symmetric, positive definite, making it a Gram matrix associated with the inner product in $V$.

Theorem: Characterization of Inner Products with Positive Definite Matrices

For a real, finite-dimensional vector space $V$ with an ordered basis $B$, a bilinear mapping $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{R}$ is an inner product if and only if there exists a symmetric, positive definite matrix $A\in {\mathbb{R}}^{n\times n}$ such that:

$$⟨x,y⟩={\hat{x}}^{\top }A\hat{y}$$

for coordinate representations $\hat{x},\hat{y}$ of $x$ and $y$ in $B$.

Properties of Symmetric, Positive Definite Matrices

1. Non-zero Kernel: The kernel (null space) of $A$ contains only the zero vector since $x^{\top }Ax>0$ for all $x\ne 0$.
2. Positive Diagonal Elements: Each diagonal element $a_{ii}$ of $A$ is positive because $A$ is positive definite.
3. Invertibility: Positive definite matrices are invertible because their eigenvalues are all positive.