Learn Linear Algebra

Theorem

Let u⃗,v⃗, and w⃗ \vec{u}, \vec{v}, \text{ and } \vec{w} be vectors in Rn \mathbb{R}^n , and let c c be a scalar. Then:

a. u⃗⋅v⃗=v⃗⋅u⃗ \vec{u} \cdot \vec{v} = \vec{v} \cdot \vec{u}
b. (u⃗+v⃗)⋅w⃗=u⃗⋅w⃗+v⃗⋅w⃗ (\vec{u} + \vec{v}) \cdot \vec{w} = \vec{u} \cdot \vec{w} + \vec{v} \cdot \vec{w}
c. (cu⃗)⋅v⃗=c(u⃗⋅v⃗)=u⃗⋅(cv⃗) (c\vec{u}) \cdot \vec{v} = c(\vec{u} \cdot \vec{v}) = \vec{u} \cdot (c\vec{v})
d. u⃗⋅u⃗≥0 \vec{u} \cdot \vec{u} \geq 0 , and u⃗⋅u⃗=0 \vec{u} \cdot \vec{u} = 0 if and only if u⃗=0⃗ \vec{u} = \vec{0} .

Theorem

Two vectors u⃗ \vec{u} and v⃗ \vec{v} are orthogonal if and only if ∥u⃗+v⃗∥2=∥u⃗∥2+∥v⃗∥2 \|\vec{u} + \vec{v}\|^2 = \|\vec{u}\|^2 + \|\vec{v}\|^2 .

Theorem

1. A vector x⃗ \vec{x} is in W⊥ W^\perp if and only if x⃗ \vec{x} is orthogonal to every vector in a set that spans W W .

2. W⊥ W^\perp is a subspace of Rn \mathbb{R}^n .

Theorem

Let A A be an m×n m \times n matrix. The orthogonal complement of the row space of A A is the nullspace of A A , and the orthogonal complement of the column space of A A is the nullspace of AT A^T :

(Row A)⊥=Nul Aand(Col A)⊥=Nul AT (\text{Row } A)^\perp = \text{Nul } A \quad \text{and} \quad (\text{Col } A)^\perp = \text{Nul } A^T

Theorem

u⃗⋅v⃗=∥u⃗∥∥v⃗∥cos⁡θ \vec{u} \cdot \vec{v} = \|\vec{u}\| \|\vec{v}\| \cos \theta

Theorem

If S={u⃗1,…,u⃗p} S = \{ \vec{u}_1, \dots, \vec{u}_p \} is an orthogonal set of nonzero vectors in Rn \mathbb{R}^n , then S S is linearly independent and hence is a basis for the subspace spanned by S S .

Theorem

Let {u⃗1,…,u⃗p} \{ \vec{u}_1, \dots, \vec{u}_p \} be an orthogonal basis for a subspace W W of Rn \mathbb{R}^n . For each y⃗∈W \vec{y} \in W , the weights in the linear combination y⃗=c1u⃗1+⋯+cpu⃗p \vec{y} = c_1 \vec{u}_1 + \cdots + c_p \vec{u}_p are given by cj=y⃗⋅u⃗ju⃗j⋅u⃗j,(j=1,…,p). c_j = \frac{\vec{y} \cdot \vec{u}_j}{\vec{u}_j \cdot \vec{u}_j}, \quad (j = 1, \dots, p).

Theorem

An m×n m \times n matrix U U has orthonormal columns if and only if UTU=I U^T U = I .

Theorem

Let U U be an m×n m \times n matrix with orthonormal columns, and let x⃗ \vec{x} and y⃗ \vec{y} be in Rn \mathbb{R}^n . Then:

a. ∥Ux⃗∥=∥x⃗∥ \|U\vec{x}\| = \|\vec{x}\|
b. (Ux⃗)⋅(Uy⃗)=x⃗⋅y⃗ (U\vec{x}) \cdot (U\vec{y}) = \vec{x} \cdot \vec{y}
c. (Ux⃗)⋅(Uy⃗)=0 (U\vec{x}) \cdot (U\vec{y}) = 0 if and only if x⃗⋅y⃗=0 \vec{x} \cdot \vec{y} = 0 .

The Orthogonal Decomposition Theorem

Let W W be a subspace of Rn \mathbb{R}^n . Then each y⃗∈Rn \vec{y} \in \mathbb{R}^n can be written uniquely in the form y⃗=y⃗^+z⃗ \vec{y} = \hat{\vec{y}} + \vec{z} where y⃗^ \hat{\vec{y}} is in W W and z⃗ \vec{z} is in W⊥ W^\perp . In fact, if {u⃗1,…,u⃗p} \{ \vec{u}_1, \dots, \vec{u}_p \} is any orthogonal basis of W W , then y⃗^=y⃗⋅u⃗1u⃗1⋅u⃗1u⃗1+⋯+y⃗⋅u⃗pu⃗p⋅u⃗pu⃗p \hat{\vec{y}} = \frac{\vec{y} \cdot \vec{u}_1}{\vec{u}_1 \cdot \vec{u}_1} \vec{u}_1 + \cdots + \frac{\vec{y} \cdot \vec{u}_p}{\vec{u}_p \cdot \vec{u}_p} \vec{u}_p and z⃗=y⃗−y⃗^. \vec{z} = \vec{y} - \hat{\vec{y}}.

Theorem

If y⃗ \vec{y} is in W=Span{u⃗1,…,u⃗p} W = \text{Span} \{ \vec{u}_1, \dots, \vec{u}_p \} , then projWy⃗=y⃗ \text{proj}_W \vec{y} = \vec{y} .

The Best Approximation Theorem

Let W W be a subspace of Rn \mathbb{R}^n , y⃗ \vec{y} any vector in Rn \mathbb{R}^n , and y⃗^ \hat{\vec{y}} the orthogonal projection of y⃗ \vec{y} onto W W . Then y⃗^ \hat{\vec{y}} is the closest point in W W to y⃗ \vec{y} , in the sense that ∥y⃗−y⃗^∥<∥y⃗−v⃗∥ \| \vec{y} - \hat{\vec{y}} \| < \| \vec{y} - \vec{v} \| for all v⃗∈W \vec{v} \in W distinct from y⃗^ \hat{\vec{y}} .

Theorem

If {u⃗1,…,u⃗p} \{ \vec{u}_1, \dots, \vec{u}_p \} is an orthonormal basis for a subspace W W of Rn \mathbb{R}^n , then projWy⃗=(y⃗⋅u⃗1)u⃗1+(y⃗⋅u⃗2)u⃗2+⋯+(y⃗⋅u⃗p)u⃗p. \text{proj}_W \vec{y} = (\vec{y} \cdot \vec{u}_1)\vec{u}_1 + (\vec{y} \cdot \vec{u}_2)\vec{u}_2 + \cdots + (\vec{y} \cdot \vec{u}_p)\vec{u}_p. If U=[u⃗1 u⃗2 ⋯ u⃗p] U = [ \vec{u}_1 \ \vec{u}_2 \ \cdots \ \vec{u}_p ] , then projWy⃗=UUTy⃗,for all y⃗∈Rn. \text{proj}_W \vec{y} = UU^T\vec{y}, \quad \text{for all } \vec{y} \in \mathbb{R}^n.