Learn Linear Algebra

Theorem

Given an $m \times n$ matrix $A$ with linearly independent columns, let $A = QR$ be a $QR$ factorization of $A$. Then, for each $\vec{b} \in \mathbb{R}^m$, the equation $A \vec{x} = \vec{b}$ has a unique least-squares solution given by:

$\hat{\vec{x}} = R^{-1} Q^T \vec{b}.$

Theorem

The set of least-squares solutions of $A\vec{x} = \vec{b}$ coincides with the nonempty set of solutions of the normal equations $A^T A \vec{x} = A^T \vec{b}$.

Theorem

The matrix $A^T A$ is invertible if and only if the columns of $A$ are linearly independent. In this case, the equation $A\vec{x} = \vec{b}$ has only one least-squares solution $\hat{\vec{x}}$, and it is given by:

$$\hat{\vec{x}} = (A^T A)^{-1} A^T \vec{b}.$$

Theorem

A vector $\hat{\vec{x}}$ is a least-squares solution of $A \vec{x} = \vec{b}$ if and only if $A \hat{\vec{x}} = \text{proj}_{\text{Col}(A)}(\vec{b})$.

Theorem

Let $A$ be an $m \times n$ matrix with orthogonal columns $\vec{a}_1, \dots, \vec{a}_n$, and let $\vec{b} \in \mathbb{R}^m$. The least-squares solution of $A \vec{x} = \vec{b}$ is $\hat{\vec{x}}$, where the $i$-th entry in $\hat{\vec{x}}$ is given by:

$\hat{x}_i = \frac{\vec{b} \cdot \vec{a}_i}{\vec{a}_i \cdot \vec{a}_i}.$