Orthogonality, Least Squares and Symmetric Matrices