Vector spaces, norms, and orthogonality

Reading: Golub & Van Loan §2.1–2.3, pp. 64–76.

By the end of this week you should be able to

• Identify the four fundamental subspaces of a matrix and their dimensions.
• Compute and compare vector norms, induced matrix norms, and the Frobenius norm.
• Use submultiplicativity and norm equivalence in a bound.

Algorithms introduced

• Orthogonal projection onto a subspace
• Gram matrices and orthonormal bases

Where this shows up in AI

Matrix norms give the Lipschitz constants of network layers, which is the basis of spectral normalization and most generalization bounds.

Materials