Rank deficiency, regularization, and total least squares

Reading: Golub & Van Loan §5.5, 6.1, 6.3, pp. 288–327.

By the end of this week you should be able to

• Characterise the minimum-norm solution of a rank-deficient problem via the pseudoinverse.
• Formulate Tikhonov regularization and interpret the regularization parameter.
• Set up a total least squares problem and say when it is the right model.

Algorithms introduced

• Truncated SVD and the Moore-Penrose pseudoinverse
• Tikhonov regularization
• Total least squares via the SVD

Where this shows up in AI

Ridge regression is Tikhonov regularization, and weight decay is the same idea once more. Total least squares is the right model whenever the inputs are noisy too.

Materials