Matrix calculus, backpropagation, and synthesis

Reading: Golub & Van Loan §12.3, pp. 707–719. Supplemented by Deisenroth, Faisal & Ong, Mathematics for Machine Learning, Chapter 5 — Golub & Van Loan has no matrix-calculus chapter.

By the end of this week you should be able to

• Use Kronecker products and the vec operator to rewrite matrix equations as linear systems.
• Differentiate matrix-valued expressions using differentials rather than index gymnastics.
• Explain backpropagation as a sequence of Jacobian-transpose products.

Algorithms introduced

• Kronecker product and vec identities
• Reverse-mode automatic differentiation

Where this shows up in AI

This is where the course meets deep learning head on. Backpropagation is not a special trick: it is reverse-mode differentiation, and seeing it that way makes second-order and natural-gradient methods comprehensible.

Materials