Backpropagation by hand
Introduction In the previous posts, we covered why taking the derivative is linked to optimisation of a machine learning model and how gradient descent is performed with a simple model . This post focuses on the core algorithm used for training neural networks, including large language models: backpropagation. It is simple to take the derivative of the loss with respect to the weight, when a model consists of a single parameter e.g., \( \hat{y} = wx \). What if the model has multiple layers of weights and they are dependent on each other? \[ h = w_1 x \] \[ \hat{y} = w_2 h \] With this model, we need to compute an internal representation \( h \), and then produce the model prediction \( \hat{y} \) with multiplication of the hidden \( h \) by the second weight \( w_2 \). Visualisation of this model is provided in the figure below. How do we compute the derivative of the loss with respect to \( w_1 \), which sits in the first layer of th...