Speech & Language Blog

Introduction Deep learning is black magic. We throw data into PyTorch and we get a model that seemingly understands language, speech, and vision. However, one fundamental technique for deep learning has remained the same. That is gradient descent. The modern optimisers such as Adam and AdamW are refinements of gradient descent. So, I believe that gradient descent is still a core technique worth learning. I've written a post on the relationship between differentiation and optimisation ( https://yasufumimoriya.blogspot.com/2026/04/differentiation-and-optimisation.html ). In this post, there was a single parameter to optimise or to find its slope. Gradient descent is essentially finding slopes of multiple parameters at once. This post focuses on how computation of gradient descent proceeds to update parameters, taking a function with two parameters as an example. Refresher: Differentiation and Optimisation This section is a refresher of the relations...

Bayesian parameter estimation The previous post  introduced the basic concept of Bayesian parameter estimation. If you haven't read it, please check that post before this one. When Bayesian estimation is not simple There are two cases when Bayesian estimation is tractable. The simplest is when a prior distribution is a conjugate prior. Computing the normalising constant of a single or low-dimensional parameter of \( \theta \) is also tractable with a grid approximation. In this post, I will stick to the Pikachu encounter rate example like in the previous post too. Imagine we have analysed 10 routes, each with its own Pikachu encounter rate: \[ \boldsymbol{\theta} = [\theta_1, \dots, \theta_{10}], \quad \theta_k \in [0, 1] \] On each route \( k \), we observed \( n_k \) Pikachu out of \( N_k \) Pokemon, so the likelihood is a product of Binomials, \( P(D|\boldsymbol{\theta}) \propto \prod_{k=1}^{10} \theta_...

Introduction The prevoius post discussed Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) Estimation , focusing on the two properties of parameter estimation methods: whether an estimation method is a point or density estimate and whether an estimation method uses prior knowledge. While MLE does not use prior knowledge, MAP integrates prior knowledge into parameter estimation. Both parameter estimation methods are a point estimate. The current post focuses on Bayesian parameter estimation. Unlike the previous approaches, the Bayesian estimation is a density estimation method that produces a full distribution of parameter values as its outcome.  As the name of the "Bayesian estimate" suggests, the formula is based on the Bayes theorem, which solves a conditional probability \( P(\theta|D) \). The mathematical definition of full Bayesian parameter estimation is below:  \[ P(\theta | D) = \frac{P(D | \theta)...

Introduction I have made several posts about probabilities and distributions.  Basic Probability Two Random Variables Chain Rule of Probability Theory Probability Distribution1 Probability Distribution2: Normal distribution Probability Distribution3: Beta and Dirichlet distribution I used arbitrary probabilities and parameters in the series. Something like, "the chance of me going to Paris next year is 70% ( Two Random Variables )", "the mean wait time to enter the Eiffel tower is 60 minutes and variance is 144 ( Probability Distribution2: Normal distribution ) and "the rate of successful task completion by your new colleague is 66% ( Probability Distribution3: Beta and Dirichlet distribution )". All those numbers are arbitrary and random. The goal of...

Introduction The sixth post of my probability theory series focuses on the Beta distribution.  Basic Probability Two Random Variables Chain Rule of Probability Theory Probability Distribution1 Probability Distribution2: Normal distribution The beta distribution has the multivariate version called the Dirichlet distribution. The Dirichlet distribution used to be very popular in Bayesian and natural language processing literature before the LLM era. A common one-linear explanation of the beta distribution is "a distribution over a probability". I hope all readers are confused, so was I when I heard this for the first time. Here is an example. I visited paris every year in the past 10 years. That makes my chance of visiting Paris next year 100% according to my travel history.  The beta distribution asks this question: "how confident are we with t...

Introduction Differentiation calculus is a core part of machine learning optimisation. Differentiation is often taught at school as finding a slope in geometry. While a theoretical concept of differentiation is important, I feel that connection between differentiation at school and differentiation for machine learning optimisation is not linked well. This post aims to bridge the gap between school math and machine learning optimisation, focusing on how the mathematical concept (differentiation) is essential for the practical algorithm (gradient descent) to support machine learning. The post begins with a very basic concept of slope finding. Finding a slope between two points The general formula to find a slope \(a\) of a function \( y = ax \) is: \[ a = \frac{y_2 - y_1}{x_2-x_1} \] The figure below illustrates rise (vertical) and run (horizontal) changes between two points, Point 1 \((x = 0, y=0)\)  and Point 2 \((x=2, y=4)\).  ...