Speech & Language Blog

Introduction The sixth post of my probability theory series focuses on the Beta distribution.  Basic Probability Two Random Variables Chain Rule of Probability Theory Probability Distribution1 Probability Distribution2: Normal distribution The beta distribution has the multivariate version called the Dirichlet distribution. The Dirichlet distribution used to be very popular in Bayesian and natural language processing literature before the LLM era. A common one-linear explanation of the beta distribution is "a distribution over a probability". I hope all readers are confused, so was I when I heard this for the first time. Here is an example. I visited paris every year in the past 10 years. That makes my chance of visiting Paris next year 100% according to my travel history.  The beta distribution asks this question: "how confident are we with this 100% chance of me visiting Paris next year"? It is a distribution ("our confidence") over probabilities ("...

Introduction Differentiation calculus is a core part of machine learning optimisation. Differentiation is often taught at school as finding a slope in geometry. While a theoretical concept of differentiation is important, I feel that connection between differentiation at school and differentiation for machine learning optimisation is not linked well. This post aims to bridge the gap between school math and machine learning optimisation, focusing on how the mathematical concept (differentiation) is essential for the practical algorithm (gradient descent) to support machine learning. The post begins with a very basic concept of slope finding. Finding a slope between two points The general formula to find a slope \(a\) of a function \( y = ax \) is: \[ a = \frac{y_2 - y_1}{x_2-x_1} \] The figure below illustrates rise (vertical) and run (horizontal) changes between two points, Point 1 \((x = 0, y=0)\)  and Point 2 \((x=2, y=4)\).  The slope of this function is \( a = \frac{4...

Introduction The fifth post of my probability series is about continuous probability distributions. Basic Probability Two Random Variables Chain Rule of Probability Theory Probability Distribution1 A quick recap: a probability distribution concerns probabilities of  all possible outcomes  of a sample space. The previous post introduced four different discrete probability distributions.  The current post introduces the most popular continuous probability distribution: normal (Gaussian) distribution. The normal distribution has been an essential tool for Automatic Speech Recognition a long time before deep learning. Normal distribution The normal distribution is a bit like the Eiffel Tower. It is a distribution known for its "bell curve". The distribution is pointy in the middle and wider towards the edges of the distribution. The name "Gaussian" distribution is from the mathematician, Gauss . The properties of the normal distribution are as follows: Number of trials:...

Introduction The fourth post of my probability series is about  probability distributions . Basic Probability Two Random Variables Chain Rule of Probability Theory In the previous posts, I discussed probabilities involving a single and multiple random variables. Briefly, a random variable in an experiment or a trial maps a specific outcome from a sample space to a numeric value.  The example of a probabilistic event which I kept using was whether I to go to Paris next year \(X=1\) or not \(X=0\). A probability of a specific event can be expressed like this: \[ P(X=1) = 0.7 \] The generalised probability is then like this: \[ P(X) \] where \(X\) can be anything in a defined sample space. A probability distribution concerns probabilities of all possible outcomes of a sample space. So, \( P(X=1)=0.7 \) is a part of a probability distribution, but the whole representation of the probability distribution is \( P(X) \). More concrete examples are: we think about "going to Paris...

Introduction  This is my third post on the probability theory: Basic Probability Two Random Variables The main focus of the series is on easy introduction to probability theory. A bit of recap : in the last post of Two Random Variables, I discussed probabilities of two random events including joint, conditional and marginal probabilities. That post was about probabilities of two dependent and independent events. The example was whether Emily and I go to Paris next year. If Emily and I did not know each other, Emily visiting Paris next year would have no impact on my chance of going to Paris (two events were independent). On the other hand, if I knew Emily and she text me about her trip to Paris, that could affect my chance of visiting Paris next year (two events were dependent). This post focuses on generalisation of multiple events and random variables. So, I am introducing a new event. An alien is visiting Paris. We need to think about how this event will impact chances of my and...