Full Bayesian Parameter Estimation

Introduction

The prevoius post discussed Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) Estimation, focusing on the two properties of parameter estimation methods: whether an estimation method is a point or density estimate and whether an estimation method uses prior knowledge. While MLE does not use prior knowledge, MAP integrates prior knowledge into parameter estimation. Both parameter estimation methods are a point estimate.

The current post focuses on Bayesian parameter estimation. Unlike the previous approaches, the Bayesian estimation is a density estimation method that produces a full distribution of parameter values as its outcome. As the name of the "Bayesian estimate" suggests, the formula is based on the Bayes theorem, which solves a conditional probability \( P(\theta|D) \).

The mathematical definition of full Bayesian parameter estimation is below: \[ P(\theta | D) = \frac{P(D | \theta) P(\theta)}{\int P(D | \theta') P(\theta') d\theta'} \] This means that the posterior \(P(\theta|D)\) is computed by multiplying the likelihood \(P(D|\theta)\) and the prior \(P(\theta)\). The denominator integrates the numerator over all possible parameter values, working as a normalising constant. The denominator can also be written as \(P(D)\), which is the marginal likelihood of data \(D\).

The characteristics of the Bayesian parameter estimation are the followings: 
  • Works well with a small amount of data
  • Exploits prior knowledge
  • Density estimation (modelling uncertainty)
  • Useful when uncertainty is important.
I will split this topic in two posts: when Bayesian estimation is simple and when Bayesian estimation is not simple.

Computing the denominator of the Bayesian estimation is feasible when a conjugate prior exists. It can also be tractable when a parameter is low-dimensional, e.g a single probability parameter.

Unfortunately, the real world problems often involve multiple parameters. Suppose we have a parameter that varies from 0 to 100 with an interval of 1, we need 100 values to compute the denominator. If we have 10 of such parameters, the number of values to compute blows up to \( 100^{10} = 10^{20} \). This is virtually impossible to calculate with our computer as of 2026. This is when Bayesian estimation is not simple. In this case, we need a technique called sampling. Sampling approximates computation of a complicated denominator involving multiple parameters.

In this post, let's take a look at a simple case of Bayesian parameter estimation.

When Bayesian estimation is simple

As discussed above, the Bayesian parameter estimation is simple when a conjugate prior exists.

Conjugate priors

An example of the conjugate prior is the Binomial distribution and the Beta distribution. \[ Posterior (Beta) \propto Likelihood (Binomial) \times Prior (Beta) \] Let's bring back the Pikachu example from the previous post.
An encounter.

The Pikachu encounter rate example is as follows:
  • Prior experience says that 1 out of 10 Pokemon is Pikachu: \(Beta(\alpha=1, \beta=9)\).
  • The new data shows that 1 out of 5 Pokemon is Pikachu: \(\binom{5}{1}\theta^1(1-\theta)^4\).
Here's a refresher of the Beta distribution. \[ P(\theta; \alpha, \beta) = \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{\mathrm{B}(\alpha, \beta)} \] where \( \mathrm{B} \) is the beta function to normalise the function into a valid probability density function: \[ \mathrm{B}(\alpha, \beta) = \int_0^1 \theta^{\alpha-1}(1-\theta)^{\beta-1} d\theta = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)} \] While MAP estimation gives us a single "best" parameter value of the Pikachu encounter rate i.e., the mode of the Beta distribution, Bayesian estimation keeps the entire posterior distribution, demonstrating uncertainty over every possible Pikachu encounter rate.

Now, let's compute the posterior distribution of the Pikachu encounter rate.
\[ \begin{aligned} P(D|\theta)P(\theta) &\propto \theta^1 \times (1-\theta)^4 \times\theta^{1-1} (1-\theta)^{9-1} \\ &\propto \theta^{2-1} \times (1-\theta)^{13-1} \\ &\propto \mathrm{Beta}(\alpha=2, \beta=13) \end{aligned} \] Notice that multiplication of the likelihood by the prior here is only proportional to the posterior. This is because we are not computing the normalising constants of both the Beta prior distribution and the Binomial likelihood coefficient. We don't actually need the normalisation constants because \(P(D)\) does not depend on \(\theta\), and the shape of the posterior distribution would not change because of it. Finally, the posterior distribution obtained is \(\mathrm{Beta}(\alpha=2, \beta=13)\), so we only need to plug in alpha and beta to normalise the posterior Beta distribution.

Non conjugate but tractable

As mentioned in Introduction, computing Bayesian estimation is still tractable with a low-dimensional parameter. Let's replace a Beta distribution with a Normal distribution as a prior. The posterior distribution is no longer the same shape as a prior distribution. Note that the Normal distribution here is truncated to [0, 1] for demonstration.

The posterior distribution we would like to compute in this case is then: \[ P(D|\theta)P(\theta) \propto \theta^k \times (1 - \theta)^{n-k} \times \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(\theta-\mu)^2}{2\sigma^2}\right) \] The posterior distribution is proportional to multiplication of the likelihood and the posterior. The term \( \frac{1}{\sqrt{2 \pi \sigma^2}} \) does not contain any \( \theta \) and it is irrelevant to the shape of the posterior distribution of \( \theta \). Therefore, we can eliminate this term from the equation. \[ P(D|\theta)P(\theta) \propto \theta^k \times (1 - \theta)^{n-k} \times \exp\left(-\frac{(\theta-\mu)^2}{2\sigma^2}\right) \] Let's fill in some Pikachu encounter rates in the equation. 
  • Keep the same values for the Binomial likelihood as the previous example. 1 out of 5 Pokemon is Pikachu \(\binom{5}{1}\theta^1(1-\theta)^4\).
  • Define the Normal distribution as a prior with the mean \( \mu = 0.1 \) and the standard deviation \( \sigma = 0.1 \). 
\[ P(D|\theta)P(\theta) \propto \theta^1 \times (1 - \theta)^{4} \times \exp\left(-\frac{(\theta-0.1)^2}{0.02}\right) \] So, the posterior from the Normal distribution prior and the Binomial likelihood is not a clean solution like \(\mathrm{Beta}(\alpha=2, \beta=13)\). We need to compute the normalisation constant this time too. \[ P(D) = \int_0^1 \theta^1 \times (1 - \theta)^{4} \times \exp\left(-\frac{(\theta-0.1)^2}{0.02} \right) d\theta \] Computing these would be difficult with pen and paper. Instead, we can evaluate the unnormalised posterior on a grid of \( \theta \) values e.g., steps of 0.001, and approximate the normalisation constant by summation. 

The figure below shows the Binomial likelihood, the Normal distribution prior and the posterior distribution of the parameter \( \theta \). The highest probability density (=Pikachu encounter rate) appears at \( \theta = 0.131 \). 
Posterior distribution = Binomial likelihood x Truncated-Normal distribution prior.

Summary

This post has covered full Bayesian parameter estimation. While maximum likelihood estimation (MLE) and maximum a posteriori estimation (MAP) estimates a single point, Bayesian parameter estimation produces the entire posterior distribution, quantifying the uncertainty of the parameters. Computing a posterior distribution can be complicated due to computation of the denominator or normalising constant of the distribution.

The denominator \( P(D) \) or the marginal likelihood of data is feasible to calculate in two cases. The first case is when a prior distribution is conjugate to a likelihood. In this case, a posterior distribution belongs to the same family as the prior distribution, and the normalising constant is easy to calculate. The second case is when a parameter dimension is not too large. A grid approximation can be applied to computation of the normalising constant, and the whole computation becomes tractable.

The next post will cover when Bayesian estimation is not simple. When a prior distribution is not a conjugate prior and when a dimension of the parameter is large, the technique called sampling is used to approximate a posterior distribution.

Comments

Post a Comment

Popular posts from this blog

Digital Signal Processing Basics

Image
Acoustic Signal I had a hard time understanding basics of digital signal processing. I think that the reason was the fact that sounds were not visible. There is a way to "see" the sound, though. This YouTube video for example demonstrates that special cameras make the sound visible.   The sound is the virbation of particles in the air. Something invisible surrounding us rapidly moves back and forth. And our ears can hear this movement. I am writing this post to cover very basics of digital signal processing that I was very slow to understand. I will write concepts of Analogue-to-Digital conversion, sampling and Nyquist frequency. Analogue vs Digital Apart from the sound not usually visible to us, I think that I initially mixed up the concept of the digital signal with the analogue signal. I was still new to the idea that computers represent everything in discrete numbers. Two important things: An anal...
Read more

Sound frequency

Image
Introduction The signal frequency is the "pitch" of the sound. Some facts about sound frequencies you might encounter in a pub quiz... Typically, male voices range from 85 to 180 Hz and female voices from 165 to 255 Hz .  Humans can easily hear the sound frequency up to 8,000 Hz and lose abilities to hear sounds beyond that frequency through age. The music note C is 261.63 Hz and E 329.63 Hz. The Python notebook is a convenient playground to generate sounds of those frequencies and listen to the sounds. https://github.com/yasumori/blog/blob/main/2025/2025_12_21_signal2.ipynb . An example code snipet to generate a 2,000 Hz sound is also below: import numpy as np from IPython import display def gen_audio(frequency, duration, sample_rate): t = np.linspace(0, duration, duration * sample_rate) return np.sin(2 * np.pi * frequency * t) hz_2000 = gen_audio(2000, 3, 44100) disp...
Read more

Setup git pre-push hook

Motivation I often forget to run tests before pushing my commits. This is a bad practice. Someone might use my broken code in a collaborative environment. I was looking for a way to automatically run tests before creating a commit or pushing commits. The solution is Git Hooks:  Customizing Git Hooks . Git Hooks Many hooks can be setup in a Git repository. For example, Git Hooks can run before creating a commit ( pre-commit ), after creating a commit ( post-commit ) and before pushing commits to a remote branch ( pre-push ). All of these are client-side hooks and client-side hooks are not shared across local repositories. The other type of hooks are server-side hooks  that run after commits are pushed to a remote server and policy is imposed on all users of the repository. I have setup  pre-push  in a Git repository to avoid forgetting to run tests before sending updates to the remote server. Setup pre-push The way to setup  pr...
Read more