Speech & Language Blog

Bayesian parameter estimation The previous post  introduced the basic concept of Bayesian parameter estimation. If you haven't read it, please check that post before this one. When Bayesian estimation is not simple There are two cases when Bayesian estimation is tractable. The simplest is when a prior distribution is a conjugate prior. Computing the normalising constant of a single or low-dimensional parameter of \( \theta \) is also tractable with a grid approximation. In this post, I will stick to the Pikachu encounter rate example like in the previous post too. Imagine we have analysed 10 routes, each with its own Pikachu encounter rate: \[ \boldsymbol{\theta} = [\theta_1, \dots, \theta_{10}], \quad \theta_k \in [0, 1] \] On each route \( k \), we observed \( n_k \) Pikachu out of \( N_k \) Pokemon, so the likelihood is a product of Binomials, \( P(D|\boldsymbol{\theta}) \propto \prod_{k=1}^{10} \theta_...

Introduction The prevoius post discussed Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) Estimation , focusing on the two properties of parameter estimation methods: whether an estimation method is a point or density estimate and whether an estimation method uses prior knowledge. While MLE does not use prior knowledge, MAP integrates prior knowledge into parameter estimation. Both parameter estimation methods are a point estimate. The current post focuses on Bayesian parameter estimation. Unlike the previous approaches, the Bayesian estimation is a density estimation method that produces a full distribution of parameter values as its outcome.  As the name of the "Bayesian estimate" suggests, the formula is based on the Bayes theorem, which solves a conditional probability \( P(\theta|D) \). The mathematical definition of full Bayesian parameter estimation is below:  \[ P(\theta | D) = \frac{P(D | \theta)...