Likelihood Functions for Discrete Probability Distributions

Suppose we observe data from a discrete distribution whose parameter $\theta$ is unknown, and we want to ask which value of $\theta$ makes the observed data most probable. The likelihood function answers this by treating the data as fixed and the parameter as the variable.

If $X$ has PMF $p(x;\theta)$ and we observe a single value $X=x$, the likelihood function is

$$L(\theta) = p(x;\theta).$$

For an iid sample $x_1, x_2, \ldots, x_n$, the likelihood is the product of the individual probabilities:

$$L(\theta) = \prod_{i=1}^{n} p(x_i;\theta).$$

Notice the change in perspective. The PMF $p(x;\theta)$ is a function of $x$ with $\theta$ held fixed (it is a probability). The likelihood $L(\theta)$ is a function of $\theta$ with the data held fixed; it is not a probability distribution over $\theta$.

Quick illustration. If $X \sim \text{Bernoulli}(p)$ and we observe $X=1$, then $L(p) = p$. If instead we observe $X=0$, then $L(p) = 1-p$.