The most important discrete distributions

Convention: discrete probability spaces¶

In this chapter $\Omega$ is always at most countable, and usually even finite. Then the power set $\mathcal{P}(\Omega)$ is a σ-algebra, so every subset of $\Omega$ is an event, and the probability measure $P$ is completely determined by the values

on the individual outcomes – because every event $A \subseteq \Omega$ is the (at most countable) disjoint union of its elements, so that $P(A) = \sum_{\omega \in A} p(\omega)$. This function $p$ is the probability mass function (pmf); discrete density is another name in common use. Everything we want to know about a discrete distribution can be read off it:

Discrete distributions in SciPy¶

scipy.stats provides objects for discrete distributions just as it does for continuous ones. The main difference: where a continuous distribution has a density pdf, a discrete distribution has a pmf, and the method is accordingly called pmf.

The methods we need over and over again are the same for every distribution. If X is a discrete random variable from scipy.stats, then:

The sf stands for survival function.

Discrete uniform distribution¶

• Parameter $n$, notation $\operatorname{Uni}\{1,\dots,n\}$

• pmf $p(k) = \dfrac{1}{n}$ for $k \in \{1,\dots,n\}$

• Expected value $\dfrac{n+1}{2}$

• Variance $\dfrac{n^2-1}{12}$

• Entropy $\log_2 n$ – the maximum among all distributions on a set with $n$ elements (see Expected value and variance, section Entropy)

• Example applications:

  • The fair die: $n=6$, so $\mathbb{E}(X) = 3.5$ and $\mathbb{V}(X) = \frac{35}{12} \approx 2.92$.

  • An idealised random number generator that returns integers in a fixed range.

• In scipy.stats: st.randint(1, n+1) – careful: as with range, the upper bound is exclusive.

We already know the expected value. For the variance we use $\mathbb{V}(X) = \mathbb{E}(X^2) - (\mathbb{E}X)^2$. With the summation formula $\sum_{i=1}^n i^2 = \frac{n(n+1)(2n+1)}{6}$ we get

and therefore

Categorical distribution¶

• Parameter a probability vector $p = (p_1,\dots,p_n) \in \Delta_n$, that is $p_i \geq 0$ and $\sum_i p_i = 1$

• pmf $p(\omega_i) = p_i$ on a set $\Omega = \{\omega_1,\dots,\omega_n\}$ with $n$ elements

• Expected value (if we identify $\omega_i$ with the number $i$) $\sum_{i=1}^n i\,p_i$ – which in general does not simplify any further

• Variance $\sum_{i=1}^n i^2 p_i - \left(\sum_{i=1}^n i\,p_i\right)^2$

• Entropy $-\sum_i p_i \log_2 p_i$

• Example applications:

  • An urn with $m$ balls in $n$ colours, $n_i$ of them of colour $i$; drawing with replacement gives $p_i = \frac{n_i}{m}$.

  • Any survey with a fixed list of mutually exclusive answers.

• In scipy.stats: no object of its own, but st.rv_discrete(values=(values, p)) builds one; if all you want is to draw samples, rng.choice(values, p=p) from the chapter Numpy is enough.

The parameter space is exactly the simplex $\Delta_n$ we met in the chapter Random variables and distributions: the uniform distribution sits at its centre, the Dirac distributions at its vertices. The categorical distribution is therefore the most general distribution on a finite set – every distribution that follows in this chapter is either a special case of it or built from it. The four jumper colours from that chapter are a categorical distribution too, with $n=4$.

Bernoulli distribution¶

• Parameter $p \in [0,1]$, notation $\operatorname{Ber}(p)$

• pmf $p(1) = p$ and $p(0) = 1-p$ on $\Omega = \{0,1\}$

• Expected value $p$

• Variance $p(1-p)$

• Entropy $h(p) := -p\log_2 p - (1-p)\log_2(1-p)$, the binary entropy function

• Example applications:

  • A single coin toss, unfair coins included.

  • Any yes/no feature: a component is defective or it is not, an email is spam or it is not.

• In scipy.stats: st.bernoulli(p)

The Bernoulli distribution is the special case $n=2$ of the categorical distribution, and the simplest nontrivial case there is – it has one parameter. Expected value and variance are quick to compute: $\mathbb{E}(X) = 0\cdot(1-p) + 1\cdot p = p$, and since $X^2 = X$ (because $0^2=0$ and $1^2=1$), we also have $\mathbb{E}(X^2) = p$ and therefore

The variance is maximal at $p = \frac12$ and vanishes for $p \in \{0,1\}$ – there the outcome is no longer random at all.

In the chapter Random variables and distributions we already estimated a value $\hat\pi$ from data without giving it a name: it was the parameter of a Bernoulli distribution.

The binary entropy function¶

The entropy of a Bernoulli distribution, as a function of $p$, deserves a look of its own. Let us differentiate it. With $\log_2 x = \frac{\ln x}{\ln 2}$ we have

so the two constants cancel and what remains is

The only zero is at $p = 1-p$, that is, at $p = \frac12$: the fair coin toss has the maximal entropy, namely $h(\frac12) = 1$ bit. That fits: the uniform distribution is the maximum entropy distribution.

Binomial distribution¶

• Parameters $n \in \mathbb{N}$ and $p \in [0,1]$, notation $\operatorname{Bin}(n,p)$

• pmf $p(k) = \dbinom{n}{k}p^k(1-p)^{n-k}$ for $k \in \{0,1,\dots,n\}$

• Expected value $np$

• Variance $np(1-p)$

• Entropy: no convenient closed expression; it is smaller than $\log_2(n+1)$, since the maximum would be the uniform distribution on $\{0,\dots,n\}$

• Example applications:

  • Tossing the same coin $n$ times and counting the heads.

  • Drawing a sample of $n$ components and counting the defective ones.

  • An urn with two colours, $N$ balls of which $n_1$ are red, $p = \frac{n_1}{N}$, drawing $n$ times with replacement.

• In scipy.stats: st.binom(n, p)

The binomial distribution arises from the Bernoulli distribution by repetition. If $X_1,\dots,X_n$ are independent random variables with $X_i \sim \operatorname{Ber}(p)$, then

counts the number of successes, and $X \sim \operatorname{Bin}(n,p)$. So it has two parameters.

Why the pmf looks the way it does: by independence, any one fixed pattern of $k$ successes and $n-k$ failures – say $1,1,0,1,0,\dots$ – has probability $p^k(1-p)^{n-k}$, whichever positions the successes occupy. And there are exactly $\binom{n}{k}$ such patterns, one for each subset of $\{1,\dots,n\}$ with $k$ elements. Summing over them gives the formula.

Expected value and variance now come for free. The expected value follows from linearity:

For the variance we need the independence – this is exactly the rule of calculation from the section Rules of calculation for the variance:

Without independence this would be false. Here you can see clearly what that rule is good for.

Which $p$ fits the data best?¶

So far we have treated $p$ as given and asked for the probability of $k$ successes. Now we turn the question around: we have found $k = 3$ defective components in a sample of $n = 10$ and want to know which $p$ fits this observation.

For every $p$ the pmf gives us a probability for exactly this observation:

Let us compare two candidates, $p = 0.2$ and $p = 0.5$. The binomial coefficient is the same in both cases and cancels, so it is enough to compare $p^3(1-p)^7$:

So our observation is about 1.7 times as likely under $p = 0.2$ as under $p = 0.5$. In this sense $p = 0.2$ fits better.

The natural next question is which $p$ fits best of all. We are looking for the maximum of $p \mapsto p^k(1-p)^{n-k}$, so we differentiate:

For $0 < p < 1$ the leading factor is nonzero, so what remains is the condition $k(1-p) = (n-k)p$. Solving it: $k - kp = np - kp$, so $k = np$ and therefore

For our observation that is $p = \frac{3}{10} = 0.3$ – simply the proportion of successes in the sample. That is exactly what you would have guessed without any calculation; what is remarkable is that here it drops out of a calculation, from a clearly stated criterion: choose the parameter that makes the observation you actually made as likely as possible.

This procedure – choosing the parameter that makes the observation most likely – is called maximum likelihood estimation. The chapter Statistical models treats it in general and for other distributions.

Digression: the Poisson binomial distribution¶

• Parameters $p_1,\dots,p_n \in [0,1]$, one probability per trial

• pmf $p(k) = \displaystyle\sum_{A \in F_k} \prod_{i \in A} p_i \prod_{i \notin A} (1-p_i)$, where $F_k$ is the set of subsets of $\{1,\dots,n\}$ with $k$ elements

• Expected value $\sum_{i=1}^n p_i$

• Variance $\sum_{i=1}^n p_i(1-p_i)$

• Example application: $n$ components from different batches, each with its own failure probability – how many of them fail?

• In scipy.stats: no object of its own; the pmf is easy enough to compute yourself.

We are no longer tossing the same coin $n$ times but $n$ different coins, each with its own success probability $p_i$. The $X_i \sim \operatorname{Ber}(p_i)$ are still independent, just no longer identically distributed. Expected value and variance therefore follow exactly as for the binomial distribution, from linearity and from the variance rule for independent summands; when all the $p_i$ are equal, as for the binomial distribution, we get $np$ and $np(1-p)$ back.

The pmf itself is unwieldy – the sum runs over $\binom{n}{k}$ subsets. In practice it is computed iteratively: start with “no coins, no successes, with probability 1” and work the coins in one at a time.

The interesting case is many small $p_i$. If we set every $p_i = \frac{\lambda}{n}$ and let $n$ grow, the expected value stays fixed at $\lambda$ while each individual trial becomes ever less likely. This is the passage to the Poisson distribution:

Poisson distribution¶

• Parameter $\lambda > 0$, notation $\operatorname{Pois}(\lambda)$

• pmf $p(k) = e^{-\lambda}\dfrac{\lambda^k}{k!}$ for $k \in \mathbb{N}_0$

• Expected value $\lambda$

• Variance $\lambda$

• Example applications:

  • How many requests reach the server in one second?

  • How many typos are there on one page?

  • How many atomic nuclei in a specimen decay in one minute?

• In scipy.stats: st.poisson(mu=λ)

The guiding question: how many events occur in a fixed period of time, if they occur independently of one another and at a constant mean rate? In contrast to every distribution in this chapter so far, this one does not have finite support – in principle there could be arbitrarily many events, even if large numbers become very unlikely.

That the pmf sums to 1 follows from the exponential series $\sum_{k \geq 0} \frac{\lambda^k}{k!} = e^{\lambda}$:

The expected value follows with an index shift – the summand for $k=0$ is 0 and drops out:

For the variance, compute $\mathbb{E}(X(X-1)) = \lambda^2$ in just the same way, which gives $\mathbb{V}(X) = \mathbb{E}(X^2) - \lambda^2 = \lambda^2 + \lambda - \lambda^2 = \lambda$. So here expected value and variance coincide – a practical test of whether a Poisson distribution can fit a dataset at all.

For completeness: the geometric distribution¶

• Parameter $p \in (0,1]$, notation $\operatorname{Geom}(p)$

• pmf $p(k) = (1-p)^{k-1}p$ for $k \in \{1,2,3,\dots\}$

• Expected value $\dfrac{1}{p}$

• Variance $\dfrac{1-p}{p^2}$

• Example applications:

  • How many times do I have to throw a die until the first six comes up? ($p = \frac16$, on average 6 throws)

  • How many connection attempts does it take until the first success?

• In scipy.stats: st.geom(p)

We repeat a Bernoulli experiment until the first success and count the trials. For the first $k-1$ trials to fail and the $k$-th to succeed has probability $(1-p)^{k-1}p$ – hence the pmf.

The geometric distribution is memoryless: if you have already thrown the die ten times in vain, the probability of needing another $k$ throws is the same as it was at the start. The die does not remember. It is thus the discrete counterpart of the exponential distribution, the only memoryless distribution in the continuous case.

Looking back¶

All the distributions in this chapter hang together:

• The categorical distribution is the most general distribution on a finite set; its parameter space is the simplex $\Delta_n$.

• The discrete uniform distribution is the centre of that simplex and has maximal entropy.

• The Bernoulli distribution is the case $n=2$.

• The binomial distribution counts the successes in $n$ independent Bernoulli trials with the same $p$, the Poisson binomial distribution allows different $p_i$.

• The Poisson distribution is their limiting case for many trials, each of them very unlikely.

• The geometric distribution turns the question around: not how many successes in a fixed number of trials, but how many trials up to the first success.

Next we leave countable sets of outcomes behind. Look at the binomial distribution not for small $p$ but for large $n$, and the passage to the limit leads not to the Poisson distribution but to the normal distribution – that is the de Moivre–Laplace theorem, and the chapter The normal distribution opens with it.