Random variables and distributions - Data Science Lecture Notes from Konrad Völkel

The typical ‘interface’ to probability distributions is the random variable. Unlike a bare distribution, a random variable also keeps track of which concrete values can turn up as outcomes.

Random variables¶

Let us recall the definition, now in the notation we will use from here on:

A reminder: if $\Omega$ and $\mathcal{X}$ are finite sets, every subset counts as an event, and then every such map is a random variable. We will see later that we reach for the term random variable precisely when we are interested in expected values or higher moments.

Where does the randomness come from?¶

Before we go on calculating, a question that is rarely asked and even more rarely answered: where in a dataset does the randomness actually come from? The short answer is that it is not in there at all. It is a decision we make.

A measurement that has already been taken is not random – the die has come to rest, the temperature has been read off, the row is sitting in the file. Collecting the data is a deterministic process too: a sensor writes down what it measures. Randomness is not something we find in the data; it is the language we use to talk about what we do not know about the data, or do not want to know. We treat the observed frequencies as if they were probabilities and draw conclusions from that model – so randomness is a modelling tool, not a natural substance.

All the same, there are three places where we do meet it quite concretely:

In the selection. When we draw a subset from the data we have – a sample, a mini-batch, a split into training and test data – the randomness is ours. We draw at random deliberately, so that the selection adds no structure of its own that we might later mistake for a result.
In sampling from a model. Drawing from a model distribution produces synthetic data. This randomness, too, is ours.
On the computer, not at all. Both of these run on a deterministic device that produces only pseudo-randomness: a sequence of numbers that looks random but is completely determined by its starting value, the seed. That is not a shortcoming; it is what makes analyses reproducible (see Programming with stochastics).

The path the data travel can be told in three pictures. First, data collection: a section of the world is chosen, raw data about it accumulate, and cleaning turns the raw data into the data we actually work with.

Figure 1:Data collection: from the world via a section and raw data to the data.

Then the modelling. A model is built from the data, and from the model, data can be generated in turn – synthetic data. Whether the model is any good comes down to whether its synthetic data resemble the real thing.

Figure 2:Modelling: from the data to the model and back to synthetic data.

And finally the path to the target variable, which is what we were after all along: the data are transformed, and from the transformed data we compute the one number we wanted to know.

Figure 3:Analysis: from the data via transformed data to the target variable.

Example

A throw of a die, spelled out in full. The section of the world is a die on a table. As raw data, suppose a camera delivers the 3D coordinates of the eight corners at every instant: $8 \cdot 3$ numbers per frame. Once the unusable frames have been weeded out, these are the data. From them we compute transformed data: the centre and the rotation angles of the die, $3 + 3$ numbers, since that is all it takes to describe how a cube is lying. Only at the end comes the target variable we actually care about: the number of pips on top, a number in $\{1,\dots,6\}$ .

The whole path from the world to the target variable is exactly what we call a random variable: a map $X \colon \Omega \to \{1,\dots,6\}$ , where $\Omega$ is the set of possible world states. Nothing about this map is random – it is a long but perfectly unambiguous rule for calculating. The randomness lies solely in our not knowing which $\omega \in \Omega$ we are in, which is why we equip $\Omega$ with a probability measure. The statement “the die is fair” then reads

P(X = 6) := P_\Omega\left( \{ \omega \in \Omega : X(\omega) = 6 \} \right) = P_\Omega\left( X^{-1}(\{6\}) \right) = \dfrac{1}{6}.

(1)

So the notation $P(X = 6)$ is shorthand: it means the probability of the set of all world states in which the die ends up showing a six. Exactly this construction – a measure on $\Omega$ turning, via $X$ , into a measure on the values of $X$ – is the subject of the next section.

Induced probability measure¶

Proof

We check the axioms:

We have $X^{-1}(\mathcal{X}) = \Omega$ , hence $P_X(\mathcal{X}) = P_\Omega\left( X^{-1}(\mathcal{X}) \right) = P_\Omega(\Omega) = 1$ .
Let $A_1,\dots$ be a sequence of pairwise disjoint events in $\mathcal{X}$ . Then

X^{-1}\left(\bigcup_{i \geq 1} A_i\right) = \left\{\omega \in \Omega\ \mid\ X(\omega) \in A_i \text{ for some } i \right\} = \bigcup_{i \geq 1} X^{-1}(A_i)

(3)

and hence

P_X\left(\bigcup_{i\geq 1} A_i\right) = P_\Omega\left( X^{-1} \left(\bigcup_{i \geq 1} A_i\right) \right) = P_\Omega\left( \bigcup_{i \geq 1} X^{-1}(A_i) \right)

(4)

= \sum_{i \geq 1} P_\Omega \left( X^{-1} A_i \right) = \sum_{i \geq 1} P_X(A_i).

(5)

Example

30 children are playing in a room: 10 of them are wearing a blue jumper, 10 a red one, 5 a yellow one and another 5 a black one. We model $\Omega =$ the set of children and $X \colon \Omega \to \{r,y,b,k\}$ as the map that sends each child to the letter standing for the colour of their jumper ( $r$ red, $y$ yellow, $b$ blue, $k$ black). With that, the distribution of $X$ is defined; it is a probability measure on the set $\{r,y,b,k\}$ . For instance, we can compute:

P_X\left(\{b,r\}\right) = P_\Omega\left(X^{-1} \left(\{b,r\}\right) \right) = \frac{20}{30}

(6)

where we have tacitly assumed that all the children are treated alike – that is, whenever we look at an outcome of $X$ , we pick a child uniformly at random and look at the colour of its jumper.

Example

Composing maps yields new random variables. Let $X$ be the throw of a fair die, that is, $\Omega = \{1,\dots,6\}$ with the uniform distribution and $X$ the identity.

(a) Let $g \colon \{1,\dots,6\} \to \{\text{even}, \text{odd}\}$ be the map that sends a number of pips to its parity. Then

P_g(\{\text{even}\}) = P_\Omega\left( g^{-1}(\{\text{even}\}) \right) = P_\Omega(\{2,4,6\}) = \dfrac{3}{6} = \dfrac{1}{2}.

(7)

Here $g \circ X$ is itself a random variable – the composition of a random variable with a map always is – and its distribution is $P_{g(X)} = P_g$ . One writes $g(X)$ for short and speaks of “the random variable parity of the throw”.

(b) Let $f$ be the map that says whether the number of pips is at least 5. For the answer “no” we decompose the preimage into elementary events and use $\sigma$ -additivity:

P_f(\{\text{no}\}) = P_\Omega\left( \{1,2,3,4\} \right) = \sum_{i=1}^4 P_\Omega\left( \{i\} \right) = 4 \cdot \dfrac{1}{6} = \dfrac{2}{3}.

(8)

That is the typical calculation with induced measures: find the preimage, cut it into disjoint pieces and add up.

(c) Composing with $x \mapsto 2x$ gives the random variable $2X$ with values in $\{2,4,6,8,10,12\}$ , each with probability $\frac16$ . So you can calculate with random variables as you do with numbers – double them, add them, square them – and the distribution changes along with them. How expected value and variance behave under such operations is settled in the chapter Expected value and variance (linearity of the expected value).

If all we are interested in is the distribution $P_X$ , we can forget about $\Omega$ : the same distribution can be obtained from other sets $\Omega$ , and from distributions on them that are not uniform.

Probability mass function¶

In everyday work you usually calculate with probabilities not via the measure $P$ on events but via a function $p(x)$ that assigns each possible value $x$ its probability directly. In the discrete case, which is what this chapter is about, this function is called the probability mass function; its continuous counterpart, the density, we will meet in a later chapter:

Example

In the previous example we have $P_X(\{r\}) = \frac{10}{30}$ , $P_X(\{y\}) = \frac{5}{30}$ , $P_X(\{b\}) = \frac{10}{30}$ , $P_X(\{k\}) = \frac{5}{30}$ , and therefore

p(r):=\frac{10}{30},\ p(y):=\frac{5}{30},\ p(b):=\frac{10}{30},\ p(k):=\frac{5}{30}

(9)

defines a pmf for $P_X$ . One also says that $p$ is the pmf of $X$ . Another common name for the pmf is discrete density.

The pmf of the uniform distribution on $\Omega$ is $p(\omega) = \frac{1}{|\Omega|}$ .

The pmf of the Dirac distribution with mass at a fixed $\xi \in \Omega$ (see The formalism of stochastics) is

p(\omega) = \begin{cases} 1 & \omega = \xi,\\ 0 & \text{otherwise.} \end{cases}

(10)

Both presuppose that $\Omega$ is at most countable – for uncountable $\Omega$ every individual probability $P(\{\omega\})$ is typically 0, and the density takes over from the pmf.

Example

We are expecting an incoming message 3 characters long and model the three characters as random variables X, Y and Z. We know in advance that each character can only come from a finite character set; for the three random variables we write this range as $\mathcal{X},\mathcal{Y}$ and $\mathcal{Z}$ respectively, and the pmfs as $p(x),\ p(y),\ p(z)$ with $x \in \mathcal{X}, y \in \mathcal{Y}$ and $z \in \mathcal{Z}$ .

We can also regard the whole message as a single random variable, XYZ. Then certain sequences of letters may be more probable than their constituents would suggest: if $X=s$ and $Y=o$ , then $Z=s$ is surely more probable than it would be for arbitrary values of $X$ and $Y$ . All in all, $XYZ=sos$ is more probable than the product of the probabilities of $X=s$ , $Y=o$ and $Z=s$ . Writing the pmf of $XYZ$ as $p(x,y,z)$ , as usual, this says:

p(x,y,z) \neq p(x)p(y)p(z)

(11)

and we also say: the random variables – or rather their distributions – are not independent.

Induced pmf¶

Probability vector¶

A distribution given in this way by $p \in \Delta_n$ is also called a categorical distribution; we meet it again in the chapter The most important discrete distributions, where we also work out its expected value and its variance. Its simplest case has a name of its own.

Geometrically, $\Delta_2$ is a line segment – the one joining the two vertices $(1,0)$ and $(0,1)$ – and $\pi$ says where on this segment you are. Choosing a Bernoulli distribution therefore means choosing a point on a line segment.

So $\Delta_n$ is the space of all possible probability distributions on an $n$ -element set. The small cases are easy to picture: $\Delta_2$ is a line segment, $\Delta_3$ a filled triangle (two-dimensional, but embedded in $\mathbb{R}^3$ ), $\Delta_4$ a filled tetrahedron. In general $\Delta_n$ is an $(n-1)$ -dimensional object with $n$ vertices, called a simplex. One dimension is always lost, because the entries have to add up to 1 – exactly what we exploited for the Bernoulli distribution, where two numbers became one.

From data to models¶

The simplest form data can take is a single number, or a sequence of $N$ numbers. We like to call that an $N$ -dimensional vector, or picture it as the column of a table; in statistics it is a sample of size $N$ . Later we will also deal with tables of more than one column, where the data come as more complicated rows, but for now we stay in the simpler one-dimensional setting.

Even a single such column carries a probability distribution with it, with no modelling assumption at all: we simply count how often each value occurs.

That this really is a probability measure is just the statement that the relative frequencies add up to 1.

Example

A dataset with a single column $x$ holding a colour:

$i$	1	2	3	4
$x_i$	blue	red	blue	green

Here $\Omega = \{\text{blue}, \text{red}, \text{green}\}$ and $N = 4$ . The distribution of the data is $P(\{\text{blue}\}) = \frac{2}{4}$ , $P(\{\text{red}\}) = \frac{1}{4}$ and $P(\{\text{green}\}) = \frac{1}{4}$ , or, written as a probability vector, $p = \left(\frac{1}{2}, \frac{1}{4}, \frac{1}{4}\right)$ .

If the values are coded as numbers $0, 1, 2, \dots$ , you get this vector in a single line: np.bincount(data) / len(data) (see Numpy, section Probability vectors).

The following example is the special case $\Omega = \{0,1\}$ , in which a single number fixes the whole distribution.

Example

Given a sample of $N$ coin tosses $(x_1,\dots,x_N) \in \{0,1\}^N$ we can ask: ‘what success probability $\pi$ was behind this coin?’

One possible estimate is $\frac{1}{2}$ , arrived at without looking at the data.

If we do look at the sample, we can compute $e := \sum_{i=1}^N x_i$ , the number of successes. Then $N-e$ is the number of failures and $\hat{\pi} := \frac{e}{N}$ an estimate of the success probability $\pi$ , provided we assume the same coin was tossed throughout.

In the language of the previous section: each toss is $\operatorname{Ber}(\pi)$ -distributed, and $\hat{\pi}$ estimates the parameter of that distribution – so we are looking for a point on the line segment $\Delta_2$ .

With that, every role in statistics has been cast, even if we will not need the names until later: the coin, which could be tossed as often as you like, is the population; the $N$ tosses are a sample from it; and $\hat{\pi}$ is a statement about the population, obtained from the sample. In the same way a bag of sweets is the population and a handful from it the sample, from which you estimate the distribution of colours in the bag. What you know or suspect before you reach in (the coin looks normal, the bag is a standard mix) will later be called the prior. That the inference from sample to population is never certain, and how to put a number on that uncertainty, is the subject of the chapters Expected value and variance (Chebyshev’s inequality) and Statistical tests.

The distribution of the data is not yet a model – it is only another way of writing the data down. A model only arises once an assumption is added. Suppose the colour dataset above has gained a fifth entry:

$i$	1	2	3	4	5
$x_i$	blue	red	blue	green	blue

The distribution of the data is now $\left(\frac{3}{5},\frac{1}{5},\frac{1}{5}\right)$ . If, however, we assume that the colours are equally frequent except for one that occurs twice as often, we choose as our model the distribution $p = \left(\frac{1}{2},\frac{1}{4},\frac{1}{4}\right)$ – even though the observed relative frequencies differ from it. A distribution chosen in this way is called a model distribution. As a rule it has fewer parameters than the dataset has entries, it is smoother, and it says something even about values that do not occur in the data at all.

A model works in both directions. Determining a model distribution from data is the estimation problem the chapter Statistical models and estimators is about. In the other direction, you can draw from a model distribution and obtain synthetic data: a dataset that looks like a real one but is not. Synthetic data are the standard tool for testing a method of analysis, because here you know the right answer and can check whether the method finds it.

We now have two distributions on the same $\Omega$ : that of the data and that of the model. The next question is that of the quality of the model – do the data look as though they had been drawn from it? One asks how likely is that?, and the number that answers the question is called the likelihood: the probability of seeing exactly the observed data if the model is correct. A model that makes this number as large as possible is the best one by the maximum likelihood principle; the principle runs through the rest of the course and gets a section of its own in the chapter Statistical models and estimators. It is not the only way to put a number on the difference between two distributions: the chapter Expected value and variance measures the distance between data and synthetic data directly in the data space, and the chapter Relative entropy measures it between the distributions themselves.

That marks out the path the rest of the course takes, and it is the path of Figure 2: choose a section of the world, collect data, clean them (Data cleaning) and translate them into usable features, build models from those, and measure the models against the data. Everything else is the working out of these five steps. The next chapter begins with the last of them: comparing data and model in a single number.