Exploratory data analysis: wine, heavy rainfall, Titanic

Wine¶

We begin with an old dataset, which can be downloaded from the UC Irvine Machine Learning Repository, among other places:

Wine. (1991). UCI Machine Learning Repository.

According to the documentation the dataset has 178 rows (called instances here, as they often are) and 14 variables. The first variable is categorical, encoded as an int from {1, 2, 3}; the others are continuous real variables (hence float). The licence is CC-BY-4.0, so we may use the dataset freely, modify it and share our results, provided we name the original source:

Riccardo Leardi, riclea@anchem.unige.it

The dataset comes as two files:

• wine.data

• wine.names

Here wine.data is the dataset proper, without any metadata, and wine.names holds the metadata in human-readable form.

From wine.names we take the following in particular:

These data are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines.

• The attributes are

  1. Alcohol

  2. Malic acid

  3. Ash

  4. Alcalinity of ash

  5. Magnesium

  6. Total phenols

  7. Flavanoids

  8. Nonflavanoid phenols

  9. Proanthocyanins

  10. Color intensity

  11. Hue

  12. OD280/OD315 of diluted wines

  13. Proline

• Number of Instances

  • class 1 59

  • class 2 71

  • class 3 48

Now that we have successfully downloaded the data and read through the metadata that come with them, we can ask what we actually want to do with them.

One obvious task is to find out whether the three classes (that is, the three producers of the wine) can be told apart by the wine’s chemical composition. In statistical terms: do the distributions of the 13 variables differ enough from class to class for the individual data points (the instances) to be assigned to a class on that basis?

We can set this up as a game: if we delete the first variable, the class, can it be reconstructed from the rest? In the simple version we do not delete the class yet, but use it to estimate three different distributions for each of the 13 attributes. Then, for every instance and ignoring its class, we judge which distribution it most likely belongs to, and check whether we were right.

In the advanced version we split the data into training and test data, or form the three (or more) classes without using the variable “class” at all. First, though, let us take a look at the data.

Quite apart from the classes, one thing is interesting straight away: how are the data distributed? We ask that now for each variable on its own.

We have to admit that this distribution does not look particularly normal. You could almost make out a bimodal distribution – a mixture, that is, of two normal distributions with expected values near 12 and 13.5. Rather than go on guessing from the picture, let us note that we shall need a method for testing such guesses rigorously.

In our example we can press the class into service as an auxiliary variable:

And indeed, clear differences show at once. The ‘blue’ wine (class 1) has more alcohol on average than the others, the ‘orange’ wine (class 2) less. Let us draw in the corresponding normal densities:

Unfortunately it is just as clear that the overlaps are so large that between 12 and 14.5 per cent alcohol by volume no sensible prediction of the class is possible.

For completeness we also model the mixture distribution:

Having seen that the first attribute, the alcohol content, depends on the class, it is natural to examine all the other attributes the same way. So we rewrite the code generically and look at all the variables at once:

The pictures suggest that, besides attribute 1 (alcohol), attributes 2 (malic acid), 12 (OD280/OD315 of diluted wines, the protein content) and 13 (proline, an amino acid) might also serve to keep the classes apart.

From this we see that malic acid is a rather poor indicator of the class, whereas protein and proline content together look considerably better suited to prediction than, say, the alcohol content. More precisely: protein content separates class 3 from classes 1 and 2, and proline content separates class 1 from classes 2 and 3. Neither separation is really clean, however, as the plot already shows.

Finally, we look at the relationship between individual attributes with a scatter plot:

Whereas alcohol content and malic acid show no obvious relationship, the protein-proline plot has a pronounced white patch: a high proline content goes together with a high protein content. It would now be interesting to compute measures of this relationship, and also to search the specialist literature for an explanation of how it comes about.

To see whether the relationship is only apparent, and perhaps really arises from a dependence on the class, we look at the scatter plot once more, with the classes distinguished by colour:

And indeed it is quite plain, as it was in the histograms before, that these two attributes taken together separate the classes reasonably well.

What we would like, of course, is a coordinate-independent procedure for separating the classes, so that we do not have to pick two variables by hand. Later, principal component analysis (PCA) will give us one.

Heavy rainfall events (CatRaRE)¶

The German weather service (DWD) makes data on heavy rainfall events available:

Catalogues of radar-based heavy precipitation events, as a basis for analysing heavy rainfall activity and its damage potential in Germany since 2001

• The description of the datasets is to be found at the DWD (in German)

• The metadata (a description of the attributes) are available as an XLSX (in German)

• The data download links have to be cited whenever the data are used (in German)
  Lengfeld, Katharina; Walawender, Ewelina; Winterrath, Tanja; Weigl, Elmar; Becker, Andreas (2021): Starkregenereignisse Version 2021.01 mit Überschreitung der DWD-Warnstufe 3 für Unwetter basierend auf RADKLIM-RW Version 2017.002,
  DOI: 10.5676/DWD/CatRaRE_W3_Eta_v2021.01

• The CSV file we want to look at can be downloaded directly

Now we have to settle on a research question. One question is always obvious: what distribution do the variables have? We can once again look at the bare frequency, that is, at how the points in time are distributed. We could follow it over the years, or average over the years and analyse how the month influences the frequency. The latter suggests itself: heavy rain is a weather phenomenon and so depends on the seasons, and we can expect a clear relationship.

At the moment the start time of a heavy rainfall event is encoded as an int. We turn that to our advantage: make a string out of the int 200101020350 and extract the month from it:

So the expected relationship shows clearly: in July and June heavy rainfall events are considerably more frequent than in winter.

Next we can look at the location, say at the variable GMD_RRmax and the place "Düsseldorf":

We see that in Düsseldorf, June and July have so far been the only months with heavy rainfall events.

Seven events are too few for looking at further correlations, so there is little to be gained from staying with Düsseldorf.

To get an impression of the spatial distribution, we look at a scatter plot of the locations of maximum precipitation in the RADKLIM grid (x_RRmax, y_RRmax):

You can almost make out Germany – and, above all, that the y axis needs rescaling. So we read up on the scaling of the RADKLIM grid and learn that every grid point corresponds to exactly $1 \times 1\,\mathrm{km}^2$. The plot therefore merely has to be square. To see a little more of the distribution, we also make every point partly transparent. Large clusters then become visible:

This shows that heavy rainfall events are more frequent in the south than in the north. We might also suspect a relationship with the duration (Duration) of the events:

Instead of the suspected relationship we see something else entirely: the duration is mostly a single-digit number, and longer events are rather rare. We can bring in the spatial relationship as well by distinguishing by colour:

Since there is not much to see here (apart from the rarity of longer events), we look at the analogous plot for the mean precipitation:

Here we begin to suspect that the scale is hard to read because the more extreme events (in light blue) take up so much of it. Excluding those events shows more:

But we get the same effect by adjusting the colormap instead:

That will do on heavy rainfall events for now.

Titanic¶

Now we get to know matplotlib’s object-oriented interface, using the Titanic dataset. On Kaggle it is part of a ‘competition’.

We download the training dataset as titanic.csv.

It is hardly surprising that in class 3 (the cheapest, in green) the ages tend to be lower than in the other classes. In passing we also notice that class 3 is more frequent than the others (which should perhaps not surprise us either).

To see whether passenger class and survival are related, a scatter plot is enough:

The scatter plot shows a correlation: passengers in class 3 more often did not survive, and passengers who survived were more often in class 1. It was only visible, however, because we set alpha = 0.005, and it suffers from ignoring the absolute frequencies of the classes (a base rate fallacy).

The correct thing to do here is to compute the probability that a passenger in class 3 survived:

We see that a first-class passenger was considerably more likely to survive than to die. For second-class passengers the odds are already worse than 50/50, and fewer than a quarter of third-class passengers survived.

It should not surprise us that these probabilities do not sum to 1: they refer to different populations. For the reverse conditional probability, on the other hand, we do expect a sum of 1:

This shows that survival status certainly does not tell us which class someone travelled in (because the passengers are not spread evenly over the classes). The two directions $P(\text{Survived} \mid \text{Pclass})$ and $P(\text{Pclass} \mid \text{Survived})$ are tied together by the class frequencies; exactly how is the business of Bayes’ theorem in the chapter Covariance and dependence, and the chapter Titanic EDA works the same thing through for the variable Sex.

It is always interesting how representative a sample is, because that tells us how far the relationships we have found might generalise. So we now look at whether the variable Sex is correlated with other variables. Admittedly we do not yet know what we are doing, but let that be motivation for learning later what it was we did:

The second return value, pvalue, is a p-value: the probability, for two variables that really are uncorrelated, of seeing a correlation coefficient at least as far from 0 as ours. Here it is 0.92, so an $r$ like this would be perfectly ordinary for uncorrelated data: the sample cannot tell the tiny deviation from 0 apart from sampling error. That is all the p-value says – in particular, it does not say how likely it is that sex and age are uncorrelated. What a p-value is and is not is settled in the chapter Statistical tests.

Here we clearly see $r > 0.5$, and a vanishingly small probability $p$ of observing such an $r$ for data that are in fact uncorrelated.

So, naively (following the Scipy Stats documentation), we expect the variable SibSp to be correlated with sex. According to the metadata, SibSp is the number of siblings and spouses travelling with the passenger. Age, too, shows a correlation with SibSp: