Unsupervised Machine Learning#


In this chapter, we’re going to look at unsupervised machine learning. We’ll be looking at clustering and dimensional reduction in particular. We’ll be leaning heavily on scikit-learn. Also in the category of unsupervised machine learning, but not covered here, are anomaly detection and outlier detection.

Before we get cracking on the chapter content, let’s do our normal set of imports.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

random_state = 42  # We'll use this throughout to make this page reproducible
prng = np.random.default_rng(random_state)


To cluster data is to use a set of high-dimensional features to create a grouping wherein data points in the same cluster are more similar to each other than data points in distinct clusters.

When might you wish to cluster? You can use clusters to get informative fixed effects based on other data. You can use clustering to create features for (supervised) machine learning. You can use clusters to help you understand different types of (groups of) data in your sample. Some more applied examples are: market segmentation, image segmentation, and anomaly detection.

As with much of the API, there are two ways to use the clustering algorithms found in the scikit-learn package: the first implements the fit method to learn clusters on training data; the second is a function, that, given training data, returns an array of integer labels corresponding to the different clusters. As ever, there are fit, fit_tranform, and transform methods so that you can create clusters, create and allocate data to clusters, or just allocate data to existing clusters, respectively.

An example: K-means#

To help illustrate exactly what’s going on in clustering, let’s look at a specific example using the K-means algorithm. K-means creates clusters by trying to separate samples in \(K\) groups of equal variance, minimising a criterion known as the inertia or within-cluster sum-of-squares:

\[ \sum_{i=0}^{n}\min_{\mu_j \in C}(||x_i - \mu_j||^2) \]

where a set of \(N\) samples is sorted into \(K\) disjoint clusters \(C\) each described by the mean point \(\mu_j\) of the samples in that cluster. The means are commonly called the cluster “centroids”; note that they are not, in general, points from the input data, \(X\).

Let’s see an example of K-means in action. First, let’s generate some data

from sklearn.datasets import make_blobs

batch_size = 45
centers = [[1, 1], [-1, -1], [1, -1]]
n_clusters = len(centers)
X_to_cluster, labels_true = make_blobs(n_samples=3000, centers=centers, cluster_std=0.7, random_state=random_state)
array([[ 1.56876808,  1.94936802],
       [ 1.47695363, -1.8075846 ],
       [ 1.25807132,  0.72466283],
       [-0.7031532 , -0.97238707],
       [ 0.79388097,  0.01692888],
       [ 0.60502668,  1.12918591]])

Now we fit K-means

from sklearn.cluster import KMeans

k_means = KMeans(init="k-means++", n_clusters=3, n_init=10)
KMeans(n_clusters=3, n_init=10)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

And plot the results:


Clustering algorithms in scikit-learn#

There are too many clustering algorithms just in scikit-learn to cover them in any real detail so instead we will simply reproduce this ridiculously useful chart and table that show all of the package’s clustering algorithms and their strengths and weaknesses.


Method name




Geometry (metric used)


number of clusters

Very large n_samples, medium n_clusters with MiniBatch code

General-purpose, even cluster size, flat geometry, not too many clusters, inductive

Distances between points

Affinity propagation

damping, sample preference

Not scalable with n_samples

Many clusters, uneven cluster size, non-flat geometry, inductive

Graph distance (e.g. nearest-neighbor graph)



Not scalable with n_samples

Many clusters, uneven cluster size, non-flat geometry, inductive

Distances between points

Spectral clustering

number of clusters

Medium n_samples, small n_clusters

Few clusters, even cluster size, non-flat geometry, transductive

Graph distance (e.g. nearest-neighbor graph)

Ward hierarchical clustering

number of clusters or distance threshold

Large n_samples and n_clusters

Many clusters, possibly connectivity constraints, transductive

Distances between points

Agglomerative clustering

number of clusters or distance threshold, linkage type, distance

Large n_samples and n_clusters

Many clusters, possibly connectivity constraints, non Euclidean distances, transductive

Any pairwise distance


neighborhood size

Very large n_samples, medium n_clusters

Non-flat geometry, uneven cluster sizes, outlier removal, transductive

Distances between nearest points


minimum cluster membership, minimum point neighbors

large n_samples, medium n_clusters

Non-flat geometry, uneven cluster sizes, outlier removal, transductive, hierarchical, variable cluster density

Distances between nearest points


minimum cluster membership

Very large n_samples, large n_clusters

Non-flat geometry, uneven cluster sizes, variable cluster density, outlier removal, transductive

Distances between points

Gaussian mixtures


Not scalable

Flat geometry, good for density estimation, inductive

Mahalanobis distances to centers


branching factor, threshold, optional global clusterer.

Large n_clusters and n_samples

Large dataset, outlier removal, data reduction, inductive

Euclidean distance between points

Bisecting K-Means

number of clusters

Very large n_samples, medium n_clusters

General-purpose, even cluster size, flat geometry, no empty clusters, inductive, hierarchical

Distances between points

Dimensional Reduction#

Dimensional reduction is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data. Reducing dimensions can be helpful because of the curse of dimensionality and the sparsity of data in higher dimensions making it either computationally infeasible or perhaps impossible to run standard algorithms. It can also be helpful to summarise and visualise high-dimensional information in a continuous way (rather than the discrete way offered by clustering).

Factor analysis is one type of dimensional reduction that may already be familiar to you as an economist. This is available in scikit-learn, along with a range of other reduction algorithms.

In the rest of this chapter, we’ll showcase just two: principal component analysis and UMAP.

Principal Component Analysis#

The absolute bread and butter of dimensional reduction algorithms is principal component analysis, so we’ll start with that as a good example of how to use dimensional reduction algorithms in general with scikit-learn-like APIs. We will project the 4-dimensional Iris data down into two dimensions using PCA.

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA

iris = load_iris()
X_iris, y_iris = iris.data, iris.target
target_names_iris = iris.feature_names
pca_iris = PCA(n_components=2)
print(f"The dimensionality of the raw data is: {X_iris.shape}")
X_dim_red = pca_iris.fit_transform(X_iris)
print(f"explained variance ratio (1st two components): {(pca_iris.explained_variance_ratio_)}")
The dimensionality of the raw data is: (150, 4)
explained variance ratio (1st two components): [0.92461872 0.05306648]
(150, 2)
colors = ["navy", "turquoise", "darkorange"]
lw = 2

fig, ax = plt.subplots()
for color, i, target_name in zip(colors, [0, 1, 2], target_names_iris):
        X_dim_red[y_iris == i, 0], X_dim_red[y_iris == i, 1], color=color, alpha=0.8, lw=lw, label=target_name
ax.legend(loc="best", shadow=False, scatterpoints=1)
ax.set_title("PCA of IRIS dataset")

You can see here that we have projected out onto two principal dimensions (axes) and that there is some good separation of points.

UMAP, or Uniform Manifold Approximation and Projection for Dimension Reduction#

UMAP is a relatively recent algorithm [McInnes, Healy, and Melville, 2018] that is somewhat similar to t-SNE (t-distributed Stochastic Neighbor Embedding). Both have issues with reproducing structure and should be used with care but both can be quite impresssive with the right parameters. t-SNE is included in scikit-learn, but for UMAP, you’ll need to install the umap-learn package.

We will use the penguins dataset to demo UMAP (following the documentation).

penguins = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv")
penguins["sex"] = penguins["sex"].str.title()
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 Male
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 Female
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 Female
3 Adelie Torgersen NaN NaN NaN NaN NaN
4 Adelie Torgersen 36.7 19.3 193.0 3450.0 Female

As ever, it does help to clean up the data. We will drop NAs and convert each feature into z-scores (number of standard deviations from the mean) for comparability. We will also only use the numeric columns.

import umap
reducer = umap.UMAP()
penguins_data = (
scaled_penguin_data = StandardScaler().fit_transform(penguins_data)

Let’s now create the embedding using UMAP:

embedding = reducer.fit_transform(scaled_penguin_data)
(333, 2)

And plot the results

species = list(penguins["species"].unique())
species_map = {i: x for i, x in zip(range(len(species)), species)}

fig, ax = plt.subplots()
for i, spec_name in species_map.items():
    this_cut = penguins.dropna(how="any")["species"].astype("category").cat.codes == i
        embedding[this_cut, 0],
        embedding[this_cut, 1],
ax.set_title("UMAP projection of the Penguins dataset")

This does a useful job of capturing the structure of the data, which you can check for yourself by looking at a pairplot of all of the numerical fields.

While the benefits of dimensional reduction are small for such low dimensional datasets, as soon as you are dealing with 6 or more dimensions, perhaps as many as thousands, the benefits become larger.