Dominion Akinrotimi
Home / Learn / Lesson 2
Lesson 2 of 3

Has my data changed? The Population Stability Index

One number that tells you how much a column's mix has shifted since last time.

By Dominion Akinrotimi · Data Quality, Explained Simply

Every month a shop sends you a file of its customers. This month's file looks fine: no blanks, no silly ages. But something feels different. Did the customers change, or did someone change how the data is collected? The Population Stability Index (PSI) turns that feeling into one number. PyDataQuality uses it when it compares a new file with an earlier one.

Step 1: sort people into bands

Take one column, say how much each customer spent, and put every customer into a band: Low, Mid or High spenders. Then work out what share of customers is in each band. Do the same for last month, using the same bands.

BandLast month (E, expected)This month (A, actual)
Low20% = 0.2040% = 0.40
Mid50% = 0.5040% = 0.40
High30% = 0.3020% = 0.20
Think of a sweet jarLast month the jar was 20% red, 50% green and 30% yellow. This month it is 40% red. Someone has clearly been adding red sweets. PSI measures how much the mix changed.

Step 2: the formula

PSI = Σ (A − E) × ln(A ÷ E)For each band: take the change in share, multiply it by the natural log of the ratio, then add up all the bands. Σ just means "add them all up".

What is ln? It is the natural logarithm, a button on any scientific calculator. All you need to know: ln of 1 is 0 (no change), ln of a number bigger than 1 is positive, and ln of a number smaller than 1 is negative.

Step 3: work it out

BandA − EA ÷ Eln(A ÷ E)Product
Low+0.202.00+0.6930.139
Mid−0.100.80−0.2230.022
High−0.100.67−0.4050.041
PSI (add them up)0.20

Why every band adds, never subtracts

Look at the signs. When a band grows, A − E is positive and so is the log. When a band shrinks, both are negative, and a negative times a negative is positive. So every band can only push the PSI up. Changes in different directions never cancel each other out, which is exactly what you want from a "how much changed" score.

Step 4: read the number

PSIWhat it means
Below 0.1Stable. Nothing worth worrying about.
0.1 to 0.25A moderate shift. Worth a look.
0.25 or moreA large change. PyDataQuality reports it as a warning.

Our shop scored 0.20: a moderate shift. Maybe there was a sale, or maybe the export now leaves out big customers. The number does not tell you why; it tells you where to look.

For comparison, if this month had been 22%, 49% and 29%, the PSI would be about 0.002: tiny wobbles, completely normal.

One detail: empty bands

If a band has 0% in either month, A ÷ E would mean dividing by zero, or taking the log of zero. Both break the maths. PyDataQuality quietly replaces a zero share with a tiny one (0.0001), so the formula still works and an empty band still counts as a big change.

Try it yourself

Two bands. Last month 50% and 50%. This month 60% and 40%. Work out the PSI. Is it stable?

Answer: (0.10 × ln 1.2) + (−0.10 × ln 0.8) = 0.018 + 0.022 = 0.04. Stable.

In Python

import numpy as np
expected = np.array([0.20, 0.50, 0.30])
actual = np.array([0.40, 0.40, 0.20])
psi = np.sum((actual - expected) * np.log(actual / expected))
print(round(psi, 2))   # 0.2

Want lessons like this for your team or class?

I teach data science and AI in plain language, to students, staff and community groups. Invite me to run a session.