Every month a shop sends you a file of its customers. This month's file looks fine: no blanks, no silly ages. But something feels different. Did the customers change, or did someone change how the data is collected? The Population Stability Index (PSI) turns that feeling into one number. PyDataQuality uses it when it compares a new file with an earlier one.
Step 1: sort people into bands
Take one column, say how much each customer spent, and put every customer into a band: Low, Mid or High spenders. Then work out what share of customers is in each band. Do the same for last month, using the same bands.
| Band | Last month (E, expected) | This month (A, actual) |
|---|---|---|
| Low | 20% = 0.20 | 40% = 0.40 |
| Mid | 50% = 0.50 | 40% = 0.40 |
| High | 30% = 0.30 | 20% = 0.20 |
Step 2: the formula
What is ln? It is the natural logarithm, a button on any scientific calculator. All you need to know: ln of 1 is 0 (no change), ln of a number bigger than 1 is positive, and ln of a number smaller than 1 is negative.
Step 3: work it out
| Band | A − E | A ÷ E | ln(A ÷ E) | Product |
|---|---|---|---|---|
| Low | +0.20 | 2.00 | +0.693 | 0.139 |
| Mid | −0.10 | 0.80 | −0.223 | 0.022 |
| High | −0.10 | 0.67 | −0.405 | 0.041 |
| PSI (add them up) | 0.20 | |||
Why every band adds, never subtracts
Look at the signs. When a band grows, A − E is positive and so is the log. When a band shrinks, both are negative, and a negative times a negative is positive. So every band can only push the PSI up. Changes in different directions never cancel each other out, which is exactly what you want from a "how much changed" score.
Step 4: read the number
| PSI | What it means |
|---|---|
| Below 0.1 | Stable. Nothing worth worrying about. |
| 0.1 to 0.25 | A moderate shift. Worth a look. |
| 0.25 or more | A large change. PyDataQuality reports it as a warning. |
Our shop scored 0.20: a moderate shift. Maybe there was a sale, or maybe the export now leaves out big customers. The number does not tell you why; it tells you where to look.
For comparison, if this month had been 22%, 49% and 29%, the PSI would be about 0.002: tiny wobbles, completely normal.
One detail: empty bands
If a band has 0% in either month, A ÷ E would mean dividing by zero, or taking the log of zero. Both break the maths. PyDataQuality quietly replaces a zero share with a tiny one (0.0001), so the formula still works and an empty band still counts as a big change.
Try it yourself
Two bands. Last month 50% and 50%. This month 60% and 40%. Work out the PSI. Is it stable?
Answer: (0.10 × ln 1.2) + (−0.10 × ln 0.8) = 0.018 + 0.022 = 0.04. Stable.
In Python
import numpy as np
expected = np.array([0.20, 0.50, 0.30])
actual = np.array([0.40, 0.40, 0.20])
psi = np.sum((actual - expected) * np.log(actual / expected))
print(round(psi, 2)) # 0.2
Want lessons like this for your team or class?
I teach data science and AI in plain language, to students, staff and community groups. Invite me to run a session.