PSI (lesson 2) needs bands. The Kolmogorov-Smirnov test, or KS test, compares two lists of numbers without any bands at all. PyDataQuality's comparison report uses it next to PSI to say whether a column's numbers have changed shape. Don't let the long name scare you: it comes down to finding the biggest gap between two lines.
Step 1: two lists
Two classes take the same quiz, marked out of 10:
Step 2: "how many have scored this much or less?"
For every possible score, ask: what share of the class scored that much or less? This running share is called the cumulative share. It always starts at 0 and climbs to 1 (everyone).
| Score | Class A, share at or below | Class B, share at or below | Gap |
|---|---|---|---|
| 2 | 1/5 = 0.2 | 0.0 | 0.2 |
| 3 | 0.4 | 0.0 | 0.4 |
| 4 | 0.4 | 0.2 | 0.2 |
| 5 | 0.6 | 0.2 | 0.4 |
| 6 | 0.6 | 0.4 | 0.2 |
| 7 | 0.8 | 0.4 | 0.4 |
| 8 | 0.8 | 0.8 | 0.0 |
| 9 | 1.0 | 0.8 | 0.2 |
| 10 | 1.0 | 1.0 | 0.0 |
Step 3: the biggest gap is the answer
Here D = 0.4. D is always between 0 (the two lists have exactly the same shape) and 1 (every score in one list is below every score in the other).
Step 4: is the gap big enough to matter?
With only five pupils per class, a gap of 0.4 could easily happen by luck. The test turns D into a p-value: the chance of seeing a gap this big if both classes were really the same. For our classes the p-value is about 0.87, so an 87% chance it is just luck. No real difference.
Now suppose Class A scored 2, 3, 4, 5, 6 and Class B scored 7, 8, 9, 10, 11. Every A is below every B, so D = 1, and the p-value is about 0.008: less than a 1% chance of luck. That is a real difference.
For big lists there is a handy shortcut. At the 5% level, the gap D is "big" when it is larger than
That last point is why PyDataQuality does not shout about every KS result on its own. With big files the test notices tiny, harmless changes, so it is shown next to PSI and the plain change in the average, and you judge them together.
Try it yourself
List A: 1, 2, 3, 4. List B: 3, 4, 5, 6. Find the biggest gap between the cumulative shares.
Answer: at the value 2, A is at 0.5 and B is at 0, so D = 0.5.
In Python
from scipy.stats import ks_2samp
result = ks_2samp([2, 3, 5, 7, 9], [4, 6, 8, 8, 10])
print(result.statistic, result.pvalue) # 0.4 0.873...
Want lessons like this for your team or class?
I teach data science and AI in plain language, to students, staff and community groups. Invite me to run a session.