Dominion Akinrotimi
Home / Learn / Lesson 3
Lesson 3 of 3

Are two lists the same shape? The KS test

Compare two lists of numbers by finding the biggest gap between two climbing lines.

By Dominion Akinrotimi · Data Quality, Explained Simply

PSI (lesson 2) needs bands. The Kolmogorov-Smirnov test, or KS test, compares two lists of numbers without any bands at all. PyDataQuality's comparison report uses it next to PSI to say whether a column's numbers have changed shape. Don't let the long name scare you: it comes down to finding the biggest gap between two lines.

Step 1: two lists

Two classes take the same quiz, marked out of 10:

Class A: 2, 3, 5, 7, 9     Class B: 4, 6, 8, 8, 10

Step 2: "how many have scored this much or less?"

For every possible score, ask: what share of the class scored that much or less? This running share is called the cumulative share. It always starts at 0 and climbs to 1 (everyone).

Think of a raceTwo teams run a race. At each second on the clock, count what fraction of each team has crossed the finish line. If one team is much faster, at some moment many of its runners are home while few of the other team are. That moment is the biggest gap.
ScoreClass A, share at or belowClass B, share at or belowGap
21/5 = 0.20.00.2
30.40.00.4
40.40.20.2
50.60.20.4
60.60.40.2
70.80.40.4
80.80.80.0
91.00.80.2
101.01.00.0

Step 3: the biggest gap is the answer

D = the largest |FA(x) − FB(x)|F means "the cumulative share". The two straight bars mean "ignore the minus sign". D is the biggest gap anywhere along the scores.

Here D = 0.4. D is always between 0 (the two lists have exactly the same shape) and 1 (every score in one list is below every score in the other).

Step 4: is the gap big enough to matter?

With only five pupils per class, a gap of 0.4 could easily happen by luck. The test turns D into a p-value: the chance of seeing a gap this big if both classes were really the same. For our classes the p-value is about 0.87, so an 87% chance it is just luck. No real difference.

Now suppose Class A scored 2, 3, 4, 5, 6 and Class B scored 7, 8, 9, 10, 11. Every A is below every B, so D = 1, and the p-value is about 0.008: less than a 1% chance of luck. That is a real difference.

The usual ruleIf the p-value is below 0.05 (a 5% chance of luck), people say the two lists really are different.

For big lists there is a handy shortcut. At the 5% level, the gap D is "big" when it is larger than

1.36 × √((n + m) ÷ (n × m))n and m are the sizes of the two lists. With 1,000 rows in each, that is about 0.06: with lots of data, even a small gap counts.

That last point is why PyDataQuality does not shout about every KS result on its own. With big files the test notices tiny, harmless changes, so it is shown next to PSI and the plain change in the average, and you judge them together.

Try it yourself

List A: 1, 2, 3, 4. List B: 3, 4, 5, 6. Find the biggest gap between the cumulative shares.

Answer: at the value 2, A is at 0.5 and B is at 0, so D = 0.5.

In Python

from scipy.stats import ks_2samp
result = ks_2samp([2, 3, 5, 7, 9], [4, 6, 8, 8, 10])
print(result.statistic, result.pvalue)   # 0.4  0.873...

Want lessons like this for your team or class?

I teach data science and AI in plain language, to students, staff and community groups. Invite me to run a session.