Dominion Akinrotimi
Home / Learn / Lesson 1
Lesson 1 of 3

Spotting a number that does not belong

Quartiles, the interquartile range and fences, with nothing harder than sorting and subtracting.

By Dominion Akinrotimi · Data Quality, Explained Simply

Imagine your class teacher writes down everyone's age. One age says 99. You know straight away that is a mistake: nobody in your class is 99. But a computer looking at thousands of numbers cannot "just know". It needs a rule. This lesson teaches the rule PyDataQuality uses, and you only need to know how to sort numbers and subtract.

Step 1: sort the numbers

Here are the ages of 11 children:

9, 10, 10, 10, 11, 12, 12, 13, 13, 14, 99Already sorted from smallest to biggest.

Step 2: find the middle, and the middle of each half

The median is the number in the very middle. With 11 numbers, the 6th one is in the middle: 12.

Now split the list into a lower half and an upper half (leave the middle number out), and find the middle of each half:

In plain wordsA quarter of the children are 10 or younger, and a quarter are 13 or older. The middle half of the class is between 10 and 13.

Step 3: measure how wide the middle half is

IQR = Q3 − Q1 = 13 − 10 = 3IQR means interquartile range: how spread out the middle half is.

Why the middle half? Because it ignores the extremes. If we used the smallest and biggest numbers, the 99 would stretch everything and hide itself. The middle half is not fooled by one silly value.

Step 4: build fences

The classic rule, invented by the statistician John Tukey, puts a fence one and a half IQRs outside the middle half:

low fence = Q1 − 1.5 × IQR = 10 − 4.5 = 5.5
high fence = Q3 + 1.5 × IQR = 13 + 4.5 = 17.5Anything outside the fences is called an outlier.

99 is far above 17.5, so it is flagged. Every other age is inside the fences. That is exactly what you knew by looking, but now a computer can do it on a million rows in a blink.

Step 5: how far outside is it?

PyDataQuality goes one step further and asks how many IQRs outside the middle half a number sits:

distance = (99 − 13) ÷ 3 ≈ 28.799 sits almost 29 "middle-half widths" above the middle half.

The tool uses wider fences than Tukey's 1.5 on purpose. Real data has plenty of honest unusual values, and a tool that complains about all of them gets ignored.

Two clever details

Long tails are normal. In a list of salaries, a few people earn far more than everyone else, and that is real, not an error. So for numbers that are never negative, PyDataQuality checks again on a "shrunk" scale (the logarithm, which turns 10, 100 and 1,000 into evenly spaced steps). A value is only flagged if it looks odd on both scales.

Small lists are left alone. With only a handful of numbers, you cannot really say what "usual" looks like, so the check waits until a column has at least 20 numbers.

Try it yourself

Shoe sizes of 11 friends: 36, 37, 37, 38, 38, 39, 39, 40, 41, 42, 380. Find Q1, Q3, the IQR and the fences. Which value gets flagged, and what mistake do you think someone made?

Answer: Q1 = 37, Q3 = 41, IQR = 4, fences 31 and 47. 380 is flagged. Someone probably typed an extra zero.

In Python

import pandas as pd
ages = pd.Series([9, 10, 10, 10, 11, 12, 12, 13, 13, 14, 99])
q1, q3 = ages.quantile(0.25), ages.quantile(0.75)
iqr = q3 - q1
print(ages[(ages < q1 - 1.5 * iqr) | (ages > q3 + 1.5 * iqr)])   # 99

Or let PyDataQuality do it, with the explanation in a sentence: pdq check ages.csv.

Want lessons like this for your team or class?

I teach data science and AI in plain language, to students, staff and community groups. Invite me to run a session.