Imagine your class teacher writes down everyone's age. One age says 99. You know straight away that is a mistake: nobody in your class is 99. But a computer looking at thousands of numbers cannot "just know". It needs a rule. This lesson teaches the rule PyDataQuality uses, and you only need to know how to sort numbers and subtract.
Step 1: sort the numbers
Here are the ages of 11 children:
Step 2: find the middle, and the middle of each half
The median is the number in the very middle. With 11 numbers, the 6th one is in the middle: 12.
Now split the list into a lower half and an upper half (leave the middle number out), and find the middle of each half:
- Lower half: 9, 10, 10, 10, 11. Its middle is 10. This is called Q1, the first quartile.
- Upper half: 12, 13, 13, 14, 99. Its middle is 13. This is Q3, the third quartile.
Step 3: measure how wide the middle half is
Why the middle half? Because it ignores the extremes. If we used the smallest and biggest numbers, the 99 would stretch everything and hide itself. The middle half is not fooled by one silly value.
Step 4: build fences
The classic rule, invented by the statistician John Tukey, puts a fence one and a half IQRs outside the middle half:
high fence = Q3 + 1.5 × IQR = 13 + 4.5 = 17.5Anything outside the fences is called an outlier.
99 is far above 17.5, so it is flagged. Every other age is inside the fences. That is exactly what you knew by looking, but now a computer can do it on a million rows in a blink.
Step 5: how far outside is it?
PyDataQuality goes one step further and asks how many IQRs outside the middle half a number sits:
- More than 3 widths away, and rare: a gentle note. It might still be correct.
- More than 10 widths away: a warning. That is usually a typing or unit mistake, like 99 instead of 9, or kilograms typed as grams.
The tool uses wider fences than Tukey's 1.5 on purpose. Real data has plenty of honest unusual values, and a tool that complains about all of them gets ignored.
Two clever details
Long tails are normal. In a list of salaries, a few people earn far more than everyone else, and that is real, not an error. So for numbers that are never negative, PyDataQuality checks again on a "shrunk" scale (the logarithm, which turns 10, 100 and 1,000 into evenly spaced steps). A value is only flagged if it looks odd on both scales.
Small lists are left alone. With only a handful of numbers, you cannot really say what "usual" looks like, so the check waits until a column has at least 20 numbers.
Try it yourself
Shoe sizes of 11 friends: 36, 37, 37, 38, 38, 39, 39, 40, 41, 42, 380. Find Q1, Q3, the IQR and the fences. Which value gets flagged, and what mistake do you think someone made?
Answer: Q1 = 37, Q3 = 41, IQR = 4, fences 31 and 47. 380 is flagged. Someone probably typed an extra zero.
In Python
import pandas as pd
ages = pd.Series([9, 10, 10, 10, 11, 12, 12, 13, 13, 14, 99])
q1, q3 = ages.quantile(0.25), ages.quantile(0.75)
iqr = q3 - q1
print(ages[(ages < q1 - 1.5 * iqr) | (ages > q3 + 1.5 * iqr)]) # 99
Or let PyDataQuality do it, with the explanation in a sentence: pdq check ages.csv.
Want lessons like this for your team or class?
I teach data science and AI in plain language, to students, staff and community groups. Invite me to run a session.