The problem: fact-checkers can't read every article. A first-pass filter that flags likely misinformation lets them spend their time where it counts.
What I built: a classifier trained on 6,306 labelled articles (half fake, half real) that reads the title and body text and predicts which kind it is.
Results
| Model (tuned) | Accuracy | Macro F1 |
|---|---|---|
| Logistic Regression | 92.2% | 0.922 |
| Random Forest | 90.5% | 0.905 |
| Naive Bayes | 88.7% | 0.887 |
Scored once on 1,262 held-out articles. The best model catches 92.9% of fake articles, and 91.7% of the articles it flags as fake really are fake.
How it works
- Text cleaned and lemmatised (NLTK), stop words and digits removed
- TF-IDF with 10,000 unigram and bigram features, fitted on training articles only
- Each model tuned by grid search with 5-fold cross-validation on the training set, scored by macro F1
- Stratified 80/20 split, so the test articles were never seen during tuning
Limitations
- The articles mostly come from the 2016 US election period, so the model learns that period's topics and outlets and needs retraining for other news.
- TF-IDF judges wording, not facts. It flags the style of misinformation and can't check whether a claim is true.