The problem: a logistics company wants to know at order time whether a delivery will arrive early, on time or late, so it can warn customers before they complain.
What I built: a model trained on 15,549 orders that uses only what's known when the order is placed, compared fairly against a simple baseline.
Results
Macro F1 on 3,110 held-out orders. Each outcome counts equally.
| Model | Accuracy | Macro F1 |
|---|---|---|
| Always predict "late" (baseline) | 57.7% | 0.244 |
| Logistic Regression | 53.4% | 0.469 |
| Random Forest | 54.2% | 0.462 |
| Hist Gradient Boosting | 52.8% | 0.477 |
Accuracy is misleading here. 58% of orders are late, so always guessing "late" scores 58% while never finding an early or on-time order. On macro F1 the models roughly double the baseline.
Where it's useful: when the model flags an order as late, it's right 87% of the time, which is precise enough to trigger a proactive delay message. It also catches 86% of early arrivals. On-time orders remain hard to tell apart.
An honest correction
My first version reported 71.7% accuracy. That number came from oversampling (SMOTE) the whole dataset before splitting it, so about 4 in 10 test rows were synthetic copies built from training rows. It also used order and customer IDs as features. This rebuild balances classes inside training only, drops IDs and the shipping date (known only after the fact), and compares every model with a baseline.
What drives delays
Shipping mode carries almost all of the signal. Shuffling it drops macro F1 by 0.17, and no other feature drops it by more than 0.01. Better predictions would need carrier, warehouse and route data.
Confusion matrix for the best model.