· 8 min read
One Split Can Lie: Honest Model Evaluation with Cross-Validation
The most common way to measure how good a machine learning model is: split the data once into training and test sets. It's fast, easy to understand and gives you one number. The problem is that the number can change a lot depending on which rows end up in the test set. In this post I show, with numbers I measured, how a single split misled me in my Brain Weight Prediction project, the trap I fell into when switching to cross-validation, and how I finally chose the model.
The project and the data
The project answers a simple question: given an adult's head size, roughly how much does their brain weigh? The data comes from a study published in 1905: head size, brain weight, sex and age group for 237 adults. The target is brain weight; its mean is 1,283 g and its standard deviation 120 g.
I use two metrics to evaluate models:
- Mean absolute error (MAE): how many grams the prediction is off on average. It's in grams and easy to interpret.
- R²: how much of the variation in brain weight is explained. 1 is perfect; 0 means "as good as always guessing the mean".
The number from a single split
The first version of the project took the classic route: train on 70% of the data, test on 30%.
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=0)
Linear regression gave R² 0.659 on this split, and the README reported "~0.658". The code was reproducible: random_state=0 produced the same split every time. But being reproducible isn't the same as being representative.
To see that, I ran the same model on the same data with 1,000 different random splits. The only thing that changed was random_state:
| R² | |
|---|---|
| Lowest | 0.352 |
| 5th percentile | 0.517 |
| Median | 0.634 |
| 95th percentile | 0.722 |
| Highest | 0.784 |
Same model, same data. Just changing which 71 people land in the test set moves R² anywhere from 0.35 to 0.78. The first version's 0.659 is a value that about a third of the splits (35%) reach: a bit above the median, on the lucky side. With 237 rows the test set holds 71 people, and whether a few unusual people land in it changes the result noticeably.
A single split is one number drawn at random from this distribution. There is no way to tell which end it came from.
Cross-validation: every row is tested once
K-fold cross-validation reduces this randomness. The data is cut into k parts and the model is trained k times; each time one part is the test set and the rest is training data. In the end every row has been in the test set exactly once, and the k scores are averaged.
When I first tried 10-fold cross-validation, the result was a surprise: R² 0.459. Much lower than the single split's 0.659. The model suddenly looked worse.
The trap: splitting sorted data without shuffling
The problem wasn't the model, it was the split. scikit-learn's KFold does not shuffle by default; it cuts the rows into parts in order. And my CSV file was sorted: men aged 20-46 first, then men over 46, then women.
I looked inside each test fold of the unshuffled split:
| Fold | Contents | R² |
|---|---|---|
| 1 | 24 men, 20-46 | 0.034 |
| 2 | 24 men, 20-46 | 0.443 |
| 3 | 15 men 46+, 9 men 20-46 | 0.499 |
| 4 | 24 men, 46+ | 0.432 |
| 5 | 24 men, 46+ | 0.548 |
| 6 | 14 men 46+, 10 women 20-46 | 0.573 |
| 7 | 24 women, 20-46 | 0.385 |
| 8 | 19 women 20-46, 4 women 46+ | 0.364 |
| 9 | 23 women, 46+ | 0.784 |
| 10 | 23 women, 46+ | 0.533 |
Most test folds come from a single group, and in one of them R² drops to 0.03.
There's an interesting detail here. I also measured the error in grams for the same two kinds of split:
| MAE | Std. dev. of brain weight in the test fold | |
|---|---|---|
| Without shuffling | 57.3 g | 101 g |
| With shuffling | 56.5 g | 117 g |
The predictions are almost equally good: the difference is under a gram. What pulls R² down is the test fold itself. In a fold made of a single sex and age group, brain weights are closer to each other. R² compares the error to that variation, so when the variation shrinks, the same error looks worse.
So the lesson is: cross-validation without shuffling measures the order of rows in the file, not how good the model is. Shuffling before splitting is not an optional setting. Shuffled 10-fold cross-validation gave R² 0.626.
I also learned that one metric isn't enough. R² alone said "the model collapsed"; MAE said "the predictions are the same, what you're measuring changed".
Repeated cross-validation and a fair comparison
Shuffling adds randomness of its own: one shuffle may be lucky and another unlucky. To remove that too, I repeated 10-fold cross-validation 5 times, each with a different shuffle. That's 50 splits in total.
def compare_models(
df: pd.DataFrame, n_splits: int = 10, n_repeats: int = 5, seed: int = 0
) -> list[Score]:
"""Bütün adayları (ve referans modeli) aynı bölmelerle değerlendirir."""
# Aynı random_state = her model birebir aynı eğitim/test bölmeleriyle sınanır
cv = RepeatedKFold(n_splits=n_splits, n_repeats=n_repeats, random_state=seed)
return [evaluate(c, df, cv) for c in [*CANDIDATES, BASELINE]]
(The comments say: "Evaluates all candidates and the baseline on the same splits" and "same random_state means every model is tested on exactly the same train/test splits".)
There are three decisions in this code:
- All models are tested on the same 50 splits. Because
random_stateis fixed, every candidate sees exactly the same training and test rows. No model gets ahead by landing on easier splits. - Preprocessing lives inside the model. Scaling and turning categories into 0/1 columns happen inside scikit-learn pipelines, so in every split these steps learn only from the training part; the test part's mean or spread never leaks into the model.
- There is a baseline. A "model" that always predicts the mean is tested on the same splits. How good the other models are only makes sense compared with it.
Results for the five candidates and the baseline, mean and standard deviation over 50 splits:
| Model | MAE (g) | R² |
|---|---|---|
| Linear (head size) | 56.7 ± 8.7 | 0.602 ± 0.128 |
| Linear (head size, sex, age group) | 56.2 ± 8.3 | 0.611 ± 0.122 |
| Polynomial, degree 2 (head size) | 57.0 ± 9.0 | 0.603 ± 0.127 |
| Ridge (head size, sex, age group) | 56.2 ± 8.3 | 0.612 ± 0.120 |
| Random Forest (head size, sex, age group) | 59.5 ± 10.1 | 0.551 ± 0.149 |
| Baseline (always the mean) | 94.5 ± 14.8 | −0.048 ± 0.055 |
The honest R² of the single-variable linear model is 0.602. That's lower than the 0.659 the first version reported, but this time 50 splits stand behind the number, and the ± 0.128 spread also explains why a single split was so unstable.
Which model? The one-standard-error rule
Ridge has the lowest MAE in the table: 56.19 g. The single-variable linear model is half a gram behind at 56.74 g. Should I pick Ridge?
Half a gram is tiny next to the uncertainty of the measurement itself. To make that precise I used the one-standard-error rule: take the best model's mean error, add the standard error of that mean to get a threshold, and among the models under the threshold pick the simplest.
def select_model(scores: list[Score]) -> tuple[Score, str]:
"""Bir standart hata kuralı: en iyiye yeterince yakın olan en basit modeli seçer."""
ranked = [s for s in scores if s.name != BASELINE.name]
best = min(ranked, key=lambda s: s.mae)
threshold = best.mae + best.mae_std / math.sqrt(best.folds)
# ranked basitlik sırasında olduğu için eşiği geçen ilk model en basitidir
chosen = next(s for s in ranked if s.mae <= threshold)
# ...
(The last comment says: "ranked is in order of simplicity, so the first model under the threshold is the simplest".)
Ridge's standard deviation is 8.3 g over 50 splits, so the standard error is 8.3 / √50 ≈ 1.17 g. The threshold is 56.19 + 1.17 = 57.36 g. The candidate list is ordered by simplicity: the first candidate is the single-variable linear model, and at 56.74 g it's under the threshold. That's the model that was chosen:
brain weight ≈ 325.6 + 0.2634 × head size
Adding sex and age group gains half a gram. That gain sits inside the measurement's uncertainty and isn't worth asking the user for two more inputs. Random Forest did worst on 237 rows: with this little data it's more prone to memorizing.
Summary
- A single train/test split is one number drawn at random. On this data the same model scored anywhere from R² 0.35 to 0.78 across 1,000 splits. The first version's 0.659 was on the lucky side.
- K-fold cross-validation puts every row in the test set once, but scikit-learn's
KFolddoesn't shuffle by default. Splitting a sorted file without shuffling dropped R² to 0.459. - That drop came from single-group test folds, not worse predictions: MAE was almost the same. Look at at least two metrics, not one.
- Repeated cross-validation, the same splits for every model, preprocessing inside pipelines and a baseline made the comparison fair.
- The one-standard-error rule gave a principled reason to pick the simplest model that is close enough to the best.
The calculator that runs the model in the browser is at brain.miracdeprem.com; its model page also shows the comparison table and why the model was chosen. The source code is on GitHub: github.com/MrcDprm/brain-weight-prediction.