Data, features and leakage

The opening note treated every object as a vector $x \in \mathbb R^d$ and every training row as an independent draw from the distribution the model will meet. Real data satisfy neither assumption on arrival. A table of loan applications has a district name, a date, an income that some applicants did not give, a free-text purpose, and columns filled in months after the application was decided. This note turns such columns into numbers a model can use, and shows the two ways that information about the target slips into the features and makes a useless model look excellent.

Notation. The object–feature matrix $X \in \mathbb R^{n \times d}$ has one row per object and one column per feature; $x_{ij}$ is feature $j$ of object $i$, and $y = (y_1, \dots, y_n)$ the targets. A categorical feature takes values $c$ in a finite set of $C$ categories; $n_c$ is the number of training rows in category $c$ and $\bar y_c$ their mean target; $\bar y$ is the overall mean.

What kinds of features there are#

Type	Example	What a model needs
Numeric	income, area, number of purchases	often a scale and a transform for skew
Binary	has a car	nothing: 0 or 1
Categorical (nominal)	district, product, browser	an encoding; the categories have no order
Ordinal	education level, rating 1–5	the order, but not necessarily equal steps
Date and time	application date	parts (hour, weekday, month) and differences (“days since”)
Text, sets	purpose of a loan, tags	a vector, for example word counts

The type decides what arithmetic is meaningful. The mean of two districts is not a district, and coding districts as 1, 2, 3 tells a linear model that district 2 lies halfway between 1 and 3. Text becomes a vector of counts, the bag of words; the NLP course builds it in the text as data and bag of words notes, and the resulting matrix is an object–feature matrix with tens of thousands of sparse columns.

Encoding categories#

One-hot encoding replaces a feature with $C$ categories by $C$ binary columns, one per category, with a single 1 in each row. A linear model then learns one weight per category, which is exactly the freedom a nominal feature should have. With an intercept in the model, the $C$ columns always sum to 1 and duplicate the intercept, so one column is usually dropped. Without a penalty this is needed for a unique fit: the $C$ columns and the intercept are linearly dependent, and infinitely many weight vectors give the same predictions. With a ridge or lasso penalty all $C$ columns can be kept, because the penalty picks one solution; dropping a column then changes the fit, since the remaining weights are shrunk toward the dropped reference category rather than toward a common value. One-hot works well up to a few hundred categories; for a feature with 100,000 values (a user, a shop, a postcode) the matrix becomes huge and most categories have a handful of rows, so their weights are estimated from almost nothing.

Target encoding replaces each category by one number: an estimate of the mean target in that category, $E[Y \mid \text{category} = c]$ (Micci-Barreca, 2001). For a rare category the raw mean $\bar y_c$ is noisy, so it is shrunk toward the overall mean:

$$g(c) = \frac{n_c \bar y_c + a\, \bar y}{n_c + a} ,$$

with a prior weight $a > 0$ that acts like $a$ extra rows at the overall mean. For a binary target this is the posterior mean of the beta–binomial model with the prior $\mathrm{Beta}(a\bar y,\ a(1 - \bar y))$, centred at $\bar y$. For a numeric target the same formula is a general shrinkage estimator; its Bayesian reading is the normal model, a normal prior on the category mean centred at $\bar y$ with normal noise, where $a$ is the ratio of the noise variance to the prior variance. Either way, a category with many rows keeps its own mean, a rare one stays near the prior. Worked example. A category with 3 rows, 2 of them positive, in data with an overall positive rate of 0.2, encoded with $a = 10$: $g = (2 + 10 \cdot 0.2)/(3 + 10) = 4/13 = 0.308$, rather than the raw $2/3$.

Why the naive target encoding leaks#

Computed on the training rows, $\bar y_c$ includes each row’s own target. In the extreme, a category that appears once is encoded by its own label, and a model that reads the encoding is reading the answer. Prokhorenkova et al. (2018), Section 3.2, give the clean version: if every category is unique and the label is a fair coin, the encoded training rows are separated perfectly by one threshold, while every test row receives the prior and the model scores 0.5. The distribution of the encoded feature given the label differs between training and test rows, which is precisely what a model must not be trained on. Leaving only the row itself out does not fix it: for a single category shared by all rows, the leave-one-out mean $(\sum_j y_j - y_i)/(n - 1)$ takes one value for positive rows and another for negative ones, and again one threshold separates them (same section).

The fix is to encode each training row with statistics computed on other rows, as in cross-validation. Out-of-fold encoding splits the training set into folds and encodes the rows of each fold with the means of the remaining folds; test rows get the means of the whole training set. Eight rows, two categories, two folds, no smoothing:

Row	Category	$y$	Fold	Naive $g$	Out-of-fold $g$
1	A	1	1	0.75	1.0
2	A	0	1	0.75	1.0
3	A	1	2	0.75	0.5
4	A	1	2	0.75	0.5
5	B	0	1	0.25	0.5
6	B	1	2	0.25	0.0
7	B	0	2	0.25	0.0
8	B	0	1	0.25	0.5

Rows 1 and 2 are encoded with rows 3 and 4 (both positive), rows 5 and 8 with rows 6 and 7. The out-of-fold values are noisier, which is the honest price of not looking at the row’s own label and the reason they are smoothed in practice. CatBoost uses an ordered variant: it shuffles the rows and encodes each one with the rows before it, so every row is encoded from data it could have seen in a stream (Prokhorenkova et al., 2018, Section 3.2).

On data where the category carries no information at all (2,000 rows, 500 categories of about 4 rows, random labels), a logistic regression on the naive encoding reaches a cross-validated ROC-AUC of 0.774; with the out-of-fold encoding fitted inside each training fold, it gets 0.48, the chance level it deserves (the Code section).

Scaling#

A feature measured in roubles and a feature measured in years differ in spread by a factor of thousands, and several methods treat numbers as if their units were comparable.

• Distances. Two applicants with incomes 50,000 and 52,000 and ages 25 and 60 are at Euclidean distance $\sqrt{2000^2 + 35^2} = 2000.3$: the age difference contributes almost nothing. In the wine data bundled with scikit-learn, the standard deviation of the proline content is 314 and that of the nonflavanoid phenols 0.12; a 5-nearest-neighbour classifier on the raw columns has a cross-validated accuracy of 0.663, and on standardized columns 0.961. The nearest-neighbours note returns to this.
• Gradient descent. For least squares with two independent centred features of standard deviations 1 and 100, the loss surface is an ellipse whose axes differ by the ratio of the scales, and the condition number of $X^\top X$ is about $10^4$; after standardization it is 1.02. The number of steps gradient descent needs grows with the condition number, as the gradient descent note shows.
• Penalties. Regularization penalizes all coefficients alike, so a feature on a large scale (with a small coefficient) is penalized less than the same information on a small scale.

Standardization subtracts the mean of each column and divides by its standard deviation; min–max scaling maps each column to $[0, 1]$; a robust version uses the median and the interquartile range so that a few extreme values do not set the scale. Trees split on thresholds of one feature at a time and are unaffected by any monotone transformation of a feature. Scaling is not always an improvement: when all features already share one unit, such as pixel intensities, standardizing inflates the noise of nearly constant columns. Whatever the method, its means and spreads are statistics of the data, and they must be computed on the training rows only, a point the section on contamination makes precise.

Missing values#

A value can be missing for reasons that matter. Rubin (1976) distinguishes values missing completely at random (a sensor dropout unrelated to anything), missing at random given the observed features, and missing not at random, where the missingness depends on the missing value itself (high earners declining to report their income). In the last case the fact that a value is missing is itself informative.

Most models need a number in every cell. The standard recipe fills the gap with a statistic of the training column (the median for skewed features, the most frequent category for categorical ones) and adds a binary missing indicator. For a linear model $w_0 + w_1 x_1 + w_2 [x_1 \text{ missing}] + \dots$ the indicator’s weight shifts the prediction for missing rows, so the model in effect learns the best fill value for itself, whatever constant was put in the cell. Gradient boosting libraries handle missing values natively by sending them down whichever branch of a split reduces the loss more.

Feature engineering#

Models find patterns in the features they are given; the features decide which patterns are easy to find. Typical constructions on tabular data:

• Ratios and differences: debt divided by income, price per square metre, days between registration and the first purchase.
• Parts of dates: hour of day, day of week, month, a holiday flag; demand and fraud both have strong weekly cycles.
• Transforms of skewed amounts: $\log(1 + x)$ for incomes and purchase sums, so that a linear model sees proportional rather than absolute differences.
• Aggregates over history: the number of purchases in the last 30 days, the mean basket size, the share of returned orders. Each must be computed as of the moment of prediction, using only events before it.
• Interactions: the product of two features, or a category crossed with another, for models that cannot form them themselves. Trees and neural networks form interactions on their own, which is one reason they need less engineering.

Target leakage#

Target leakage is a feature that carries information about the target that will not be available at prediction time (Kaufman et al., 2012). Typical forms:

• Computed after the event. Predicting loan default with “number of calls from the collections department”, or hospital readmission with a diagnosis code entered at the readmission. The feature exists in the historical table because the outcome already happened.
• A disguised copy of the target. “Total amount repaid” when predicting whether a loan is repaid; “account closed date is filled in” when predicting churn.
• Aggregates that include the future. “Mean spend of the customer” computed over the whole history, including the months after the prediction date.
• Artefacts of collection. Rows sorted by outcome so that the row identifier predicts the label, or positive examples gathered from a different source with a different file format.

The symptom is a model that is far better than anyone expected, with one or two features doing most of the work. The cure is a timestamp for every feature and the rule that a feature may only use information from before the moment of prediction. Kapoor and Narayanan (2023) found leakage in published work in 17 scientific fields, affecting 294 papers; in their re-analysis of civil-war prediction, the complex models lost their advantage over logistic regression once the leakage was removed.

Train–test contamination#

The second kind of leakage does not need a bad feature. Contamination happens when any step of the pipeline that is fitted to data, not only the model, sees the rows used for evaluation:

• a scaler or an imputer whose means were computed on all rows;
• a vocabulary, a TF-IDF weighting or a word-embedding model trained on all texts, as the NLP evaluation note discusses for text;
• a target encoding computed on all rows;
• feature selection on all rows: keeping the features most correlated with the label, then cross-validating a model on them;
• duplicates or near-duplicates of the same object in the training and the evaluation set.

How much harm a step does depends on how much it learns about the target. A scaler learns only means and spreads; on the wine data, standardizing all rows before cross-validation gives the same accuracy as standardizing inside each training fold, 0.961 in both cases. Feature selection learns which features agree with the labels of all rows, the evaluation rows included, and the effect is dramatic. The classic demonstration uses data with no signal at all (Hastie, Tibshirani and Friedman, 2009, Section 7.10.2; Ambroise and McLachlan, 2002):

Figure

Feature selection outside cross-validation finds signal in pure noise

With 1,000 noise features and 50 objects, some features agree with the labels by chance, and selecting them on all rows picks exactly the chance agreements that the held-out folds will then “confirm”. The leak grows with the number of features selected, up to an estimated accuracy of 0.99 for 200 features, and disappears when all 1,000 are kept, because then nothing was selected. Repeating the selection inside each training fold removes it completely: the estimate stays at the true 0.5. The Code section repeats the experiment with 10,000 noise features, the 20 best of them and a logistic regression: 0.87 when selected on all rows, 0.49 inside the folds.

The practical rule is to express every data-dependent step as part of the model, a pipeline fitted on the training part of each split and only applied to the evaluation part. In scikit-learn this is make_pipeline(StandardScaler(), SelectKBest(...), LogisticRegression()) passed whole to the cross-validation function. The same rule covers the person tuning the model: choosing features by looking at test-set results is feature selection on the test set.

What changed#

	Raw table	Prepared features
Categories	strings or arbitrary codes	one-hot columns, or a target encoding computed out of fold
Scales	units of measurement	standardized where distances, gradients or penalties are used
Missing values	empty cells	filled from training statistics, plus an indicator
Time	a date column	parts, differences, history aggregates as of the prediction time
Evaluation	any split of the final table	every fitted step inside the training fold; features restricted to information available at prediction time

Limitations#

• Engineering is problem-specific. No recipe says which ratios or aggregates matter; domain knowledge and validation decide, and every feature tried is one more chance to overfit the validation set.
• Leakage cannot be detected from the table alone. Whether a column was filled in before or after the outcome is a fact about the process that produced the data, not about the numbers.
• Out-of-fold encodings cost variance. They are noisier than the naive ones and need smoothing; with very rare categories, one-hot with regularization or hashing may be better.
• Missing-data assumptions are untestable. Data missing not at random cannot be distinguished from data missing at random using the observed values alone.

What you should remember#

• Feature types decide what arithmetic is valid; categories need an encoding, dates need parts and differences, text needs a vector.
• One-hot gives each category its own weight; target encoding replaces a category by its smoothed mean target $\frac{n_c \bar y_c + a \bar y}{n_c + a}$, a posterior mean for a binary target and a shrinkage estimate in general.
• The naive target encoding leaks the row’s own label, and so does leave-one-out; compute it out of fold or in a random order.
• Scale features for distance-based models, gradient descent and penalties; trees do not care.
• Fill missing values from training statistics and add an indicator; missingness can be informative.
• Target leakage: a feature unavailable at prediction time. Contamination: a fitted step that saw the evaluation rows. Both almost always make estimates optimistic; feature selection on all rows can turn pure noise into 99% accuracy.
• Put every fitted step into a pipeline that is refitted inside each training fold.

Code#

Target encoding smoothed and out of fold, feature selection inside and outside cross-validation, and scaling. The printed numbers come from NumPy 2.1.3 and scikit-learn 1.9.1. Passing a splitter as cv to TargetEncoder needs scikit-learn 1.9 or later; on versions 1.3 to 1.8, TargetEncoder(cv=5, shuffle=True, random_state=0) does the same out-of-fold encoding with stratified internal folds (its AUC is 0.493 under scikit-learn 1.9.1):

import numpy as np
from sklearn.datasets import load_wine
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, TargetEncoder

rng = np.random.default_rng(0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

# smoothed target encoding of a category with 3 objects, 2 positive, global rate 0.2, prior weight 10
print(round((2 + 10 * 0.2) / (3 + 10), 3))                                    # 0.308

# pure noise: 100 objects, 10,000 features, labels independent of everything; keep the 20 "best" features
X = rng.normal(size=(100, 10_000))
y = rng.permutation(np.repeat([0, 1], 50))
X20 = SelectKBest(f_classif, k=20).fit_transform(X, y)                       # selected on all 100 rows: leak
wrong = cross_val_score(LogisticRegression(), X20, y, cv=cv).mean()
right = cross_val_score(make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression()), X, y, cv=cv).mean()
print(round(wrong, 3), round(right, 3))                                       # 0.87 0.49

# target encoding of a noise category with 500 values (about 4 rows each), 2,000 rows, random labels
cat = rng.integers(0, 500, size=(2000, 1))
yb = rng.integers(0, 2, size=2000)
counts, sums = np.bincount(cat[:, 0], minlength=500), np.bincount(cat[:, 0], weights=yb, minlength=500)
naive = (sums / np.maximum(counts, 1))[cat]                                   # mean label of the category, own row included
auc_naive = cross_val_score(LogisticRegression(), naive, yb, cv=cv, scoring='roc_auc').mean()
oof = make_pipeline(TargetEncoder(cv=KFold(5, shuffle=True, random_state=0)), LogisticRegression())   # a splitter as cv: scikit-learn >= 1.9
auc_oof = cross_val_score(oof, cat, yb, cv=cv, scoring='roc_auc').mean()      # out-of-fold encoding inside each fold
print(round(auc_naive, 3), round(auc_oof, 3))                                 # 0.774 0.48

# scaling: kNN on the wine data, raw, standardized on all rows, standardized inside each training fold
Xw, yw = load_wine(return_X_y=True)
print(Xw.std(axis=0).max().round(2), Xw.std(axis=0).min().round(2))         # 314.02 0.12 (proline, nonflavanoid phenols)
raw = cross_val_score(KNeighborsClassifier(5), Xw, yw, cv=cv).mean()
outside = cross_val_score(KNeighborsClassifier(5), StandardScaler().fit_transform(Xw), yw, cv=cv).mean()
inside = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier(5)), Xw, yw, cv=cv).mean()
print(round(raw, 3), round(outside, 3), round(inside, 3))                     # 0.663 0.961 0.961

# the condition number of X^T X for two independent features on scales 1 and 100, raw and standardized
Z = rng.normal(size=(10_000, 2)) * [1, 100]
Zs = StandardScaler().fit_transform(Z)
print(f'{np.linalg.cond(Z.T @ Z):.2g}', f'{np.linalg.cond(Zs.T @ Zs):.3g}')   # 1e+04 1.02

Next#

Leakage and contamination are failures of the evaluation: the model is tested on data it has, in some way, already seen. Avoiding them is necessary but not sufficient. Even with a clean split, a validation score is a random number with an error bar, a model chosen as the best of many looks better than it is, and the way the data are split must match the way the model will be used. The next note builds the evaluation protocol: hold-out sets, cross-validation, group and time splits, and nested cross-validation for choosing models.

References#

• E. Sokolov et al. Машинное обучение (HSE), practical session 4: data preprocessing and lecture 1: introduction.
• D. Micci-Barreca. A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems. ACM SIGKDD Explorations, 3(1), 2001.
• L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, A. Gulin. CatBoost: unbiased boosting with categorical features. NeurIPS, 2018. Section 3.2 (target statistics: greedy, holdout, leave-one-out and ordered).
• D. B. Rubin. Inference and missing data. Biometrika, 63(3), 1976.
• S. Kaufman, S. Rosset, C. Perlich, O. Stitelman. Leakage in data mining: formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), 2012.
• S. Kapoor, A. Narayanan. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 2023.
• C. Ambroise, G. J. McLachlan. Selection bias in gene extraction on the basis of microarray gene-expression data. PNAS, 99(10), 2002.
• T. Hastie, R. Tibshirani, J. Friedman. The Elements of Statistical Learning, 2nd ed. Springer, 2009. Section 7.10.2 (the wrong and right way to do cross-validation).