Classification metrics

The logistic regression note produced probabilities, and the classifiers of the linear classifiers note produce scores. Two questions remain. Where should the threshold go that turns a score into a decision? And how good is the classifier? Accuracy, the share of correct answers, is the obvious answer to the second question and fails exactly where classification matters most: fraud, disease, defects and churn are rare, and a model that never flags anything is right almost always. This note builds the metrics that see the errors accuracy hides, shows how costs fix the threshold, and proves the identity that makes the area under the ROC curve a ranking metric.

Notation. A classifier gives each example a score $s(x)$, larger for “more positive” (a probability, a logit, a margin). A threshold $t$ turns it into a decision, $\hat y = [s(x) \ge t]$, the convention of scikit-learn’s roc_curve and precision_recall_curve. Labels are $y \in \{0, 1\}$ with 1 the positive class, the one being looked for. A test set has $n_+$ positives and $n_-$ negatives, and $\pi = n_+/(n_+ + n_-)$ is the prevalence, the share of positives.

The confusion matrix#

At a fixed threshold every test example falls into one of four cells:

	predicted positive, $\hat y = 1$	predicted negative, $\hat y = 0$
actual positive, $y = 1$	true positive (TP)	false negative (FN)
actual negative, $y = 0$	false positive (FP)	true negative (TN)

Every threshold metric is a function of these four counts. With the class sizes fixed ($\mathrm{TP} + \mathrm{FN} = n_+$, $\mathrm{FP} + \mathrm{TN} = n_-$), only two of them are free, so it takes two numbers to describe a classifier at one threshold: one for each kind of error.

Worked example. The screening test of the probability course detects a disease in 90% of the sick and gives a false alarm to 9% of the healthy, and 1% of people are sick. Among 10,000 people, 100 are sick, so $\mathrm{TP} = 90$ and $\mathrm{FN} = 10$; of the 9,900 healthy, $\mathrm{FP} = 891$ and $\mathrm{TN} = 9{,}009$. The test is right about $(90 + 9{,}009)/10{,}000 = 91.0\%$ of people. A “test” that declares everybody healthy is right about 99%.

Accuracy and the imbalance trap#

Accuracy is $(\mathrm{TP} + \mathrm{TN})/n$, and splitting the count by true class shows what it weighs:

$$\text{accuracy} = \pi \cdot \frac{\mathrm{TP}}{n_+} + (1 - \pi) \cdot \frac{\mathrm{TN}}{n_-} .$$

It is an average of the success rates on the two classes, weighted by prevalence. When $\pi = 0.01$, performance on the positives contributes one hundredth of the number. Any accuracy must therefore be read against the majority baseline, $\max(\pi, 1 - \pi)$, the accuracy of always predicting the larger class.

The trap also works the other way: requirements that sound modest in terms of the rare class correspond to accuracies that sound absurd. Among 100,000 transactions with 2% fraud, a detector that finds 90% of the 2,000 frauds ($\mathrm{TP} = 1{,}800$, $\mathrm{FN} = 200$) and is right 90% of the time it raises an alarm ($\mathrm{FP} = 200$) makes 400 errors, an accuracy of 99.6%. In the Code section, logistic regression on a dataset with 5.9% positives reaches an accuracy of $0.9419$ at the default threshold $0.5$, against a majority baseline of $0.9409$: it finds 30 of 443 positives.

Precision, recall and F1#

Two rates describe the positive class:

$$\text{precision} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}} = P(y = 1 \mid \hat y = 1), \qquad \text{recall} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}} = P(\hat y = 1 \mid y = 1) .$$

Precision asks how many of the alarms are real; recall, also called sensitivity or the true positive rate (TPR), how many of the real cases raise an alarm. They condition on different events, and the difference is the base-rate effect of Bayes’ formula. Recall is a property of the classifier on the positives alone. Precision mixes in the negatives and therefore the prevalence: with the false positive rate $\mathrm{FPR} = \mathrm{FP}/n_-$,

$$\text{precision} = \frac{\pi \cdot \text{recall}}{\pi \cdot \text{recall} + (1 - \pi) \cdot \mathrm{FPR}} .$$

For the screening test, recall is $0.9$ but precision is $0.9 \cdot 0.01/(0.9 \cdot 0.01 + 0.09 \cdot 0.99) = 90/981 = 0.092$: nine alarms in ten are false.

Each of the two is easy to make perfect alone: flag everything and recall is 1, flag only the single most certain case and precision is usually 1. The F1 score combines them by the harmonic mean,

$$F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} = \frac{2\,\mathrm{TP}}{2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN}} ,$$

which stays close to the smaller of the two; the screening test has $F_1 = 0.167$. The weighted version counts recall $\beta$ times as important as precision (van Rijsbergen, 1979):

$$\begin{aligned} F_\beta &= \frac{(1 + \beta^2) \cdot \text{precision} \cdot \text{recall}}{\beta^2 \cdot \text{precision} + \text{recall}} \\ &= \frac{(1 + \beta^2)\,\mathrm{TP}}{(1 + \beta^2)\,\mathrm{TP} + \beta^2\,\mathrm{FN} + \mathrm{FP}} . \end{aligned}$$

$\beta = 1$ gives $F_1$, $\beta = 2$ favours recall and $\beta = 0.5$ precision; the screening test has $F_2 = 450/1381 = 0.326$. The counting form is defined unless $\mathrm{TP} = \mathrm{FP} = \mathrm{FN} = 0$, a set with no positives and no alarms, where scikit-learn returns the value of its zero_division argument. F1 ignores the true negatives entirely, which is what makes it useful when they are plentiful and uninteresting, and misleading when they are not. With several classes these metrics are computed per class and averaged, macro or micro, as the NLP evaluation note shows for text and the multiclass note develops.

Choosing the threshold#

The confusion matrix belongs to one threshold; the classifier is the whole family of thresholds. Raising $t$ flags fewer examples: recall can only fall, while precision usually rises, though not monotonically, since every negative that drops out raises it and every positive that drops out lowers it. Where to stop is a decision about the application, made on a validation set and never on the test set (validation note):

• A recall floor. Screening for a disease: find at least 95% of cases, and among the thresholds that do, take the one with the best precision.
• A precision floor. Alerts that a person must read: at least half of them must be real.
• A budget. A call centre can make 800 calls among 4,000 customers: rank by score and call the top 800, so the threshold is the 800th highest score and the metric is the precision among the top $k$, precision@k. If several customers share that score, $[s \ge t]$ selects more than 800, and a tie-break, at random or by a second key, decides who is called.
• The best F1. When nothing else is specified. For calibrated probabilities the F1-maximizing threshold equals half of the maximal F1 (Lipton, Elkan and Narayanaswamy, 2014), so it is never above 0.5 and is low when the task is hard.

In the Code section, the threshold that maximizes F1 on the validation set is $0.152$ (half the validation F1 is $0.175$, close, as the model is only approximately calibrated). On the test set it raises F1 from $0.121$ at the default $0.5$ to $0.339$.

Costs decide the threshold#

When the consequences of the two errors can be priced, the threshold follows from the probabilities. Let a false negative cost $c_{\mathrm{FN}}$, a false positive $c_{\mathrm{FP}}$, and correct decisions nothing. For an example with calibrated probability $p = P(y = 1 \mid x)$, flagging it costs $(1 - p)\, c_{\mathrm{FP}}$ in expectation and passing it costs $p\, c_{\mathrm{FN}}$. Flag it when the first is smaller:

$$p > t^* = \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}} + c_{\mathrm{FN}}}$$

(Elkan, 2001). Equal costs give $t^* = 0.5$, the default; anything else moves it. If a missed fraud costs 500 and an unnecessary review of a legitimate transaction 5, then $t^* = 5/505 \approx 0.0099$: review every transaction with a fraud probability above 1%. This counts a review that catches a fraud as free. If instead every review costs 5, flagging costs 5 whatever $p$ is, passing costs $500p$, and the threshold is $5/500 = 0.01$, almost the same. In the Code section this rule cuts the total test cost from 206,615 at the threshold 0.5, and 127,520 at the F1-optimal threshold, to 36,435. The rule needs calibrated probabilities, which is where the previous note pays off; with raw scores the threshold must be searched on validation data.

Imbalance is not a problem in itself. A calibrated model with the cost threshold already handles rare classes. Resampling the training set or weighting the classes (class_weight in scikit-learn) changes the probabilities the model learns rather than the decision rule: with weight $c$ on the positives, the minimizer of the weighted log-loss at a point is $c\eta/(c\eta + 1 - \eta)$ instead of $\eta$, and the threshold 0.5 on it is the threshold $1/(1 + c)$ on $\eta$. Weighting is thus another way to move the threshold, at the price of probabilities that are no longer calibrated; it can still help when the model’s capacity is better spent on the rare class.

The ROC curve#

Figure

A threshold on the scores: what ROC and precision see

As $t$ falls from $+\infty$ to $-\infty$, the classifier moves from flagging nothing to flagging everything. The ROC curve (receiver operating characteristic) plots its recall against its false positive rate along the way, from $(0, 0)$ to $(1, 1)$. Both coordinates are conditional on the true class, as the upper panel of the figure shows: they are areas under the two class-conditional score distributions, and neither involves the prevalence.

On a finite test set the curve is a staircase. Sort the examples by decreasing score and walk down the list: each positive moves the curve up by $1/n_+$, each negative moves it right by $1/n_-$ (a tie between a positive and a negative moves it diagonally). A perfect ranking climbs to $(0, 1)$ before moving right; a random ranking stays near the diagonal; a curve below the diagonal becomes one above it when the scores are negated.

Worked example. Eight test examples, sorted by score:

Score	0.9	0.8	0.7	0.6	0.4	0.3	0.2	0.1
Label	1	1	0	1	0	1	0	0

With $n_+ = n_- = 4$ the staircase goes up, up, right, up, right, up, right, right. The area under it is a sum of four vertical strips of width $1/4$, one under each step to the right, with heights $2/4, 3/4, 4/4, 4/4$: the area is $(2 + 3 + 4 + 4)/16 = 13/16 = 0.8125$.

Two properties follow from the construction. The curve depends only on the order of the scores, so any strictly increasing transformation, Platt’s calibration map included, leaves it unchanged (a map that is only non-decreasing, such as isotonic regression, can create ties, which count $\tfrac12$ in the AUC below): the ROC curve measures ranking, not probabilities. And it does not depend on the prevalence, which is its strength when the prevalence of deployment is unknown and its weakness when positives are rare, as the lower panel of the figure already suggests.

Costs on the ROC plane. The expected cost per example at an operating point is $c_{\mathrm{FN}}\, \pi\, (1 - \mathrm{TPR}) + c_{\mathrm{FP}}\, (1 - \pi)\, \mathrm{FPR}$. Points of equal cost lie on straight lines of slope $c_{\mathrm{FP}}(1 - \pi)/(c_{\mathrm{FN}}\, \pi)$, and the best threshold is where the highest such line touches the curve (Fawcett, 2006). Expensive misses or common positives make the lines flat and move the best point up and to the right; rare positives or expensive false alarms make them steep and move it towards the origin.

AUC is the probability of ranking a positive above a negative#

Theorem. For scores $s_1, \dots, s_n$ on a test set with positives $\mathcal P$ and negatives $\mathcal N$,

$$\mathrm{AUC} = \frac{1}{n_+ n_-} \sum_{i \in \mathcal P} \sum_{j \in \mathcal N} \Big( [s_i > s_j] + \tfrac12 [s_i = s_j] \Big) .$$

Proof. Assume first that all scores are distinct. The staircase has a horizontal step of width $1/n_-$ for each negative $j$ and vertical steps only in between, so its area is the sum of the rectangles under the horizontal steps. When the walk reaches negative $j$, it has already passed exactly the positives with higher scores, so the step is at height $\mathrm{TPR} = \#\{i \in \mathcal P : s_i > s_j\}/n_+$. Summing over the negatives,

$$\mathrm{AUC} = \sum_{j \in \mathcal N} \frac{1}{n_-} \cdot \frac{\#\{i \in \mathcal P : s_i > s_j\}}{n_+} = \frac{1}{n_+ n_-} \sum_{i \in \mathcal P} \sum_{j \in \mathcal N} [s_i > s_j] .$$

If a positive and a negative share a score, the walk passes both at once along a diagonal segment, whose area is half of the rectangle the two orders would give: each tied pair counts $\tfrac12$. $\square$

The sum is the Mann–Whitney $U$ statistic divided by $n_+ n_-$, which is how Hanley and McNeil (1982) introduced the AUC to medicine. In the worked example, the four positives outrank $4, 4, 3$ and $2$ of the four negatives, $13$ pairs out of $16$, the area found above. Let $S_+$ and $S_-$ be the scores of a positive and a negative drawn independently from the population. If the test scores are independent draws, positives distributed as $S_+$ and negatives as $S_-$, every term of the sum has expectation $P(S_+ > S_-) + \tfrac12 P(S_+ = S_-)$. The terms are dependent, since each score enters $n_-$ or $n_+$ pairs, so the plain law of large numbers does not apply; but the variance of their average still falls like $1/n_+ + 1/n_-$ (Hanley and McNeil give it in closed form), and by Chebyshev’s inequality the pair share converges, as both classes grow, to

$$\mathrm{AUC} = P(S_+ > S_-) + \tfrac12 P(S_+ = S_-) .$$

Worked example: Gaussian scores. If negatives score $N(0, 1)$ and positives $N(\mu, \sigma^2)$, independently, then $S_+ - S_- \sim N(\mu, 1 + \sigma^2)$ and $\mathrm{AUC} = \Phi\big(\mu/\sqrt{1 + \sigma^2}\big)$. Classifier A of the figures, with $\mu = 1.5$ and $\sigma = 1$, has $\Phi(1.5/\sqrt2) = 0.856$; classifier B, with the same mean and $\sigma = 2$, has $\Phi(1.5/\sqrt5) = 0.749$. The $2\,\mathrm{AUC} - 1$ used in credit scoring under the name Gini coefficient is the same number rescaled to $[-1, 1]$.

The pair form explains both uses and limits of the AUC. It is the natural metric when the output is a ranking, and a pairwise surrogate of it is what learning to rank optimizes. But every pair counts the same, whether it involves the top of the list, which a user sees, or the bottom, which nobody does.

When ROC flatters: the precision–recall curve#

When positives are rare, the false positive rate divides by a huge $n_-$. A detector with recall $0.9$ at $\mathrm{FPR} = 0.01$ sits near the top-left corner of the ROC plane, yet with 1,000 positives among a million negatives it raises 10,000 false alarms for 900 true ones, a precision of $0.083$. The precision–recall curve plots precision against recall as the threshold moves. Its baseline is not a diagonal but the horizontal line at precision $\pi$, what flagging at random achieves. Its summary is the average precision, the precision at the rank of each positive averaged over the positives,

$$\mathrm{AP} = \frac{1}{n_+} \sum_{k\,:\, y_{(k)} = 1} \text{precision@}k ,$$

where $y_{(k)}$ is the label of the $k$-th highest score. AP is the area under the precision–recall staircase that holds each precision constant until the next positive. In the worked example the positives sit at ranks 1, 2, 4 and 6, with precisions $1, 1, 3/4, 4/6$, so $\mathrm{AP} = 0.854$.

Two cautions. Precision–recall points should not be joined by straight lines: between two operating points precision changes nonlinearly with recall, and linear interpolation overstates the area, which is why AP sums steps instead (Davis and Goadrich, 2006). And the curves are tied together: for fixed class sizes, one classifier’s ROC curve lies above another’s everywhere if and only if its precision–recall curve does (Davis and Goadrich, 2006). The two plots disagree only when the curves cross, and then the areas weight the crossing regions very differently.

Figure

ROC and precision–recall curves of two classifiers

Classifiers A and B show such a crossing. A ranks better on average and wins on ROC area, $0.857$ against $0.748$ on the simulated test set of the figure. B’s positive scores are more spread out, so some of them land far above every negative: at a false positive rate of 1% it already finds 35% of the positives, against A’s 21%. That region is a sliver of the ROC plot and the beginning of the precision–recall plot, where B wins on average precision, $0.431$ against $0.338$ (on a large sample from the same distributions, $0.44$ against $0.34$).

Which to report. Prefer the precision–recall curve and AP when positives are rare and the top of the ranking is what gets acted on: retrieval, fraud alerts, screening with limited follow-up. On imbalanced data it shows differences that the ROC plot compresses into its left edge (Saito and Rehmsmeier, 2015). Prefer ROC and AUC when both classes matter, when the classifier will serve several operating points, or when the prevalence will change: ROC is invariant to it, while the same classifier’s AP falls as positives become rarer, so APs from datasets with different prevalences cannot be compared. Report the prevalence next to any AP, and a curve rather than a single area when the curves may cross.

Metrics with error bars#

Every number above is computed on a finite test set and is an estimate with sampling error. Recall is a proportion over the $n_+$ positives only, so its error depends on how many positives the test set has, not on its total size: 40 detections out of 50 positives give a recall of $0.8$ with a 95% Wilson interval of $[0.67, 0.89]$, as in the confidence intervals note. For AUC, AP and F1 there is no simple formula, and the bootstrap resamples the test set instead. In the Code section, 500 bootstrap resamples give AUC $0.801$ with a 95% interval $[0.779, 0.822]$ and AP $0.266$ with $[0.227, 0.311]$: the rare-class metric is the less precise one. To compare two classifiers on the same test set, resample examples and compute the difference on each resample, keeping the pairing, as in comparing two models on one test set; for AUC, the test of DeLong, DeLong and Clarke-Pearson (1988) is the analytic alternative. The same care applies to the threshold: one tuned on a small validation set fits its noise.

What changed#

	Accuracy at threshold 0.5	The metrics of this note
Errors	one count	two kinds, FP and FN, in a confusion matrix
Rare positives	hidden by the majority class	recall, precision, F1, AP focus on them
Threshold	fixed at 0.5	chosen from a requirement, a budget, F1 or costs ($t^* = c_{\mathrm{FP}}/(c_{\mathrm{FP}} + c_{\mathrm{FN}})$)
All thresholds at once		ROC curve and AUC (ranking, prevalence-free); PR curve and AP (top of the ranking, prevalence-dependent)
Uncertainty	a single number	Wilson intervals, the bootstrap, paired comparisons

Limitations#

• A test set describes one population. When the prevalence or the inputs shift at deployment, precision, AP and the best threshold shift with them; recall and ROC survive a pure prevalence shift, nothing survives a change in what the features mean.
• Areas hide shape. Two classifiers with the same AUC or AP can behave differently at the operating point that matters; look at the curve near it.
• Ranking metrics ignore calibration. AUC and AP are unchanged by any strictly increasing transformation of the scores; use the log-loss, the Brier score or a reliability diagram to judge probabilities.
• Costs are rarely known exactly. The cost threshold is only as good as the cost estimates; it is worth reporting how the decision changes over a plausible range.
• Labels are noisy. Errors in the test labels cap every metric and can reorder close classifiers.

What you should remember#

• The confusion matrix (TP, FP, FN, TN) underlies every threshold metric. Accuracy weights the classes by prevalence; compare it with the majority baseline.
• Precision $= P(y = 1 \mid \hat y = 1)$ depends on prevalence; recall $= P(\hat y = 1 \mid y = 1)$ does not. F1 is their harmonic mean and ignores true negatives.
• Choose thresholds on validation data, from a requirement, a budget or costs: with calibrated probabilities flag when $p > c_{\mathrm{FP}}/(c_{\mathrm{FP}} + c_{\mathrm{FN}})$.
• The ROC curve plots TPR against FPR over all thresholds; it depends only on the ranking and not on prevalence. Its area is the probability that a random positive outranks a random negative, $\Phi(\mu/\sqrt{1 + \sigma^2})$ for Gaussian scores.
• The precision–recall curve and average precision focus on the top of the ranking and reveal what ROC compresses when positives are rare; ROC dominance and PR dominance are equivalent, but the areas can disagree when curves cross.
• Report metrics with intervals (Wilson, bootstrap) and compare models with paired resampling.

Code#

The screening example, AUC and AP by hand, the two simulated classifiers, a real model on imbalanced data with threshold and cost choices, and error bars:

import numpy as np
from scipy.stats import norm
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score, precision_recall_curve, f1_score, confusion_matrix

rng = np.random.default_rng(0)

# 1. the screening test per 10,000 people: prevalence 1%, recall 0.9, false positive rate 0.09
tp, fn, fp, tn = 90, 10, 891, 9009
precision, recall = tp / (tp + fp), tp / (tp + fn)
print(round((tp + tn) / 10_000, 4), round(precision, 4), round(2 * precision * recall / (precision + recall), 4))
# 0.9099 0.0917 0.1665: accuracy, precision, F1
print(round(5 * tp / (5 * tp + 4 * fn + fp), 3))   # 0.326: F2 = 450 / 1381

# 2. eight scored examples: AUC as the share of correctly ordered pairs, AP as mean precision at the positives
s = np.array([0.9, 0.8, 0.7, 0.6, 0.4, 0.3, 0.2, 0.1]); y = np.array([1, 1, 0, 1, 0, 1, 0, 0])
pairs = np.mean([si > sj for si in s[y == 1] for sj in s[y == 0]])
hits = np.cumsum(y)[y == 1] / (np.flatnonzero(y) + 1)
print(pairs, roc_auc_score(y, s), round(hits.mean(), 4), round(average_precision_score(y, s), 4))
# 0.8125 0.8125 0.8542 0.8542

# 3. classifiers A and B of the figures on a large sample: AUC against Phi(mu / sqrt(1 + sigma^2))
n_pos, n_neg = 20_000, 380_000
labels = np.r_[np.ones(n_pos), np.zeros(n_neg)]
for name, sd in (('A', 1.0), ('B', 2.0)):
    scores = np.r_[1.5 + sd * rng.normal(size=n_pos), rng.normal(size=n_neg)]
    print(name, round(roc_auc_score(labels, scores), 3), round(norm.cdf(1.5 / np.sqrt(1 + sd ** 2)), 3), round(average_precision_score(labels, scores), 3))
# A 0.857 0.856 0.336 | B 0.75 0.749 0.436: A wins on AUC, B on AP

# 4. a real model on imbalanced data: accuracy against the majority baseline, the F1-optimal threshold
X, yy = make_classification(n_samples=30_000, n_features=10, n_informative=4, weights=[0.95], flip_y=0.02, random_state=1)
Xtr, Xrest, ytr, yrest = train_test_split(X, yy, test_size=0.5, stratify=yy, random_state=0)
Xva, Xte, yva, yte = train_test_split(Xrest, yrest, test_size=0.5, stratify=yrest, random_state=0)
model = LogisticRegression().fit(Xtr, ytr)
p_va, p_te = model.predict_proba(Xva)[:, 1], model.predict_proba(Xte)[:, 1]
print(round(yte.mean(), 4), round(np.mean((p_te >= 0.5) == yte), 4), round(1 - yte.mean(), 4))   # 0.0591 0.9419 0.9409
print(confusion_matrix(yte, p_te >= 0.5).ravel())                         # [7034 23 413 30]: tn, fp, fn, tp at 0.5
P, R, T = precision_recall_curve(yva, p_va)
F = 2 * P[:-1] * R[:-1] / (P[:-1] + R[:-1] + 1e-12)
t_best = T[F.argmax()]                                                    # chosen on validation, not test
print(round(t_best, 3), round(F.max() / 2, 3), round(f1_score(yte, p_te >= 0.5), 3), round(f1_score(yte, p_te >= t_best), 3))
# 0.152 0.175 0.121 0.339: best threshold, half the best F1, test F1 at 0.5 and at the best threshold

# 5. costs: a missed positive costs 500, a false alarm 5, so act when p > 5 / (5 + 500)
t_cost = 5 / 505
for t in (0.5, t_best, t_cost):
    pred = p_te >= t
    print(round(t, 4), int(500 * np.sum(~pred & (yte == 1)) + 5 * np.sum(pred & (yte == 0))))
# 0.5 206615 | 0.1516 127520 | 0.0099 36435: total cost on the test set

# 6. error bars: bootstrap the test set for AUC and AP, Wilson interval for a recall of 40/50
idx = rng.integers(0, len(yte), (500, len(yte)))                         # 500 resamples of the test set
aucs = [roc_auc_score(yte[i], p_te[i]) for i in idx]
aps = [average_precision_score(yte[i], p_te[i]) for i in idx]
print(round(roc_auc_score(yte, p_te), 3), np.percentile(aucs, [2.5, 97.5]).round(3), round(average_precision_score(yte, p_te), 3), np.percentile(aps, [2.5, 97.5]).round(3))
# 0.801 [0.779 0.822] 0.266 [0.227 0.311]
k, n, z = 40, 50, 1.96
c, h = (k + z * z / 2) / (n + z * z), z * np.sqrt(k * (n - k) / n + z * z / 4) / (n + z * z)
print(round(c - h, 3), round(c + h, 3))                                   # 0.67 0.888

Next#

The metrics judge a classifier once it is trained; they do not say which of many equally good classifiers to train. When the classes can be separated, infinitely many hyperplanes have zero training error, and the surrogate losses of the linear classifiers note either cannot choose among them (the hinge, once every margin exceeds 1) or run off to infinity (the logistic loss). The next note answers with geometry: the support vector machine takes the hyperplane with the widest margin, and its dual shows that the answer depends on a handful of training points.

References#

• T. Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874, 2006.
• J. A. Hanley, B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36, 1982.
• J. Davis, M. Goadrich. The relationship between precision-recall and ROC curves. ICML, 233–240, 2006.
• T. Saito, M. Rehmsmeier. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432, 2015.
• C. Elkan. The foundations of cost-sensitive learning. IJCAI, 2001. Section 2 (the optimal threshold from a cost matrix).
• Z. C. Lipton, C. Elkan, B. Narayanaswamy. Optimal thresholding of classifiers to maximize F1 measure. ECML PKDD, 225–239, 2014.
• E. R. DeLong, D. M. DeLong, D. L. Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44(3), 837–845, 1988.
• C. J. van Rijsbergen. Information Retrieval, 2nd ed. Butterworths, 1979. Chapter 7 (evaluation, the F measure).
• C. D. Manning, P. Raghavan, H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008. Chapter 8 (precision, recall, F measure, average precision).
• E. Sokolov et al. Машинное обучение (HSE): lecture 4 (classification metrics, ROC and PR curves), seminar 5 (AUC as the share of correctly ordered pairs).