03 / Resources/Evaluation · Confusion Matrix, Precision-Recall & F1

Interactive Diagnostic Essay · Topic 03

Confusion Matrix, Precision-Recall & F1

Every classification error has a name and a cell. Dissect the confusion matrix in its 2×2 and 3×3 forms, learn what TP, FP, FN and TN really cost, and see how Precision, Recall and F1 are balanced under severe class imbalance.

01 / The Ledger of Predictions

What Is a Confusion Matrix?

A confusion matrix is a simple table that cross-tabulates what the model predicted against what was actually true. It is called “confusion” because it shows exactly where the model gets confused — and where it does not.

For a binary problem, the matrix is 2×2. Read it by position: rows are the actual class, columns are the predicted class. The diagonal (top-left, bottom-right) always holds correct predictions; everything off the diagonal is a specific kind of mistake.

Running Example · Spam Filter (N = 100)

Test set: 20 spam emails (the positive class) and 80 legitimate emails. The classifier flagged 23 emails as spam. The matrix on the right records every one of the 100 verdicts.

Two cells will dominate everything that follows: the diagonal $\text{TP} + \text{TN} = 87$ correct predictions, and the two error cells $\text{FP} + \text{FN} = 13$ mistakes — each with a very different real-world cost.

BINARY CONFUSION MATRIX · SPAM FILTERN = 100

Predicted · Spam

Predicted · Not Spam

Actual · Spam

TP · 1515Spam caught

FN · 55Spam missed

Actual · Not Spam

FP · 88Ham flagged

TN · 7272Ham ignored

Rows = Actual Class · Columns = Predicted Class

02 / The Four Verdicts

TP, FP, FN & TN — Highlighted in Both Matrices

Select an outcome below. The corresponding square lights up in the 2×2 binary matrix and in the 3×3 multiclass matrix simultaneously — the same four ideas extend from two classes to many.

True Positive

Predicted Positive · Actually Positive

The model raised the alarm, and the alarm was real. The prediction and the ground truth agree on the positive class.

Spam ExampleA spam email is correctly flagged and moved to the junk folder.

Every correct detection. The numerator of both Precision and Recall — the only cell every metric wants to grow.

Feeds Precision ↑ · Recall ↑ · Accuracy ↑

2×2 · Binary — TP Highlighted

Pred +

Pred −

Act +

Act −

3×3 · Multiclass — TP Highlighted

View as:

Pred Cat

Pred Dog

Pred Bird

Act Cat

TP42

FN5

FN3

Act Dog

FP7

TN38

TN5

Act Bird

FP2

TN4

TN34

The one-vs-rest trick: to evaluate the Cat class, mentally collapse the 3×3 into a 2×2 — the diagonal cell

M[0][0] = 42

becomes TP, its row becomes FN, its column becomes FP, and everything else is TN.

03 / Beyond Binary

Reading a 3×3 Matrix Per Class

With $K$ classes the matrix becomes $K \times K$ . There is no single global TP/FP/FN anymore — instead we compute one binary confusion matrix per class and derive per-class metrics:

Precision of class Cat

P_{0} = \frac{M[0][0]}{\sum_{i} M[i][0]} = \frac{42}{42 + 7 + 2}

Recall of class Cat

R_{0} = \frac{M[0][0]}{\sum_{j} M[0][j]} = \frac{42}{42 + 5 + 3}

Mnemonic: precision sums down the column (everything predicted as this class), recall sums across the row (everything that truly was this class).

Per-class numbers give $K$ scores — but leaderboards need one. Three averaging strategies collapse them, each hiding different things:

MACRO

\frac{1}{K}\sum_k P_k

Unweighted mean over classes. Every class counts equally — best when each class matters equally, regardless of how rare it is.

MICRO

\frac{\sum_k TP_k}{\sum_k (TP_k + FP_k)}

Pool all counts globally first. Dominated by frequent classes — for single-label multiclass it equals accuracy.

WEIGHTED

\sum_k \frac{n_k}{N}\, P_k

Support-weighted mean. Frequent classes dominate — a compromise that still reports rare-class failures.

Worked Example · Per-Class Recall

Cat84.0%42/50 caught

Dog76.0%38/50 caught

Bird85.0%34/40 caught

Macro recall here = 0.817. If one rare class scored near zero, macro averaging would expose it — micro averaging would bury it.

04 / Derived Metrics

From Four Counts to Every Metric

Every classification metric is just a different ratio over the same four cells. The question is never “which formula” — it is which error cell hurts your application.

PrecisionPositive Predictive Value

\text{Precision} = \frac{TP}{TP + FP}

“Of everything I flagged, how much was real?” Punishes FP. Optimize when false alarms are expensive — spam filters, recommendation reranking.

RecallSensitivity · TPR

\text{Recall} = \frac{TP}{TP + FN}

“Of all real positives, how many did I catch?” Punishes FN. Optimize when misses are expensive — cancer screening, fraud detection.

AccuracyOverall Correctness

\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

“How often am I right overall?” Collapses under imbalance — 95% negatives make the trivial “always negative” model 95% accurate.

SpecificityTrue Negative Rate

\text{Specificity} = \frac{TN}{TN + FP}

“How well do I leave negatives alone?” The mirror of recall — the two together form the ROC curve of Topic 04.

Worked · Spam Filter (TP 15 · FP 8 · FN 5 · TN 72)

ACCURACY87.0%(15+72)/100

PRECISION65.2%15/(15+8)

RECALL75.0%15/(15+5)

SPECIFICITY90.0%72/(72+8)

05 / The Accuracy Paradox

Precision–Recall Trade-off Simulator

In fraud detection, rare pathology, or cyber threats, positive events occur in fewer than 5% of cases. A naive model predicting “all negative” achieves 95% accuracy while finding zero actual threats.

POSITIVE EVENT BASE RATE10%

DECISION THRESHOLD (τ)0.45

OVERALL ACCURACY:76.4%

PRECISION:24.6%

RECALL:66.0%

HARMONIC F1 SCORE:0.359

Drag $\tau$ down: recall climbs, precision collapses into false alarms. Drag it up: precision sharpens, recall starves. Accuracy barely notices — watch how static it stays while the errors swap places. That numbness is the paradox.

LIVE 2×2 CONFUSION MATRIX (N = 1000)τ = 0.45

TRUE POSITIVE (TP)66Correctly caught rare events

FALSE POSITIVE (FP)202False alarms / spurious flags

FALSE NEGATIVE (FN)34Missed positive events (danger!)

TRUE NEGATIVE (TN)698Correct normal rejections

PRECISION–RECALL CURVEOperating point @ τ = 0.45

Lower τ slides the operating point right into the precision cliff · Higher τ pushes it up-left

06 / One Number to Rule Them

F1 & the Generalized F-Beta

The arithmetic mean of $100\%$ precision and $0\%$ recall is $50\%$ — concealing total failure. The harmonic mean heavily penalizes asymmetry: whenever one side collapses, F1 collapses with it.

F1 Score Formulation

F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}}

Because $F_1$ can be rewritten purely in confusion-matrix counts, it never touches TN — under extreme imbalance it stays honest where accuracy cannot.

Real applications are rarely symmetric. When missed detections matter $\beta$ times more than false alarms, the generalized $F_\beta$ score shifts the balance:

Generalized F-Beta Metric

F_\beta = (1 + \beta^2) \frac{\text{Precision} \cdot \text{Recall}}{(\beta^2 \cdot \text{Precision}) + \text{Recall}}

BETA WEIGHT (β > 1 FAVORS RECALL · β < 1 FAVORS PRECISION)β = 1.0

F_β AT CURRENT OPERATING POINT (P 25% · R 66%):0.359

β = 1 is plain F1 — precision and recall weighted equally, the neutral harmonic compromise.

07 / The Bridge To ROC

Same Four Cells, A Different Curve

Nothing here is new data. The same four cells can be plotted in two coordinate systems, and each produces its own curve. Slide a single decision threshold across the score axis: every position yields one confusion matrix, and every confusion matrix yields one point. Connect the points and you have plotted recall against precision — the curve of this article — or, if you prefer, TPR against FPR, which is the ROC curve of Topic 04. The axes differ; the ledger underneath does not.

The two axes are the same numbers wearing different labels. $\text{TPR} = \text{Recall} = \frac{TP}{TP + FN}$ and $\text{FPR} = 1 - \text{Specificity} = \frac{FP}{FP + TN}$ . Both curves sweep every threshold, so both are threshold-invariant in the sense of summarising the full range without committing to one operating point.

They part ways on prevalence. TPR and FPR each normalise inside their own class — TPR divides by all the real positives, FPR by all the real negatives — so the ROC curve is unchanged if you keep the model fixed and make the positive class a hundred times rarer. Precision refuses that trick: it divides by $TP + FP$ , which contains the negative class through the false alarms. Change the class balance and the PR curve moves even though the model did not.

Worked Contrast · The Spam Filter Under Two Balances

As measured · 20 spam / 80 ham

Recall = 15/20 = 0.750
FPR = 8/80 = 0.100
Precision = 15/23 = 0.652

Same 8 false alarms · 20 spam / 9,980 ham

Recall = 15/20 = 0.750
FPR = 8/9,980 ≈ 0.0008
Precision = 15/23 = 0.652

The false alarms did not change; only the pool of legitimate mail did. ROC's FPR collapses toward zero and the point climbs into the top-left corner, flattered by a denominator it never had to earn. Precision is unmoved at 65.2% — it has no TN term to hide behind. When positives are rare, the PR curve is the honest view.

THRESHOLD SWEEP · THE SAME VERDICTS, RE-PLOTTED8s loop · τ → PR operating point

At τ = 0.50 the sweep sits exactly on the running example · R 75% · P 65.2%

That's Not All · The Same Cells, Re-Cut

Metric	Cells Used	Blind To	Answers
Accuracy	(TP + TN) / N	the FP / FN split	How often am I right overall?
Precision	TP / (TP + FP)	FN · TN	How much of what I flagged is real?
Recall · TPR	TP / (TP + FN)	FP · TN	How many real positives did I catch?
Specificity	TN / (TN + FP)	FN · TP	How well do I leave negatives alone?
FPR	FP / (FP + TN)	FN · TP	How often do negatives get flagged?
F1	2TP / (2TP + FP + FN)	TN	A single harmonic compromise.
Fβ	weighted P & R	TN	A cost-weighted compromise.
ROC curve	TPR vs FPR	the operating threshold	Every threshold, at once.
AUC	∫ TPR d(FPR)	threshold · calibration	How well does it rank?
PR curve	Precision vs Recall	TN · prevalence	Ranking quality when positives are rare.

← Topic 02: Logistic Regression Topic 04: ROC & AUC →