\( \newcommand{\vw}{\mathbf{w}} \newcommand{\vx}{\mathbf{x}} \newcommand{\vt}{\mathbf{t}} \newcommand{\vy}{\mathbf{y}} \newcommand{\vz}{\mathbf{z}} \newcommand{\vq}{\mathbf{q}} \newcommand{\vk}{\mathbf{k}} \newcommand{\vv}{\mathbf{v}} \newcommand{\vc}{\mathbf{c}} \newcommand{\vp}{\mathbf{p}} \newcommand{\vh}{\mathbf{h}} \newcommand{\vmu}{\boldsymbol{\mu}} \newcommand{\vsigma}{\boldsymbol{\sigma}} \newcommand{\vepsilon}{\boldsymbol{\epsilon}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\given}{\,\vert\,} \)

Lecture 15 — Trustworthy AI: Calibration & Uncertainty

Deep Learning · Session 15

Kasemsit Teeyapan

Department of Computer Engineering, Chiang Mai University

Today

  1. Accurate ≠ trustworthy — a model can be confidently wrong
  2. Two uncertainties: epistemic (reducible) vs aleatoric (intrinsic)
  3. Calibration — does “90% confident” mean right 90% of the time?
  4. Modern nets are miscalibrated; temperature scaling fixes it cheaply
  5. Estimating uncertainty: ensembles, MC dropout, and selective prediction

Reading: Guo et al. (2017). Bishop covers epistemic/aleatoric uncertainty (§2); calibration is paper-based.

Recap → accurate is not enough

  • L14: scale → higher accuracy
  • but deployment needs more than accuracy:
    • when should we trust a prediction?
    • when should the model abstain and ask a human?

A softmax output of 0.99 is not a probability of being correct — unless the model is calibrated.

A melanoma classifier (L0) that is 95% accurate but 99% confident on everything is dangerous: it never flags the cases a doctor should double-check.

Two kinds of uncertainty

Epistemic — what the model doesn’t know

  • from limited data / model
  • reducible: more data, better model → less of it
  • high in regions far from training data (→ L16)

Aleatoric — intrinsic noise in the data

  • label overlap, sensor noise, ambiguity
  • irreducible: more data won’t remove it
  • e.g. a blurry image that’s genuinely ambiguous

A trustworthy model reports both — and especially flags epistemic uncertainty on inputs unlike anything it trained on.

What is calibration?

A model is calibrated if, among all predictions made with confidence \(p\), the fraction correct is also \(p\).

\[\Pr(\,\hat y = y \mid \text{conf} = p\,) = p\]

  • plot accuracy vs confidence → the reliability diagram
  • perfect calibration = the diagonal

Expected Calibration Error — average gap, weighted by bin size:

\[\text{ECE} = \sum_{b} \frac{|B_b|}{n}\,\bigl|\,\text{acc}(B_b) - \text{conf}(B_b)\,\bigr|\]

Modern networks are overconfident

Guo et al. (2017): compared to older/shallower models, modern deep nets are more accurate but less calibrated.

  • depth, width, and reduced regularization (L8) all worsen calibration
  • the softmax saturates → confidence ≫ accuracy

An irony of scale (L14): the same trends that improve accuracy tend to make raw confidence less trustworthy. We must fix it explicitly.

Demo: reliability & temperature scaling

import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(0); K, M, beta = 10, 4000, 2.6
z = rng.normal(0, 2.0, size=(M, K))
sm = lambda z: np.exp(z - z.max(1, keepdims=True)) / np.exp(z - z.max(1, keepdims=True)).sum(1, keepdims=True)
labels = np.array([rng.choice(K, p=p) for p in sm(z)]); correct = (z.argmax(1) == labels)

def reliability(conf, ax, title):
    bins = np.linspace(0, 1, 11); acc, mid, e = [], [], 0
    for i in range(10):
        m = (conf > bins[i]) & (conf <= bins[i+1])
        if m.sum(): acc.append(correct[m].mean()); mid.append(conf[m].mean()); e += abs(correct[m].mean()-conf[m].mean())*m.sum()/M
    ax.plot([0,1],[0,1],"k--",lw=1); ax.bar(mid, acc, width=0.09, color="#5d2c80", alpha=0.8)
    ax.set_title(f"{title}\nECE = {e:.3f}", fontsize=8); ax.set_xlabel("confidence", fontsize=7)
    ax.set_xlim(0,1); ax.set_ylim(0,1); ax.set_ylabel("accuracy", fontsize=7)

fig, ax = plt.subplots(1, 2, figsize=(6.4, 2.8))
reliability(sm(z*beta).max(1),       ax[0], "raw (overconfident)")
reliability(sm(z*beta/beta).max(1),  ax[1], f"temperature-scaled (T={beta})")
plt.tight_layout(); plt.show()

Temperature scaling

Divide the logits by a single learned \(T>1\) before softmax:

\[p_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}\]

  • \(T>1\) softens the distribution → lowers confidence
  • fit \(T\) on a validation set (one parameter!)
  • accuracy unchanged (argmax is invariant)

Pillar: the cheapest trustworthy-AI fix in existence — one number, no retraining, and your confidences become honest.

Estimating uncertainty

Calibration fixes confidence; these estimate uncertainty directly:

  • Deep ensembles — train \(K\) models; disagreement = uncertainty (strong baseline)
  • MC dropout — keep dropout (L8) on at test time, sample many forward passes
  • Bayesian NN — distribution over weights (principled, expensive)

All three turn a single point prediction into a spread. A wide spread on a new input = “I’m not sure” — exactly the epistemic signal we want.

Why it matters: selective prediction

With trustworthy confidence, a model can abstain:

  • predict only when confidence > threshold; else defer to a human
  • trades coverage for accuracy on what it does answer
  • essential in medicine (L0), driving, finance

This only works if confidence is calibrated and uncertainty rises off-distribution. The next lecture asks: what happens when test data isn’t like training data?

Summary

  1. Accurate ≠ trustworthy — confidence must be meaningful, not just high
  2. Epistemic (reducible) vs aleatoric (irreducible) uncertainty
  3. Calibration: confidence \(p\) should mean correct with probability \(p\) (measure with ECE)
  4. Modern nets are overconfident; temperature scaling fixes it with one parameter
  5. Ensembles / MC dropout estimate uncertainty → enable selective prediction

Something to think about: temperature scaling makes confidences honest on the test distribution. Why might it fail to keep them honest on inputs from a different distribution? (Hint: L16.)

📌 Read before next class

Distribution Shift & Out-of-Distribution Generalization

What happens when test data isn’t like training data — and how do we detect it? The course comes full circle to generalization (L4).

Reading: Hendrycks & Gimpel, A Baseline for Detecting Misclassified and OOD Examples (2017). References: Guo et al., On Calibration of Modern Neural Networks (2017); Lakshminarayanan et al., Deep Ensembles (2017); Gal & Ghahramani, MC Dropout (2016).

⌂ Home