Lecture 2 — Linear Regression
Deep Learning · Single-layer Networks
Choices of basis function
Polynomial
Gaussian
Sigmoid
Your data is flat, then jumps to a higher level and stays there. Which of the three captures that step most naturally — and which fights it?
The polynomial of Lecture 0 used \(\phi_j(x)=x^j\). Gaussian/sigmoid bases are local vs. global — the choice is an inductive bias.
Bishop Fig 4.2
Basis ① — Polynomial: a global basis
\[ \phi_j(x) = x^{\,j} \]
- one curve covers the whole axis — global support
- so every weight \(w_j\) bends the entire fit at once
- Purpose: capture a smooth, overall trend with just a few terms
- knobs: only the degree \(M\) (how many powers)
Downside: a point in one region tugs the curve far away → edge wiggles & wild extrapolation — the polynomial overfitting of Lecture 0.
Basis ② — Gaussian: a local bump
\[ \phi_j(x) = \exp\!\Big(\!-\frac{(x-\mu_j)^2}{2s^2}\Big) \]
- a bell centred at \(\mu_j\), width \(s\); ≈ 0 far away — local support
- weight \(w_j\) = the height of the bump at \(\mu_j\)
- Purpose: build the curve from local bumps — nearby \(x \Rightarrow\) nearby \(y\)
- knobs: each centre \(\mu_j\) and the width \(s\) (small = sharp/overfit, large = smooth)
Move one data point → only the nearby bumps react. Local support = stable & controllable — the idea behind RBFs and kernels.
Gaussian — building \(y(x)\)
Stack 4 faint weighted bumps \(w_j\phi_j\) → the bold curve \(y(x)\):
\[ y(x, \vw)=\sum_{j=1}^4 w_j\,e^{-\frac{(x-\mu_j)^2}{2s^2}} \]
\(s=0.25\)
\(\vmu=(-0.6,\,-0.2,\,0.2,\,0.6)\)
\(\vw=(1,\,-1,\,1.2,\,-0.8)\)
Bias (\(\phi_0=1\)) omitted here for clarity.
Each bump acts only near its centre \(\mu_j\) → editing one weight reshapes \(y\) locally. Shrink the width \(s\) and the bumps sharpen — more capacity, more overfitting.
Basis ③ — Sigmoid: a soft threshold
\[ \phi_j(x) = \sigma\!\Big(\frac{x-\mu_j}{s}\Big),\qquad \sigma(a)=\frac{1}{1+e^{-a}} \]
- a soft step from 0 → 1, switching on at \(\mu_j\) (steepness \(1/s\))
- Purpose: a switch / threshold — model on–off & saturation
- two sigmoids subtracted make a bump → can mimic Gaussian locality too
This \(\sigma\) is the neural-net activation: a hidden layer = sigmoid basis whose \(\mu_j,s\) are learned, not chosen by hand → Lecture 5.
Sigmoid — building \(y(x)\)
Stack 3 faint weighted steps \(w_j\,\sigma(\cdot)\) → the bold curve \(y(x)\):
\[ y(x, \vw)=\sum_{j=1}^3 w_j\,\sigma\!\Big(\tfrac{x-\mu_j}{s}\Big) \]
\(s=0.1\)
\(\vmu=(-0.4,\,0.1,\,0.5)\)
\(\vw=(1,\,-1.6,\,1.2)\)
Bias (\(\phi_0=1\)) omitted here for clarity.
Each step switches on at its \(\mu_j\); up- and down-steps stack into plateaus. A hidden layer simply learns these \(\mu_j,\,s,\,w_j\) → Lecture 5.
Polynomial, Gaussian, Sigmoid: Same \(\vw\)-and-sum machinery across all three — but the family of curves you can reach is set entirely by the basis.
Basis ④ — ReLU: a piecewise-linear ramp
\[ \phi_j(x)=\max(0,\ x-\mu_j)=(x-\mu_j)_+ \]
- flat 0 until \(\mu_j\), then a straight ramp (slope 1) — a “hinge”
- Purpose: build piecewise-linear functions; each ReLU adds one kink at \(\mu_j\)
- vs sigmoid: an unbounded ramp, not a saturating step
- knobs: the kink location \(\mu_j\) (height/slope via \(w_j\))
The modern default activation: no saturation → gradients don’t vanish for \(x>0\). A ReLU network is a piecewise-linear function approximator → Lecture 5.
Not in Bishop Fig 4.2 — ReLU is the activation most modern deep nets actually use.
ReLU — building \(y(x)\)
Stack 3 faint weighted ramps \(w_j\phi_j\) → the bold, piecewise-linear \(y(x)\):
\[ y(x,\vw)=\sum_{j=1}^3 w_j\,\max(0,\,x-\mu_j) \]
\(\vmu=(-0.5,\,0,\,0.5)\)
\(\vw=(2,\,-3,\,2)\)
Bias (\(\phi_0=1\)) omitted here for clarity.
At \(x=0\) only the first ramp is on: \(y(0)=2(0.5)-3(0)+2(0)=\mathbf{1}\). Each ReLU adds a kink (slope change) at its \(\mu_j\) → the curve is piecewise-linear.
ReLU joins the family: same \(\vw\)-and-sum machinery, but the curves are piecewise-linear — and this is the basis modern deep nets actually use → Lecture 5.
This is a single-layer network
\[ y(\vx,\vw) = \sum_{j=0}^{M-1} w_j\,\phi_j(\vx) \]
- The basis functions \(\phi_j\) are fixed features
- The output is a weighted sum — a single layer of weights \(\vw\)
Deep learning’s big idea (later): learn the features \(\phi_j\) instead of fixing them by hand.
Bishop Fig 4.1
Least squares is a projection
- \({\vt}\) lives in \(\R^N\); the reachable predictions \(\boldsymbol{\Phi}\vw\) span a subspace \(\mathcal S\)
- Least squares picks \(\vy=\boldsymbol{\Phi}\vw_{\mathrm{ML}}\) = the orthogonal projection of \(\vt\) onto \(\mathcal S\)
- The residual \(\vt-\vy\) is ⟂ to \(\mathcal S\) → that is the normal equations
- จากรูป: \(N=3\) samples, \(M=2\) พารามิเตอร์ → \(\mathcal{S}\) เป็นระนาบ 2D plane ใน 3D space
Bishop Fig 4.3 — geometric view of least squares
Two extensions: online & multi-output
Sequential learning (§4.1.5) — the closed form needs all data at once. Update one point at a time instead: \[\vw^{(\tau+1)}=\vw^{(\tau)}+\eta\big(t_n-\vw^{(\tau)\mathsf T}\boldsymbol\phi_n\big)\boldsymbol\phi_n\] - “error × input” again — this is (stochastic) gradient descent (L6), and the rule behind training on huge / streaming data
Multiple outputs (§4.1.7) — for a vector target \(\vt\in\R^K\), stack a weight matrix \(W\):
\[W_{\mathrm{ML}}=\big(\boldsymbol\Phi^{\mathsf T}\boldsymbol\Phi\big)^{-1}\boldsymbol\Phi^{\mathsf T}\mathbf{T}\] - the same \(\boldsymbol\Phi^\dagger\) solves every output column independently — one fit, \(K\) predictions
Bishop Fig 4.4
From distribution to a single prediction
The noise model gives us a full distribution at every \(x\):
\[p(t \given x) = \mathcal{N}(t \mid y(x,\vw),\, \beta^{-1})\]
The red curve \(f^\star(x)\) is the underlying truth; the blue bell at \(x_0\) shows how noisy observations scatter around it.
In practice we must commit to one number — a point prediction. The distribution alone is not enough.
Which single number should we report from \(p(t|x)\)? The mean, the mode, or something else — and does it depend on the loss we care about?
Bishop Fig 4.5
🔬 Demo: Gaussian basis + regularization
How is the design matrix defined here? Which basis family is this? What do you expect ridge (\(\lambda>0\)) to do to the fit near the data edges?
import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(1)
x = np.linspace(0, 1, 12); t = np.sin(2*np.pi*x) + 0.15*rng.standard_normal(x.size)
mu, s = np.linspace(0, 1, 9), 0.12
def basis(z): return np.hstack([np.ones((z.size,1)), np.exp(-0.5*((z[:,None]-mu)/s)**2)])
Phi = basis(x); xs = np.linspace(0, 1, 200); truth = np.sin(2*np.pi*xs)
w_mle = np.linalg.lstsq(Phi, t, rcond=None)[0] # MLE (λ = 0)
lam = 0.1
w_ridge = np.linalg.solve(lam*np.eye(Phi.shape[1]) + Phi.T@Phi, Phi.T@t) # ridge
fig, ax = plt.subplots(1, 2, figsize=(8.5, 2.5), sharey=True)
for a, w, label, color in [(ax[0], w_mle, f"MLE (λ=0)", "r"),
(ax[1], w_ridge, f"Ridge (λ={lam})", "C0")]:
a.plot(x, t, "o", ms=4); a.plot(xs, basis(xs)@w, color, lw=2, label=label)
a.plot(xs, truth, "g--", lw=1, label="truth"); a.legend(fontsize=7); a.set_title(label)
plt.tight_layout(); plt.show()