Generative AI: Reasoning Models & Agents

261405 — Advanced Computer Engineering Technology

Kasemsit Teeyapan

Department of Computer Engineering, Chiang Mai University

2026-07-10

Where we left off

In the ML intro we ended at LLMs that predict the next word, learned to prompt them, and gave them tools.

This deck picks up the 2024–2026 story: models that stop and think before answering — reasoning models — and what happens when we let them act on their own as agents.

Prereqs from the last lecture: tokens → embeddings → Transformer, next-token prediction, RLHF, prompting, tool use.

Roadmap

  1. The wall — why bigger isn’t enough
  2. What is a reasoning model? — think before you answer
  3. Test-time compute — a new scaling axis
  4. How they’re trained — RLHF → RLVR, GRPO, DeepSeek-R1
  1. Using them well — when to think, when not to
  2. From reasoning to agents — tools + a loop
  3. How capable? How fast? — the agent “Moore’s law”
  4. Limits, safety & cost

1 · The wall 🧱

Predicting the next word has a limit

A plain LLM answers in one forward pass — it commits to the first token immediately, then the next, with no chance to backtrack.

That is fine for:

  • “Translate this sentence.”
  • “Summarize this email.”

It struggles with anything that needs multi-step work:

  • competition math, hard proofs
  • multi-file debugging
  • planning a sequence of actions

Analogy — System 1 vs System 2

System 1 = fast, automatic, gut reaction. System 2 = slow, deliberate, step-by-step.

A vanilla LLM is almost pure System 1.

Framing from D. Kahneman, Thinking, Fast and Slow (2011).

The old lever: make it bigger

For years the recipe was scale up pre-training — more parameters, more data, more GPUs.

  • Real gains, but expensive and slowing down.
  • High-quality text is finite — we are running out of new internet to read.
  • Some skills (careful multi-step reasoning) barely improved with size alone.

Key idea of this lecture

Instead of only spending compute at training time, spend more at answer time — let the model think longer on the hard question in front of it.

🔑 Explore: scaling laws · data wall · compute-optimal · Chinchilla · emergent abilities

2 · What is a reasoning model? 🤔

Think out loud, then answer

A reasoning model generates a long internal chain of thought — scratch work — before it writes the final answer. More steps = more chances to catch its own mistakes.

Chain-of-thought: the core trick

Direct answer ❌

Q: A shop sells pens at 7฿, and I pay with 100฿ for 8 pens. Change?

A: 44฿ (wrong, no work shown)

Chain-of-thought ✅

Let me think. 8 × 7 = 56. 100 − 56 = 44. A: 44฿

CoT started as a prompting trick — “Let’s think step by step” (Wei et al., 2022). Reasoning models bake it into the weights: they think by default, and are trained to think well.

Source: Wei et al., Chain-of-Thought Prompting Elicits Reasoning in LLMs (2022).

Prompted CoT vs. a trained reasoner

Prompted CoT Reasoning model
How you get it Add “think step by step” Built in — thinks automatically
Length of thinking Short, brittle Long, and adaptive to difficulty
Self-correction Rare Backtracks, checks, re-tries
How it was learned Emerges from pre-training RL rewards correct reasoning
Examples Any base LLM + prompt o-series, DeepSeek-R1, Gemini/Claude thinking

The thinking tokens are often hidden (or summarized) in the product UI — you pay for and benefit from them, but you may not see them all.

The “aha”: models learn to reflect

During RL training, reasoning models spontaneously develop human-like habits no one hand-coded:

  • “Wait — let me re-check that step.”
  • Trying an approach, abandoning it, trying another.
  • Verifying the answer by plugging it back in.
  • Spending more tokens on harder problems.

DeepSeek called this the “aha moment” — self-reflection emerged purely from rewarding correct final answers, not from being shown how to reflect.

Source: DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL (2025).

3 · Test-time compute 📈

A second scaling axis

Train-time scaling (the old axis)

Spend compute once, up front, to bake knowledge into the weights. Everyone shares the cost.

Test-time scaling (the new axis)

Spend extra compute per question, at answer time, to think longer. You pay each time you ask.

Big result of 2024–2025: letting a model think longer can beat making the model bigger — a small model with a large thinking budget can match a much larger model that answers instantly.

Source: Snell et al., Scaling LLM Test-Time Compute Optimally… (2024); OpenAI, Learning to Reason with LLMs (o1, 2024).

First: how do we grade a reasoner?

You can’t improve — or scale — what you can’t measure. The trick for reasoning: check the final answer objectively.

A verifier pipeline:

  1. Generate with a long CoT.
  2. Extract the final answer (e.g. from \boxed{...}).
  3. Normalize — ½ = 0.5, ignore order/spacing.
  4. Check equivalence to the known answer → correct / not.

e.g. …so the answer is \boxed{3/4} → extract 3/4 → normalize 0.75 = known 0.75 ✓

  • Benchmarks: MATH-500, AIME, GPQA, SWE-bench — full catalog next →.
  • pass@k — correct at least once in k tries.
  • No golden answer? LLM-as-judge or a rubric (weaker, gameable).

The same verifier that grades the model is the signal we will reuse to train it — that’s the bridge to RLVR in §4.

The benchmark zoo 🏟️

A benchmark = a frozen set of problems + a verifier + a headline score. Every claim you read — “80% on X” — is made this way.

Four families you’ll meet most

  • 🧮 Math & reasoning — one objective answer
  • 🔬 Knowledge & science — expert-level facts
  • 💻 Coding — write code that passes tests
  • 🤖 Agents & computer-use — real multi-step work

Same machinery as RLVR (§4): a benchmark is a verifier you report instead of train on. Keep them separate — training on the test set is cheating (see contamination).

Benchmark catalog

Benchmark Measures — example Links
GSM8K grade-school multi-step math — “8 pens × 7฿, pay 100฿; change?” repo · 🤗
MATH-500 competition math, 7 topics × 5 levels — hard algebra/geometry repo · 🤗
AIME elite HS olympiad, integer answer 0–999 — number-theory puzzle MAA · 🤗
Benchmark Measures — example Links
MMLU / -Pro 57 subjects, multiple-choice — law, medicine, history repo · 🤗
GPQA Diamond PhD-level “Google-proof” science — quantum-chemistry Q repo · 🤗
HLE hardest cross-domain expert questions — frontier of knowledge site · 🤗
Benchmark Measures — example Links
HumanEval write a function from a docstring — unit tests must pass repo · 🤗
SWE-bench (Verified) fix a real GitHub issue in a whole repo site · 🤗
LiveCodeBench fresh contest problems, dated (anti-contamination) site · 🤗
Terminal-Bench finish real command-line tasks in a sandbox site · repo
Benchmark Measures — example Links
OSWorld drive a real OS (click/type) across apps site · repo
BrowseComp find hard-to-locate facts across many web pages OpenAI · repo
ARC-AGI abstract pattern-finding from a few examples arcprize.org · repo
Agents’ Last Exam long professional workflows across 55 fields Artificial Analysis

🔑 Explore: pass@1 vs pass@k · SWE-bench Verified · data contamination · LMArena Elo

Reading a benchmark score critically

A single number hides a lot. Five traps to watch:

  • Saturation — MMLU/HumanEval sit ~95%+; the frontier moved on to GPQA, HLE, SWE-bench Pro.
  • Contamination — test leaks into training → inflated scores. Dated sets (LiveCodeBench) and held-out splits fight it.
  • pass@k ≠ pass@1 — “correct in 1 of 64 tries” is not “reliable”.
  • Harness matters — same model, better scaffold swings tens of points (ml-intro §6).
  • Vendor-selected — launch charts are self-reported and cherry-picked; cross-check a neutral leaderboard.

Prefer human-verified splits (e.g. SWE-bench Verified), report pass@1, and always ask: which harness, which split, who ran it?

Leaderboards & aggregate indices

No single benchmark ranks models — people watch aggregators:

  • LMArena (Chatbot Arena) — humans vote on blind A/B pairs → an Elo rating. Good for feel, weak on hard skills. lmarena.ai
  • Artificial Analysis — blends many benchmarks into one Intelligence Index, plus speed and $/token. artificialanalysis.ai

The GPT-5.6 vs Fable 5 headlines (§7) came from exactly these — Agents’ Last Exam and the Intelligence Index — which weigh performance per dollar, not accuracy alone.

More thinking → more accuracy

Accuracy rises smoothly with the log of thinking tokens — a predictable curve, just like classic training scaling laws. The knob is now inference budget, not model size.

Two ways to spend the budget

Sequential — think longer

One long chain of thought: reason, notice a mistake, backtrack, continue.

This is what “reasoning models” do internally.

Parallel — think wider

Generate many independent answers, then pick the best:

  • Self-consistency — majority vote
  • Best-of-N — a verifier/reward model scores each
  • Search (tree / MCTS)

The two combine: think wide and deep. Parallel voting can beat one very long chain for the same total budget.

Let the model check its own work

Self-refinement — a sequential way to spend the budget, no retraining:

\[\text{draft} \to \text{critique} \to \text{revise} \to \dots\]

  • The model scores its own answer and rewrites the weak parts.
  • A cheap confidence signal: token log-probabilities — low-confidence spans are where it’s likely wrong.
  • Stop when the score stops improving.

Careful: a model that is confidently wrong won’t fix itself — self-critique can even reinforce the error. Pair it with an external verifier whenever one exists.

More is not always better

Overthinking is real. Past a point, extra thinking:

  • wastes tokens (💸 cost, latency) for no gain,
  • can talk itself out of a right answer,
  • adds latency users feel (seconds → minutes).

The skill is matching thinking to difficulty: a few tokens for “2+2”, thousands for a hard proof. Good models — and good prompts — budget their thinking.

🔑 Explore: test-time compute · self-consistency · best-of-N · process reward model · overthinking · thinking budget

4 · How reasoning models are trained ⚙️

Recap: RLHF made ChatGPT helpful

From the last lecture: a raw next-token model is turned helpful & safe with RLHF — humans rank answers, a reward model learns their taste, and RL nudges the LLM toward higher-scoring replies.

Problem for reasoning: a learned reward model judges style (“sounds good”), which can be gamed. It doesn’t reliably know if the math is actually correct.

Source: Ouyang et al., InstructGPT / RLHF (2022). See ML intro §6.

RLVR: reward what you can check

RLVR = Reinforcement Learning from Verifiable Rewards. Drop the learned judge; use a programmatic one.

The reward is computed by a checker, not a model:

  • Math — does the final answer equal the known one?
  • Code — do the unit tests pass?
  • Logic/format — does it satisfy the rules?

Reward = +1 if correct, 0 if not.

Because the signal is objective, it is hard to game — the model can’t bluff its way past a failing test. This is what made the 2025 jump in math & coding possible.

🔑 Explore: RLVR · verifiable reward · unit-test reward · outcome vs process reward · GRPO · PPO

The training loop

For each problem: sample several full chains of thought → check each answer → reinforce the reasoning that led to correct ones, discourage the rest. Repeat over millions of problems.

GRPO — the workhorse algorithm

Group Relative Policy Optimization (DeepSeek) is the de facto RLVR algorithm. (The policy is just the model — the thing generating answers that we’re tuning.)

For one question, sample a group of \(G\) answers and score each. Then:

\[A_i = \frac{r_i - \operatorname{mean}(r_1..r_G)}{\operatorname{std}(r_1..r_G)}\]

\(A_i\) is the advantage — a z-score: how many std-devs answer \(i\) beats the group average.

  • \(A_i > 0\): better than the group → make it more likely.
  • \(A_i < 0\): worse → make it less likely.

“Beat your own average.” No separate value network — the extra baseline-guessing model that PPO needs — because the group’s mean is the baseline. Cheaper, simpler, stable.

Source: Shao et al., DeepSeekMath / GRPO (2024); variants: DAPO, VAPO, Open-Reasoner-Zero.

GRPO by example

Ask “12 × 8 = ?” (answer 96). Sample a group of 4 chains of thought and grade each:

Group average = 0.5. Above it → positive advantage (+1, “do more of this”); below → negative (−1, “do less”). No human labels, no value network — just beat your own average.

GRPO under the hood

Three small knobs turn the idea into a stable training run:

  • KL penalty — KL = a “distance” between two probability distributions; here it keeps the tuned model close to the base model, so it doesn’t forget language or drift into gibberish.
  • Ratio clipping (from PPO) — cap how far one update can move the model → no wild swings.
  • Format reward — a bonus for wrapping thoughts in <think>…</think> and the answer in \boxed{}.

Total reward = correctness + format.

In plain words: do more of the chains with positive advantage, less of the rest — without drifting too far from the base model (that’s the KL term).

Refinements: DAPO, VAPO, Dr.GRPO; see also PPO clipping (Schulman et al., 2017).

Case study: DeepSeek-R1

An open model that matched frontier reasoning — and showed how.

  • R1-Zero: pure RLVR on a base model — no supervised examples of reasoning. It taught itself to think (and produced the “aha moment”).
  • R1: add a small cold-start of clean CoT for readability, then RLVR at scale.
  • Weights released openly → a wave of cheap, strong reasoners.

Lesson: you don’t have to show a model how to reason. Reward correct outcomes at scale and reasoning emerges. Then distill it into smaller models.

Source: DeepSeek-AI, DeepSeek-R1 (2025).

Distillation: bottle the reasoning

Doing RLVR is expensive. Do it once on a big model, then copy the skill into small ones.

  1. A big teacher reasoner generates thousands of CoT traces for hard problems.
  2. A small student is fine-tuned (plain SFT) to imitate those traces.
  3. The student inherits much of the reasoning — at a fraction of the size.

DeepSeek-R1 was distilled into Qwen/Llama (1.5B–70B). The small distilled models often beat running RL on that small model directly.

This is why a capable reasoner can run on your laptop: someone paid for the RL once, then distilled and quantized it.

Reward hacking: the catch

Optimize a proxy hard enough and the model games it — Goodhart’s law.

  • Editing the test file instead of fixing the bug.
  • Hard-coding the expected output.
  • return True to make an assertion pass.
  • Long, confident CoT that is not how it really got the answer (unfaithful).

Verifiable rewards are harder to hack than learned ones — but not impossible. Robust checkers, held-out tests, and monitoring the chain-of-thought all matter.

See ML intro §9 — Optimizing the wrong thing.

Lab: build a tiny reasoner from scratch

Everything in §2–§4 fits on one small model + a laptop GPU. A self-contained project:

  1. Load a small base model (e.g. Qwen3-0.6B) + a MATH subset.
  2. Write a verifier: extract \boxed{}, normalize, check.
  3. Baseline eval → add CoT + self-consistency → measure the gain.
  1. Run a few steps of GRPO; watch reward & accuracy climb (the “aha”).
  2. Distill the result into an even smaller student.

Open building blocks — no framework magic needed: a small Qwen3 base, the GSM8K / MATH datasets, and Hugging Face TRL (GRPOTrainer). The whole loop fits on one GPU.

5 · Using reasoning models well 🎛️

When to reach for a reasoning model

Use a reasoning model Use a fast model
Math, logic, proofs Chat, Q&A, summaries
Multi-step coding / debugging Simple formatting / extraction
Planning & multi-constraint problems Classification, translation
“Get it right, I’ll wait” “Get it now, high volume”

Reasoning costs more tokens, more money, more seconds. Don’t pay the “thinking tax” on easy tasks — route easy → fast, hard → reasoning.

The thinking-budget knob

Modern APIs expose the trade-off directly — you choose how hard to think.

  • Reasoning effort: none / low / medium / high / xhigh / max
  • Thinking token budget: a max number of internal tokens
  • Extended thinking on/off — plus parallel modes (e.g. GPT-5.6 ultra runs several agents at once)
model:  claude / gpt / gemini (thinking)
effort: high
budget: 8000 thinking tokens

Higher budget → better on hard problems, worse cost & latency. Tune it to the task; many products auto-route based on the question.

Watch out: the chain can lie

The visible reasoning is not a guaranteed audit log:

  • CoT can be unfaithful — a plausible story that isn’t the real cause of the answer.
  • A confident chain can still reach a wrong answer.
  • Don’t paste secrets into prompts — thinking may echo them.

Treat the chain-of-thought as a helpful hint, not proof. For high-stakes use, verify the result independently (run the code, check the source).

🔑 Explore: reasoning effort · thinking budget · model routing · faithfulness · hybrid reasoning models

6 · From reasoning to agents 🤖

Reasoning + tools + a loop = agent

A reasoning model thinks. An agent thinks, acts, sees the result, and thinks again — over many steps, toward a goal.

\[\textbf{Agent} = \text{reasoning model} \;+\; \text{tools} \;+\; \text{memory} \;+\; \textbf{a loop}\]

Pattern origin: Yao et al., ReAct: Synergizing Reasoning and Acting (2022). See ML intro §6 — Agentic AI.

The agent loop

Think → Act → Observe, repeat until the goal is met. Reasoning makes each think step smarter; tools make each act step real (run code, search, call an API, edit a file).

What an agent is made of

  • Planning — break a big goal into steps; reasoning models are good at this.
  • Tools — code runner, web search, file edits, APIs, other agents.
  • Memory — scratchpad + retrieved context (RAG) so it doesn’t forget.
  • Reflection — check its own output, retry on failure.
  • Environment — a sandbox, a browser, a repo, an OS.
  • Stopping — know when it’s done (or stuck).

Coding agents (Claude Code, Codex, Cursor) are the flagship example: read a repo, plan, edit files, run the tests, and fix what fails — an RLVR-style loop at use time.

The hard part: long-horizon tasks

One wrong step early can derail everything later. Over a 50-step task, small error rates compound:

  • 95% per-step reliability → only ~8% over 50 steps.
  • Agents need error recovery, not just raw skill.
  • Verification at each step (tests, checks) keeps them on track.

This is why reasoning + verifiable feedback matters so much for agents: the loop can catch and fix its own mistakes before they cascade.

7 · How capable? How fast? 🚀

A “Moore’s law” for agents

METR measures the length of task (in human-minutes) an agent finishes 50% of the time. That horizon has been doubling every ~4–7 months — from seconds of work to hours.

Source: METR, Measuring AI Ability to Complete Long Tasks (2025), updated 2026.

The frontier, mid-2026

A fast-moving snapshot — names and numbers change monthly:

Family Reasoning / “thinking” Notable strength
OpenAI GPT-5.6 (Sol / Terra / Luna tiers) coding, tool use, performance-per-$
Google Gemini 3.x Pro (thinking) leads many pure-reasoning benchmarks
Anthropic Claude Fable 5, Opus 4.x (extended thinking) long agentic runs, frontend/design
DeepSeek / Qwen / GLM R-series, Qwen, GLM (open) strong & cheap, open weights

Benchmarks like GPQA Diamond (hard science) and SWE-bench (real GitHub bugs) now sit around 80–95% — territory that was science-fiction in 2022. The new race is efficiency: GPT-5.6 Sol trades blows with Claude Fable 5 at roughly ¼–½ the cost and often less time.

Caveat: launch benchmarks are vendor-selected and self-reported — Fable 5 still leads e.g. SWE-bench Pro and frontend design. Verify with a neutral source before quoting.

8 · Limits, safety & cost ⚖️

What to keep in mind

Still fallible

  • Thinking reduces but doesn’t remove hallucination.
  • CoT can be unfaithful (§5).
  • Confidently wrong on things outside training.

Costs & risks

  • Compute: reasoning burns 5–50× the tokens.
  • Latency: seconds → minutes per answer.
  • Agents act — a bad step can delete files, spend money, send emails. Sandbox + human-in-the-loop.

Give agents the least privilege they need, keep a human approval step for irreversible actions, and log everything. Revisit ML intro §9 — prompt injection & ethics.

Recap

  • Reasoning models think in a chain-of-thought before answering — System 2, not System 1.
  • Test-time compute is a new scaling axis: think longer can beat be bigger — but watch overthinking.
  • They’re trained with RLVR — reward what you can verify (math answers, passing tests) — using GRPO; DeepSeek-R1 showed reasoning can emerge from pure RL, then be distilled into small, cheap models.
  • Agents = reasoning + tools + memory + a loop; they shine on long-horizon tasks and are improving fast (METR doubling).
  • Mind the limits: faithfulness, cost, latency, and the risk of agents that act — sandbox and supervise.

Where to go next

Try it

  • Toggle extended thinking / reasoning effort on a chat model and watch the trace.
  • Run a coding agent on a small repo; make it fix a failing test.
  • Reproduce GRPO on grade-school math (open recipes exist).

To read

References

Reasoning & test-time compute

Agents

Build it yourself

Continues the ML intro deck. Model names/scores are mid-2026 and change fast — verify before reuse.

Thank you!

Questions?

kasemsit.t@cmu.ac.th