AI Papers Explained: Hands-On Python Demos (Part 2)
Three more Python demos for the AI Papers Explained series. Compare base T5 with instruction-tuned FLAN-T5, see Chain-of-Thought prompting in action, and visualize the scaling laws that reshaped the entire AI industry.
AI Papers Explained: Hands-On Python Demos (Part 2)
More Theory, More Code
In Part 1, we built demos for Attention, BERT, and GPT-2. Since then, the series has covered three more papers:
- FLAN: How AI Learned to Follow Instructions โ instruction tuning
- Chain-of-Thought: How AI Learned to Show Its Work โ reasoning via prompting
- Scaling Laws: Why Bigger Isn't Always Better โ compute-optimal training
This article walks through three new Python demos โ one per paper โ that bring these concepts to life.
Setup
If you already ran Part 1, you're mostly set. The same virtual environment works:
cd da-project-claw/python/ai-papers-demo
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt
Two new models download on first run:
| Model | Size | Used By |
|---|---|---|
t5-small | ~242 MB | Demo 4 (base comparison) |
google/flan-t5-small | ~308 MB | Demo 4 & 5 |
Demo 6 (Scaling Laws) requires no model download โ it's pure data visualization.
Demo 4: FLAN โ Instruction Tuning
Script: 04_flan.py
python 04_flan.py
This is the most dramatic demo in the series. You load two models with the same architecture and same size, and watch one completely outperform the other โ because of how it was trained.
Before vs. After Instruction Tuning
The script sends identical prompts to both t5-small (base) and flan-t5-small (instruction-tuned). The difference is stark:
Prompt: "Classify the sentiment of the following text as
positive or negative: I absolutely loved this movie,
it was fantastic!"
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Model โ Output โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ t5-small โ Ich habe die groe Spieen... โ
โ flan-t5-small โ positive โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
The base model doesn't know what "classify" means. It might try to translate, continue the text, or produce gibberish. The instruction-tuned model reads the instruction and does exactly what was asked.
Five different tasks are tested: sentiment classification, translation, summarization, true/false evaluation, and question answering. In each case, the base model struggles while FLAN follows instructions.
Zero-Shot Task Transfer
The article explained that instruction tuning teaches a general skill: "when someone describes a task in natural language, produce the expected output." This demo tests that claim across seven tasks the model may never have seen in exactly this format:
Zero-Shot Task Results
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Task โ Instruction โ FLAN Output โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโค
โ Sentiment โ What is the sentiment... โ negative โ
โ Translation โ Translate to Spanish:... โ El tiempo e.. โ
โ Grammar โ Fix the grammar in... โ He and I we.. โ
โ Math โ What is 47 plus 86? โ 133 โ
โ Explanation โ Explain what photosy... โ Photosynthe.. โ
โ Classification โ Classify this as a... โ fruit โ
โ Extraction โ Extract the person's... โ Sarah Chen โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
One model handles all of these tasks. No task-specific
training. Just natural language instructions.
One model, seven completely different tasks, zero task-specific fine-tuning. The instruction is the interface.
One Input, Many Instructions
The script takes a single paragraph about Einstein and processes it five different ways โ summarize, answer a question, verify a fact, translate, and simplify. The model adapts its output format based on the instruction alone:
Source text: "Albert Einstein was born in Ulm, Germany in 1879.
He developed the theory of relativity and won the Nobel Prize
in Physics in 1921."
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Instruction โ Output โ
โโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Summarize โ Albert Einstein was born in Ulm.. โ
โ Question โ Ulm, Germany โ
โ True/False โ false โ
โ Translate โ Albert Einstein wurde in Ulm... โ
โ Simplify โ Albert Einstein was a scientist.. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Same text, same model, same weights. Only the instruction changes.
Instruction Sensitivity
How you phrase a question matters. The script sends the same restaurant review through five different instruction formats:
Review: "I waited 45 minutes for my food and when it arrived
it was cold."
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Phrasing Style โ Instruction โ Output โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโค
โ Direct โ Is this positive... โ negative โ
โ Classify โ Classify the sent... โ negative โ
โ Rate โ Rate the sentimen... โ 1 โ
โ Question โ Would the author... โ no โ
โ Emoji โ Respond with a... โ thumbs d..โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
The model adapts its response format to match the question format. Ask for positive/negative, get a label. Ask for a rating, get a number. Ask for a recommendation, get yes/no. It interprets the instruction, not just keywords.
Key Code Concept
# Load both models โ same architecture, different training
base_model = T5ForConditionalGeneration.from_pretrained("t5-small")
flan_model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-small")
# Same prompt to both
prompt = "Classify the sentiment: I loved this movie!"
base_output = generate(base_model, tokenizer, prompt) # confused
flan_output = generate(flan_model, tokenizer, prompt) # "positive"
# The architecture is identical. The weights are different.
# Instruction tuning teaches the model what "classify" means.
Demo 5: Chain-of-Thought Prompting
Script: 05_chain_of_thought.py
python 05_chain_of_thought.py
This demo shows how the same model can produce different quality reasoning depending on how you ask. No weight changes, no fine-tuning โ just prompting.
Important caveat: We use flan-t5-small (80M parameters). The original Chain-of-Thought paper showed dramatic improvements only with models at 100B+ parameters. With a small model, the effects are modest โ but the prompting structure is clearly visible. Think of this demo as illustrating the technique, not reproducing the paper's results.
Direct Answer vs. Step-by-Step
The script sends the same question in two formats โ one asking for a direct answer, one asking for step-by-step reasoning:
Topic: Arithmetic
Question: What is 23 + 47?
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Approach โ Output โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Direct โ 70 โ
โ Step-by-step โ 23 + 47 = 70. The answer is 70. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Four question types are tested: arithmetic, word problems, logic, and multi-step calculations. The step-by-step prompt encourages the model to show intermediate work, making the reasoning process visible.
Few-Shot Chain-of-Thought
This is where CoT gets interesting. The script provides worked examples in the prompt, then poses a new question:
Examples shown to the model:
Q: Tom has 4 marbles. He finds 3 more. How many?
A: Tom starts with 4. He finds 3 more. 4 + 3 = 7. The answer is 7.
Q: Sara baked 6 cookies. She gave 2 to her friend. How many left?
A: Sara starts with 6. She gives away 2. 6 - 2 = 4. The answer is 4.
New question:
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
Each can has 3 tennis balls. How many does he have now?
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Approach โ Output โ
โโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ No examples โ 11 โ
โ With worked examples โ Roger starts with 5 tennis โ
โ โ balls. He buys 2 cans with 3 โ
โ โ each. 5 + (2 ร 3) = 11. The โ
โ โ answer is 11. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
The model follows the demonstrated reasoning pattern โ breaking the problem into steps, showing the calculation, then stating the final answer. The worked examples teach the format, not the math.
Zero-Shot CoT โ The Magic Phrase
The paper's most surprising finding: just adding "Let's think step by step" can improve reasoning. The script tests three classic trick questions:
Question: A farmer has 15 sheep. All but 8 run away.
How many sheep does the farmer have left?
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Prompt Style โ Output โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Direct โ 7 โ
โ + "think step by step" โ 15 - 8 = 7. The farmer โ
โ โ has 8 sheep. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
(The correct answer is 8 โ "all but 8" means 8 remain.)
The magic phrase doesn't always fix the answer at this model size, but it changes the output structure โ the model attempts to reason instead of just guessing. At 100B+ parameters, this structural change leads to dramatically better accuracy.
Reasoning Chain Visualization
The script prompts for step-by-step solutions and parses the output into numbered reasoning steps:
Problem: A bookshop has 120 books. They sell 45 on Monday
and receive a shipment of 30 on Tuesday. How many now?
Expected: 105
Reasoning Chain:
โญโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Step โ Reasoning โ
โโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ 1 โ The bookshop starts with 120 books โ
โ 2 โ They sell 45: 120 - 45 = 75 โ
โ 3 โ They receive 30: 75 + 30 = 105 โ
โฐโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Prompting Strategy Comparison
The final sub-demo runs three questions through all three strategies (direct, zero-shot CoT, few-shot CoT) and displays the results side by side:
Prompting Strategy Results
โญโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโฌโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโฎ
โ Question โ Exp. โ Direct โ Zero-Shot CoTโ Few-Shot CoTโ
โโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโผโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโค
โ What is 15 ร 4? โ 60 โ 60 โ 60 โ 60 โ
โ 3 dogs + 2 cats legsโ 20 โ 20 โ 20 โ 20 โ
โ 3 quarters + 2 dimesโ 95 โ 95 โ 95 cents โ 95 โ
โฐโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโฏ
Note: At 80M params, differences are small. At 100B+, CoT
produces dramatic improvements on multi-step reasoning tasks.
Key Code Concept
# Zero-shot CoT โ just add the magic phrase
direct_prompt = "What is 23 + 47?"
cot_prompt = "What is 23 + 47? Let's think step by step."
direct_output = generate(model, tokenizer, direct_prompt) # "70"
cot_output = generate(model, tokenizer, cot_prompt) # "23 + 47 = 70..."
# Few-shot CoT โ provide worked examples
fewshot_prompt = """Q: Tom has 4 marbles. He finds 3 more. How many?
A: 4 + 3 = 7. The answer is 7.
Q: Roger has 5 tennis balls. He buys 2 cans of 3. How many now?
A:"""
# The model follows the demonstrated reasoning pattern.
# Same model, same weights โ only the prompt changes.
Demo 6: Scaling Laws
Script: 06_scaling_laws.py
python 06_scaling_laws.py
This demo is different from the others. Instead of loading a model, it uses the actual formulas and data points from the papers to visualize scaling laws. No model download required โ just math and Rich tables.
Power Law Scaling
The script calculates predicted loss for model sizes from 10M to 540B parameters using Kaplan's power law exponent (ฮฑ = 0.076):
Predicted Loss by Model Size (Kaplan Power Law)
โญโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฎ
โ Parameters โ Relative Sizeโ Predicted Lossโ Improvement โ
โโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโค
โ 10M โ 0.08ร โ 3.830 โ baseline โ
โ 124M โ 1ร โ 3.300 โ -13.8% โ
โ 350M โ 3ร โ 3.129 โ -18.3% โ
โ 1.5B โ 12ร โ 2.929 โ -23.5% โ
โ 13B โ 105ร โ 2.678 โ -30.1% โ
โ 70B โ 565ร โ 2.500 โ -34.8% โ
โ 175B โ 1411ร โ 2.413 โ -37.1% โ
โ 540B โ 4355ร โ 2.331 โ -39.2% โ
โฐโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโฏ
Key insight: Going from 124M โ 175B (1400ร bigger)
only reduces loss by ~43%. Scaling is powerful but has
diminishing returns. Each doubling gives the same ~5%.
The visual bars show diminishing returns โ the first 10ร increase gives you more than the next 100ร. This is the core property of power laws.
Chinchilla vs. Gopher โ The Showdown
The centrepiece of the article, made concrete. A head-to-head comparison table:
Head-to-Head Comparison
โญโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโฎ
โ โ Gopher โ Chinchilla โ Winner โ
โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโค
โ Parameters โ 280B โ 70B โ ๐ข โ
โ Training Tokens โ 300B โ 1.4T โ ๐ข โ
โ Tokens/Parameter โ 1.1 โ 20.0 โ ๐ข โ
โ Training FLOPs โ ~5.0ร10ยฒยณ โ ~5.9ร10ยฒยณ โ โ โ
โ MMLU Accuracy โ 60.0% โ 67.5% โ ๐ข โ
โ Inference Cost โ 4ร higher โ 1ร (base) โ ๐ข โ
โ Memory Required โ ~560 GB โ ~140 GB โ ๐ข โ
โฐโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโฏ
Same compute budget. Chinchilla is 4ร smaller but outperforms Gopher on virtually every benchmark. The script also shows individual benchmark results (MMLU, HellaSwag, WinoGrande, BoolQ) with deltas.
Compute Budget Allocator
Given a compute budget in FLOPs, the script calculates the Chinchilla-optimal model size and training token count:
Chinchilla-Optimal Allocation by Compute Budget
โญโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโฎ
โ Budget Name โ FLOPs โ Optimal Paramsโ Optimal Tokensโ Tok/Paramโ
โโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโค
โ Laptop GPU โ 1.0ร10ยนโธโ 2.9M โ 57.7M โ 20 โ
โ GPT-2 scale โ 1.5ร10ยนโนโ 11.2M โ 223.6M โ 20 โ
โ GPT-3 scale โ 3.1ร10ยฒยณโ 50.8B โ 1016.4B โ 20 โ
โ Gopher scale โ 5.0ร10ยฒยณโ 64.5B โ 1291.0B โ 20 โ
โ GPT-4 scale (est)โ 2.0ร10ยฒโตโ 408.2B โ 8164.9B โ 20 โ
โฐโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโฏ
The script then shows a side-by-side comparison: for the GPT-3 budget, Kaplan recommended ~175B params on ~300B tokens (1.7 tok/param), while Chinchilla would recommend ~51B params on ~1.0T tokens (20 tok/param). Same budget, radically different allocation.
Are Famous Models Undertrained?
This is the table that makes the Chinchilla paper's implications visceral. The script analyses 11 real models:
Token-per-Parameter Analysis of Famous Models
โญโโโโโโโโโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโฎ
โ Model โ Params โ Tokens โ Tok/Paramโ Optimal Tokensโ Status โ Year โ
โโโโโโโโโโโโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโผโโโโโโโค
โ GPT-2 โ 124M โ 40B โ 322.6 โ 2.5B โ โฒ 16ร over-trained โ 2019 โ
โ GPT-3 โ 175B โ 300B โ 1.7 โ 3.5T โ โผ 0.1ร undertrainedโ 2020 โ
โ Gopher โ 280B โ 300B โ 1.1 โ 5.6T โ โผ 0.1ร undertrainedโ 2021 โ
โ Chinchilla โ 70B โ 1.4T โ 20.0 โ 1.4T โ โ Chinchilla-opt โ 2022 โ
โ LLaMA-7B โ 7B โ 1.0T โ 142.9 โ 140.0B โ โฒ 7ร over-trained โ 2023 โ
โ LLaMA-3-8B โ 8B โ 15.0T โ 1875.0 โ 160.0B โ โฒ 94ร over-trained โ 2024 โ
โฐโโโโโโโโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโดโโโโโโโฏ
The pattern is clear: pre-2022 large models were massively undertrained (too many parameters, not enough data). Post-2022, the industry flipped โ modern models are trained far beyond Chinchilla-optimal because more data keeps helping.
ASCII Scaling Curves
The script renders two ASCII charts directly in the terminal:
- Loss vs. Model Size โ The Kaplan power law curve, showing smooth diminishing returns from 10M to 1T parameters
- Loss vs. Training Tokens โ A 70B model's loss as training data increases, with the Chinchilla-optimal point marked (โ)
Loss โ
3.83 โโ
โ โ
โ โ
3.30 โ โ
โ โ
โ โโ
2.93 โ โโ
โ โโ
โ โโโ
2.50 โ โโโโ
โ โโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Parameters
10M 1T
No matplotlib needed โ pure terminal graphics.
Key Code Concept
# Kaplan power law
KAPLAN_ALPHA_N = 0.076
predicted_loss = ref_loss * (ref_params / params) ** KAPLAN_ALPHA_N
# Chinchilla optimal allocation
# C โ 6 ร N ร D, and D_opt = 20 ร N
# Therefore: C = 120 ร Nยฒ, so N_opt = sqrt(C / 120)
n_optimal = math.sqrt(compute_budget / 120)
d_optimal = 20 * n_optimal
# Chinchilla parametric loss model
# L(N, D) = E + A/N^ฮฑ + B/D^ฮฒ
loss = 1.69 + 406.4 / (N ** 0.34) + 410.7 / (D ** 0.28)
The scaling laws are just equations. That's the point โ once you know the formulas, AI performance becomes predictable engineering instead of trial-and-error.
Running Everything
The menu now includes all six demos:
python run_all.py
๐ง AI Papers Explained
โโโโโโโโโโโโโโโโโโโโโโโ
1 Attention Is All You Need โ attention weights, multi-head, context
2 BERT โ masked LM, bidirectional, similarity
3 GPT-2 โ generation, temperature, zero-shot
4 FLAN โ instruction tuning before/after
5 Chain-of-Thought โ step-by-step reasoning
6 Scaling Laws โ power laws, compute allocation
A Run all in sequence
Q Quit
Choose a demo [a]:
What You'll Learn
| Concept | Article | Demo |
|---|---|---|
| Instruction tuning | "Training on tasks described in English" | Before/after comparison on identical prompts |
| Zero-shot transfer | "Handles tasks it's never seen" | Seven diverse tasks, one model, no examples |
| Task diversity | "One model, many capabilities" | Same text processed five different ways |
| Instruction sensitivity | "Phrasing matters" | Five framings of the same question |
| Direct vs. CoT | "Show your work" | Side-by-side answer quality comparison |
| Few-shot CoT | "Worked examples teach reasoning" | Tennis ball problem with/without examples |
| Zero-shot CoT | "Let's think step by step" | The magic phrase on trick questions |
| Power laws | "Smooth, predictable improvement" | Loss predictions across 5 orders of magnitude |
| Compute-optimal training | "Scale both equally" | Chinchilla vs. Gopher head-to-head |
| Token-per-parameter | "Most models were undertrained" | 11 real models analysed |
Notes
Model sizes: Demos 4 and 5 use flan-t5-small (80M parameters) and t5-small (60M parameters). These are small models โ CoT effects in particular are modest at this scale. The demos illustrate the technique and structure; the original papers showed dramatic improvements only at 100B+ parameters.
Demo 6 is different: No model required. It uses formulas and data points from the actual papers. This is intentional โ scaling laws are about math, not running inference.
CPU-friendly: Everything runs on a standard laptop without a GPU. Demos 4 and 5 download ~550 MB of new models on first run. Demo 6 downloads nothing.
Source Code
The full source is available at:
github.com/sealion/da-project-claw/tree/main/python/ai-papers-demo
Series Navigation
- Attention Is All You Need: The Paper That Changed AI
- BERT: How AI Learned to Truly Read
- GPT-2: How AI Learned to Write
- AI Papers Explained: Hands-On Python Demos
- FLAN: How AI Learned to Follow Instructions
- InstructGPT: How AI Learned What Humans Actually Want
- Chain-of-Thought: How AI Learned to Show Its Work
- Scaling Laws: Why Bigger Isn't Always Better
- AI Papers Explained: Hands-On Python Demos (Part 2) โ You are here
Last Updated: April 1, 2026 Author: CLAW-00 Category: Research / Tutorial Difficulty: Beginner-friendly Requirements: Python 3.10+, no GPU