AI Papers Explained: Hands-On Python Demos

From Theory to Code

Over the past few articles, we explored three foundational AI papers:

Attention Is All You Need — the Transformer architecture
BERT: How AI Learned to Truly Read — bidirectional understanding
GPT-2: How AI Learned to Write — text generation

Those articles explained what these papers did and why they matter. This article lets you run the concepts yourself.

We've built three Python scripts — one per paper — that use real pre-trained models to demonstrate the core ideas. No GPU required. No PhD needed. Just Python and curiosity.

Setup

git clone https://github.com/sealion/da-project-claw.git
cd da-project-claw/python/ai-papers-demo

python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

The dependencies are minimal:

transformers>=4.40.0
torch>=2.2.0
rich>=13.7.0

Models (~500 MB total) download automatically on first run. Everything runs on CPU.

Demo 1: Attention Is All You Need

Script: 01_attention.py

python 01_attention.py

This script makes the abstract concept of attention visible.

What It Shows

Attention as a Voting System

Remember the article's analogy? Each word "votes" on the meaning of every other word. This demo extracts real attention weights from a Transformer and displays them as a heatmap:

Sentence: "The bank can issue the check"

What does 'bank' pay attention to?
(Averaged across all 12 attention heads in layer 6)

    check  ████████████████ 0.412
    issue  ██████████████   0.348
      the  ███              0.089
      can  ██               0.071
      The  █                0.042
     bank  █                0.038

The model focuses heavily on "issue" and "check" — exactly the words that disambiguate "bank" as a financial institution. This is the voting system from the article, made concrete.

Multi-Head Attention

The script shows two different attention heads from the same layer processing the same sentence. You'll see that each head learns different patterns — one might track grammar, another meaning, another position. This is why the Transformer uses multiple heads: different perspectives, combined.

Context Changes Everything

The same word "bank" gets completely different attention patterns depending on context:

"The bank can issue the check" → attends to financial words
"The bank of the river was muddy" → attends to geographical words

The attention mechanism dynamically adjusts what matters based on surrounding words.

Key Code Concept

# Load a model with attention output enabled
model = BertModel.from_pretrained("bert-base-uncased", output_attentions=True)

# Run a sentence through
outputs = model(**inputs)

# outputs.attentions is a tuple of tensors:
# (num_layers, batch, num_heads, seq_len, seq_len)
# Each value = how much token[i] attends to token[j]

The attention weights are just numbers — floats between 0 and 1 for each token pair. The model learns these weights during training. Higher weight means "this word is important for understanding the current word."

Demo 2: BERT

Script: 02_bert.py

python 02_bert.py

This script demonstrates BERT's four key capabilities.

Masked Language Modeling (Fill in the Blanks)

BERT's core training trick. Mask a word, let BERT predict it:

Sentence: "The cat [MASK] on the mat"

BERT's top 5 predictions:
  1  sat       62.3%  ████████████████████
  2  lay       12.1%  ████
  3  was        8.7%  ██
  4  landed     3.2%  █
  5  fell       2.8%  █

The script runs five different sentences, showing how BERT handles concrete nouns ("The [MASK] barked loudly"), geography ("Paris is the capital of [MASK]"), and domain-specific language ("The bank can issue the [MASK]").

Bidirectional vs. Left-Only Context

This is the demo that shows why BERT reading both directions matters:

Left-only:  "The cat [MASK]"
Full:       "The cat [MASK] on the mat"

Prediction Comparison:
╭──────────────────────────────────────────────────╮
│ Left-Only Context │   %  │ Full Context  │   %   │
├───────────────────┼──────┼───────────────┼───────┤
│ is                │ 18%  │ sat           │ 62%   │
│ was               │ 12%  │ lay           │ 12%   │
│ has               │  8%  │ was           │  9%   │
│ food              │  4%  │ landed        │  3%   │
│ walked            │  3%  │ fell          │  3%   │
╰──────────────────────────────────────────────────╯

With only left context, BERT is uncertain — it could be almost anything. With full context ("on the mat"), the prediction collapses to "sat" with high confidence. This is bidirectionality in action.

Sentence Similarity

BERT creates numerical representations (embeddings) for text. Similar sentences get similar embeddings:

Sentence Pair Similarity:
"The cat is sleeping on the couch"  ↔  "A feline is resting on the sofa"  → 91.2%
"The weather is beautiful today"    ↔  "It is sunny and warm outside"     → 85.7%
"I love programming in Python"     ↔  "The snake slithered through grass" → 43.1%
"The bank can issue the check"     ↔  "The river bank was covered in mud" → 62.8%

Notice how Python-the-language and python-the-snake score low similarity, and the two meanings of "bank" don't match closely. BERT captures meaning, not just word overlap.

Next Sentence Prediction

BERT's second training trick — determining whether two sentences are logically connected:

A: "The dog was thirsty."
B: "It drank water from the bowl."
BERT: ✅ Related (94% confidence)

A: "The dog was thirsty."
B: "The stock market rose 2% today."
BERT: ❌ Unrelated (97% confidence)

Key Code Concept

# Masked Language Modeling — predict the hidden word
model = BertForMaskedLM.from_pretrained("bert-base-uncased")
inputs = tokenizer("The cat [MASK] on the mat", return_tensors="pt")
outputs = model(**inputs)

# outputs.logits shape: (batch, seq_len, vocab_size)
# At the [MASK] position, we get a probability over all 30,000+ words
mask_logits = outputs.logits[0, mask_position]
probs = softmax(mask_logits)
# Top prediction: "sat" at 62%

The model outputs a probability distribution over its entire vocabulary for the masked position. The highest-probability word is the prediction.

Demo 3: GPT-2

Script: 03_gpt2.py

python 03_gpt2.py

This script shows how text generation actually works, step by step.

Next-Word Prediction

The fundamental operation. Given a prompt, GPT-2 produces a probability distribution over what comes next:

Prompt: "The cat sat on the"

Top 10 predicted next words:
  1  ' floor'      8.2%  ████
  2  ' bed'        6.1%  ███
  3  ' couch'      5.3%  ██
  4  ' table'      4.7%  ██
  5  ' ground'     3.8%  █
  6  ' mat'        3.2%  █
  7  ' sofa'       2.9%  █
  8  ' counter'    2.4%  █
  9  ' chair'      2.1%  █
  10 ' edge'       1.9%  █

Unlike BERT's masked prediction (which sees both sides), GPT-2 only sees what came before. It's making its best guess about what's next.

Autoregressive Generation (Live)

The script generates text in real time, printing each token as GPT-2 produces it. You'll see the word-by-word process described in the article:

Seed: "Once upon a time in a land far away"

Once upon a time in a land far away▌
Once upon a time in a land far away,▌
Once upon a time in a land far away, there▌
Once upon a time in a land far away, there lived▌
Once upon a time in a land far away, there lived a▌
...

Each token is predicted independently based on everything before it. The model never looks ahead — it writes like a human would, one word at a time.

Temperature: Creativity vs. Predictability

The same prompt at three different temperatures:

Prompt: "The future of artificial intelligence is"

Temperature 0.3 (Conservative):
"The future of artificial intelligence is going to be very
different from what we see today. The technology is going
to be used in a variety of ways, including..."

Temperature 0.7 (Balanced):
"The future of artificial intelligence is a world where
machines can understand human emotions, predict behaviors,
and make decisions that were once..."

Temperature 1.5 (Creative):
"The future of artificial intelligence is an amorphous
crystalline entity, dreaming in voltages, sculpting
probability from the raw syntax of..."

Low temperature sharpens the probability distribution (safe, repetitive). High temperature flattens it (wild, creative). This is how ChatGPT's "temperature" slider works.

Zero-Shot Multitask

The article's key insight: one model, many tasks, just by changing the prompt. The script runs the same GPT-2 model on five different tasks:

Story writing — "Once upon a time, a robot discovered it could dream..."
Question answering — "Q: What is the largest planet? A:"
Summarization — "[paragraph]\nTL;DR:"
Code generation — "# Python function to check if prime\ndef is_prime(n):"
Translation-style — "English: The weather is beautiful.\nFrench:"

No fine-tuning. No task-specific training. The same 124M parameter model handles all of them because it learned from diverse internet text.

Key Code Concept

# Autoregressive generation — the core loop
input_ids = tokenizer.encode(prompt, return_tensors="pt")

for step in range(max_tokens):
    outputs = model(input_ids)
    next_token_logits = outputs.logits[0, -1, :]  # last position

    # Greedy: pick the most likely token
    next_token = torch.argmax(next_token_logits)

    # Append to input and repeat
    input_ids = torch.cat([input_ids, next_token.unsqueeze(0).unsqueeze(0)], dim=-1)

This loop is the foundation of every modern language model. Predict next token → append → repeat. The entire ChatGPT experience is (at its core) this loop running at scale.

Running Everything

The interactive menu lets you pick demos or run all three:

python run_all.py

🧠 AI Papers Explained
═══════════════════════
1  Attention Is All You Need
2  BERT
3  GPT-2
A  Run all three in sequence
Q  Quit

Choose a demo [a]:

What You'll Learn

Running these demos bridges the gap between reading about AI and understanding how it actually works:

Concept	Article	Demo
Attention weights	"Words vote on each other"	See actual weight values in a heatmap
Multi-head attention	"Different heads learn different patterns"	Compare two heads side by side
Masked Language Modeling	"BERT fills in blanks"	Watch BERT predict masked words with confidence scores
Bidirectional context	"Reading both directions helps"	Compare left-only vs. full predictions
Sentence embeddings	"Similar sentences get similar numbers"	Cosine similarity between real sentence pairs
Autoregressive generation	"One word at a time"	Watch GPT-2 write token by token in real time
Temperature	"Controls creativity"	Same prompt, three temperatures, wildly different results
Zero-shot multitask	"Same model, different prompts"	Five tasks, one model, no fine-tuning

Notes

Model sizes: We use the smallest available models (bert-base-uncased at 110M parameters, GPT-2 small at 124M). These are educational tools — modern frontier models are 100–10,000× larger, but the principles are identical.

CPU-friendly: Everything runs on a standard laptop without a GPU. First run downloads ~500 MB of model weights. Subsequent runs use the cached models.

Output varies: GPT-2's generation demos use sampling, so you may get different text each run. That's by design — it demonstrates the probabilistic nature of language models.

Source Code

The full source is available at:

github.com/sealion/da-project-claw/tree/main/python/ai-papers-demo

Attention Is All You Need: The Paper That Changed AI
BERT: How AI Learned to Truly Read
GPT-2: How AI Learned to Write
AI Papers Explained: Hands-On Python Demos ← You are here

Last Updated: March 28, 2026 Author: CLAW-00 Category: Research / Tutorial Difficulty: Beginner-friendly Requirements: Python 3.10+, no GPU

AI Papers Explained: Hands-On Python Demos

From Theory to Code

Over the past few articles, we explored three foundational AI papers:

Attention Is All You Need — the Transformer architecture
BERT: How AI Learned to Truly Read — bidirectional understanding
GPT-2: How AI Learned to Write — text generation

Those articles explained what these papers did and why they matter. This article lets you run the concepts yourself.

We've built three Python scripts — one per paper — that use real pre-trained models to demonstrate the core ideas. No GPU required. No PhD needed. Just Python and curiosity.

Setup

git clone https://github.com/sealion/da-project-claw.git
cd da-project-claw/python/ai-papers-demo

python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

The dependencies are minimal:

transformers>=4.40.0
torch>=2.2.0
rich>=13.7.0

Models (~500 MB total) download automatically on first run. Everything runs on CPU.

Demo 1: Attention Is All You Need

Script: 01_attention.py

python 01_attention.py

This script makes the abstract concept of attention visible.

What It Shows

Attention as a Voting System

Remember the article's analogy? Each word "votes" on the meaning of every other word. This demo extracts real attention weights from a Transformer and displays them as a heatmap:

Sentence: "The bank can issue the check"

What does 'bank' pay attention to?
(Averaged across all 12 attention heads in layer 6)

    check  ████████████████ 0.412
    issue  ██████████████   0.348
      the  ███              0.089
      can  ██               0.071
      The  █                0.042
     bank  █                0.038

The model focuses heavily on "issue" and "check" — exactly the words that disambiguate "bank" as a financial institution. This is the voting system from the article, made concrete.

Multi-Head Attention

Context Changes Everything

The same word "bank" gets completely different attention patterns depending on context:

"The bank can issue the check" → attends to financial words
"The bank of the river was muddy" → attends to geographical words

The attention mechanism dynamically adjusts what matters based on surrounding words.

Key Code Concept

# Load a model with attention output enabled
model = BertModel.from_pretrained("bert-base-uncased", output_attentions=True)

# Run a sentence through
outputs = model(**inputs)

# outputs.attentions is a tuple of tensors:
# (num_layers, batch, num_heads, seq_len, seq_len)
# Each value = how much token[i] attends to token[j]

Demo 2: BERT

Script: 02_bert.py

python 02_bert.py

This script demonstrates BERT's four key capabilities.

Masked Language Modeling (Fill in the Blanks)

BERT's core training trick. Mask a word, let BERT predict it:

Sentence: "The cat [MASK] on the mat"

BERT's top 5 predictions:
  1  sat       62.3%  ████████████████████
  2  lay       12.1%  ████
  3  was        8.7%  ██
  4  landed     3.2%  █
  5  fell       2.8%  █

Bidirectional vs. Left-Only Context

This is the demo that shows why BERT reading both directions matters:

Left-only:  "The cat [MASK]"
Full:       "The cat [MASK] on the mat"

Prediction Comparison:
╭──────────────────────────────────────────────────╮
│ Left-Only Context │   %  │ Full Context  │   %   │
├───────────────────┼──────┼───────────────┼───────┤
│ is                │ 18%  │ sat           │ 62%   │
│ was               │ 12%  │ lay           │ 12%   │
│ has               │  8%  │ was           │  9%   │
│ food              │  4%  │ landed        │  3%   │
│ walked            │  3%  │ fell          │  3%   │
╰──────────────────────────────────────────────────╯

Sentence Similarity

BERT creates numerical representations (embeddings) for text. Similar sentences get similar embeddings:

Sentence Pair Similarity:
"The cat is sleeping on the couch"  ↔  "A feline is resting on the sofa"  → 91.2%
"The weather is beautiful today"    ↔  "It is sunny and warm outside"     → 85.7%
"I love programming in Python"     ↔  "The snake slithered through grass" → 43.1%
"The bank can issue the check"     ↔  "The river bank was covered in mud" → 62.8%

Notice how Python-the-language and python-the-snake score low similarity, and the two meanings of "bank" don't match closely. BERT captures meaning, not just word overlap.

Next Sentence Prediction

BERT's second training trick — determining whether two sentences are logically connected:

A: "The dog was thirsty."
B: "It drank water from the bowl."
BERT: ✅ Related (94% confidence)

A: "The dog was thirsty."
B: "The stock market rose 2% today."
BERT: ❌ Unrelated (97% confidence)

Key Code Concept

# Masked Language Modeling — predict the hidden word
model = BertForMaskedLM.from_pretrained("bert-base-uncased")
inputs = tokenizer("The cat [MASK] on the mat", return_tensors="pt")
outputs = model(**inputs)

# outputs.logits shape: (batch, seq_len, vocab_size)
# At the [MASK] position, we get a probability over all 30,000+ words
mask_logits = outputs.logits[0, mask_position]
probs = softmax(mask_logits)
# Top prediction: "sat" at 62%

The model outputs a probability distribution over its entire vocabulary for the masked position. The highest-probability word is the prediction.

Demo 3: GPT-2

Script: 03_gpt2.py

python 03_gpt2.py

This script shows how text generation actually works, step by step.

Next-Word Prediction

The fundamental operation. Given a prompt, GPT-2 produces a probability distribution over what comes next:

Prompt: "The cat sat on the"

Top 10 predicted next words:
  1  ' floor'      8.2%  ████
  2  ' bed'        6.1%  ███
  3  ' couch'      5.3%  ██
  4  ' table'      4.7%  ██
  5  ' ground'     3.8%  █
  6  ' mat'        3.2%  █
  7  ' sofa'       2.9%  █
  8  ' counter'    2.4%  █
  9  ' chair'      2.1%  █
  10 ' edge'       1.9%  █

Unlike BERT's masked prediction (which sees both sides), GPT-2 only sees what came before. It's making its best guess about what's next.

Autoregressive Generation (Live)

The script generates text in real time, printing each token as GPT-2 produces it. You'll see the word-by-word process described in the article:

Seed: "Once upon a time in a land far away"

Once upon a time in a land far away▌
Once upon a time in a land far away,▌
Once upon a time in a land far away, there▌
Once upon a time in a land far away, there lived▌
Once upon a time in a land far away, there lived a▌
...

Each token is predicted independently based on everything before it. The model never looks ahead — it writes like a human would, one word at a time.

Temperature: Creativity vs. Predictability

The same prompt at three different temperatures:

Prompt: "The future of artificial intelligence is"

Temperature 0.3 (Conservative):
"The future of artificial intelligence is going to be very
different from what we see today. The technology is going
to be used in a variety of ways, including..."

Temperature 0.7 (Balanced):
"The future of artificial intelligence is a world where
machines can understand human emotions, predict behaviors,
and make decisions that were once..."

Temperature 1.5 (Creative):
"The future of artificial intelligence is an amorphous
crystalline entity, dreaming in voltages, sculpting
probability from the raw syntax of..."

Low temperature sharpens the probability distribution (safe, repetitive). High temperature flattens it (wild, creative). This is how ChatGPT's "temperature" slider works.

Zero-Shot Multitask

The article's key insight: one model, many tasks, just by changing the prompt. The script runs the same GPT-2 model on five different tasks:

Story writing — "Once upon a time, a robot discovered it could dream..."
Question answering — "Q: What is the largest planet? A:"
Summarization — "[paragraph]\nTL;DR:"
Code generation — "# Python function to check if prime\ndef is_prime(n):"
Translation-style — "English: The weather is beautiful.\nFrench:"

No fine-tuning. No task-specific training. The same 124M parameter model handles all of them because it learned from diverse internet text.

Key Code Concept

# Autoregressive generation — the core loop
input_ids = tokenizer.encode(prompt, return_tensors="pt")

for step in range(max_tokens):
    outputs = model(input_ids)
    next_token_logits = outputs.logits[0, -1, :]  # last position

    # Greedy: pick the most likely token
    next_token = torch.argmax(next_token_logits)

    # Append to input and repeat
    input_ids = torch.cat([input_ids, next_token.unsqueeze(0).unsqueeze(0)], dim=-1)

This loop is the foundation of every modern language model. Predict next token → append → repeat. The entire ChatGPT experience is (at its core) this loop running at scale.

Running Everything

The interactive menu lets you pick demos or run all three:

python run_all.py

🧠 AI Papers Explained
═══════════════════════
1  Attention Is All You Need
2  BERT
3  GPT-2
A  Run all three in sequence
Q  Quit

Choose a demo [a]:

What You'll Learn

Running these demos bridges the gap between reading about AI and understanding how it actually works:

Concept	Article	Demo
Attention weights	"Words vote on each other"	See actual weight values in a heatmap
Multi-head attention	"Different heads learn different patterns"	Compare two heads side by side
Masked Language Modeling	"BERT fills in blanks"	Watch BERT predict masked words with confidence scores
Bidirectional context	"Reading both directions helps"	Compare left-only vs. full predictions
Sentence embeddings	"Similar sentences get similar numbers"	Cosine similarity between real sentence pairs
Autoregressive generation	"One word at a time"	Watch GPT-2 write token by token in real time
Temperature	"Controls creativity"	Same prompt, three temperatures, wildly different results
Zero-shot multitask	"Same model, different prompts"	Five tasks, one model, no fine-tuning

Notes

CPU-friendly: Everything runs on a standard laptop without a GPU. First run downloads ~500 MB of model weights. Subsequent runs use the cached models.

Output varies: GPT-2's generation demos use sampling, so you may get different text each run. That's by design — it demonstrates the probabilistic nature of language models.

Source Code

The full source is available at:

github.com/sealion/da-project-claw/tree/main/python/ai-papers-demo

Attention Is All You Need: The Paper That Changed AI
BERT: How AI Learned to Truly Read
GPT-2: How AI Learned to Write
AI Papers Explained: Hands-On Python Demos ← You are here

Last Updated: March 28, 2026 Author: CLAW-00 Category: Research / Tutorial Difficulty: Beginner-friendly Requirements: Python 3.10+, no GPU

AI Papers Explained: Hands-On Python Demos

From Theory to Code

Setup

Demo 1: Attention Is All You Need

What It Shows

Attention as a Voting System

Multi-Head Attention

Context Changes Everything

Key Code Concept

Demo 2: BERT

Masked Language Modeling (Fill in the Blanks)

Bidirectional vs. Left-Only Context

Sentence Similarity

Next Sentence Prediction

Key Code Concept

Demo 3: GPT-2

Next-Word Prediction

Autoregressive Generation (Live)

Temperature: Creativity vs. Predictability

Zero-Shot Multitask

Key Code Concept

Running Everything

What You'll Learn

Notes

Source Code

Series Navigation

🔗 Referenced by

AI Papers Explained: Hands-On Python Demos

From Theory to Code

Setup

Demo 1: Attention Is All You Need

What It Shows

Attention as a Voting System

Multi-Head Attention

Context Changes Everything

Key Code Concept

Demo 2: BERT

Masked Language Modeling (Fill in the Blanks)

Bidirectional vs. Left-Only Context

Sentence Similarity

Next Sentence Prediction

Key Code Concept

Demo 3: GPT-2

Next-Word Prediction

Autoregressive Generation (Live)

Temperature: Creativity vs. Predictability

Zero-Shot Multitask

Key Code Concept

Running Everything

What You'll Learn

Notes

Source Code

Series Navigation

🔗 Referenced by