Back to journal Learning How Transformers Work with PyTorch

Learning How Transformers Work with PyTorch

A practical study note on how token representations, attention, and sequence order fit together in a transformer.

What I studied

I used a small PyTorch exercise to make the main transformer ideas concrete. Rather than treating the model as one mysterious block, I followed the path from a sequence of tokens to vector representations, attention, and a prediction.

The exercise was for learning. It was not a production model or a trained language model, and the code below is a small illustration of attention rather than a complete transformer implementation.

Tokens and order

Text first becomes tokens: pieces of text represented by integer IDs. An embedding table maps each ID to a vector the network can process. Those vectors carry information about the token, while positional information lets the model distinguish “the cat chased the dog” from “the dog chased the cat.” The same words in a different order can describe a different event.

A transformer block updates these vectors using attention and a feed-forward network. Residual paths and normalization help the representations pass through many such updates. These components work together; attention is useful, but it is only one part of the block.

Self-attention

Self-attention lets each token build a new representation using information from other tokens in the sequence. The query asks what this position needs, keys describe available positions, and values carry the information to combine. Similarity scores between queries and keys become weights for the values.

Scaled dot-product attention: softmax(QKᵀ / √dₖ)V

The scale keeps the scores in a range where the softmax can produce useful weights. In a decoder-only language model, a causal mask also prevents a token from looking ahead at future tokens when predicting the next one.

A small PyTorch step

The core calculation can be expressed in a few tensor operations. This snippet omits batching details, masking, multiple attention heads, and the surrounding transformer block:

import math
import torch

scores = query @ key.transpose(-2, -1) / math.sqrt(key.size(-1))
weights = torch.softmax(scores, dim=-1)
context = weights @ value

Following the shapes through these operations was more helpful than memorizing the formula: the scores compare positions, the weights say how much to use from each one, and the weighted values form the context passed onward.

What I learned

Writing a small implementation made the data flow easier for me to reason about. The most useful shift was to follow the dimensions and intermediate tensors through the model, rather than treating the architecture as a list of new terms. This remains a learning note, not a claim of production-scale model training.

STUDY NOTE

A short learning project exploring the main parts of a transformer.