Before the model can predict,
it has to learn.
Training repeatedly measures error and adjusts parameters. The gradient tells us how changes to a parameter affect that error. Start with a tiny single-weight example.
Watch one weight learn.
We want a tiny model to predict the target 4 from the input 2. Its prediction is weight × input. Each gradient-descent step changes the weight to reduce squared error.
Next gradient: -16. Next weight: 1.60.
The model predicts 2 × weight, but the correct answer should be 4. Its mistake is measured by squared error. Gradient descent figures out which direction to change the weight. Pressing “Apply one gradient step” updates it, so you can watch the loss fall.
Research & verificationLearning through gradient descent
Publication review: Peer-reviewed scientific journal article.
Independent verification: Our training experiment is original scalar regression, not an experimental reproduction of the paper. Its math has unit tests, not external certification.
Peer review of a paper does not certify this demo.
Training real language models is vastly more complex: billions of examples, many parameter matrices, multiple layers, optimizers, and hardware. This exercise only shows the mechanism of updating one parameter toward a target.
Tokens don't live
in isolation.
Transformer attention lets each position combine information from other allowed positions. A simplified calculation can explain the core idea without pretending to reproduce a full transformer.
Which tokens get attention?
For one query vector, attention compares it with key vectors using dot products, scales them by √2, and applies softmax so the weights sum to 1. Change the query to see the distribution shift.
Attention(Q, K) = softmax(QKᵀ / √dₖ)
An attention mechanism compares a query to several keys. It gives bigger scores to keys that match the query more closely. Softmax turns those scores into weights that add to 100%. Here you can move a two-number query and see how the bars change.
Research & verificationScaled dot-product attention
Publication review: Peer-reviewed machine-learning conference paper.
Independent verification: Our query/key numbers are invented teaching values. The attention implementation has unit tests but has not been independently reviewed.
Peer review of a paper does not certify this demo.
Real transformers contain many attention heads across layers, with learned projections and often additional mechanisms. This demonstration only calculates one synthetic query against three fixed keys, without value vectors.
Enough diagrams.
Let's calculate.
This time the weights are genuine pretrained weights, the tokens are real GPT-2-family vocabulary IDs, and the probabilities come from a model rather than an invented example.
Meet a real language model.
This is DistilGPT-2, a smaller GPT-2-family model. It isn't a modern chat assistant. It predicts text continuations using real learned weights, and you can inspect the actual token IDs and its next-token probability distribution.
Load the model, then click “See real tokens” or generate a continuation to examine its vocabulary IDs.
These are real token IDs from DistilGPT-2's tokenizer. Click any token to inspect its original 768-dimensional embedding weights. Spaces are shown as ␣.
Click “Predict next token” to compare the top eight choices from the model's full vocabulary.
These probabilities come from a direct model forward pass at the final input position. Softmax is calculated over all 50,257 vocabulary tokens at temperature 1, before top-k filtering or random sampling. The eight shown usually do not total 100%.
This runs a real pretrained DistilGPT-2 on your computer, not a scripted chatbot. The tokenizer turns your text into IDs. The model assigns scores to possible next tokens, turns those scores into probabilities, and chooses tokens to continue the sentence. Temperature changes how random sampling behaves; it does not make answers more correct.
Research & verificationGenerative language modeling
Publication review: Public technical report; not a peer-reviewed conference or journal publication.
Independent verification: The inference panel uses the public DistilGPT-2 checkpoint and Transformers.js. Its browser integration is not independently audited or benchmark-certified.
Public DistilGPT-2 model card ↗Peer review of a paper does not certify this demo.
A note about limitations: DistilGPT-2 is a small, older, unaligned text-completion model, not an instruction-following assistant. It can produce inaccurate, biased or inappropriate content. Generations vary with settings. The node playground, attention explainer and this real model are different educational views, not one unified network.
Teach the process, not the hype.
DistilGPT-2 uses the GPT-2 architecture family with fewer layers. It can demonstrate genuine next-token computation without requiring a remote inference server. Its capability is intentionally limited compared with modern assistants.
Math lab vs. real inference.
Training and attention exercises have transparent, hand-picked numbers. The inference lab uses pretrained, quantized model weights. These labs explain related concepts, but you should not mistake their charts for actual internal DistilGPT-2 attention or hidden states.