Seeing how language models learn by training a tiny one.

The problem

It's hard to build intuition for language models when the ones you use have billions of parameters. A model small enough to train in seconds makes every step visible: tokens, vectors, probabilities, and updates.

What I did

• Predicted each result in writing before every run, then compared.
• Wrote 355 new teaching passages on tense and agreement, opposites, and spatial relations, without reading the test items.
• Trained two experiments with identical settings: 3,000 steps at a learning rate of 0.001, on a laptop CPU.
• Traced one token all the way through: word to ID to vector to probability to gradient to update.
• Built a small chat interface and was clear about what it is: a sentence continuer, not a chatbot.

Results

What I'd do differently

The model learned templates, not language. And because the public tests guided my choices, a good score doesn't prove it generalizes. Next: write my own held-out test set and never look at it while building the corpus.

Artifacts

Write-up and evidence on GitHub ↗