Organizing a Kitchen Table#
When trying to explain different AI concepts to my family, one concept that never landed was how words turn into math. They understood “LLMs are predicting the next word” but not the mechanics of how a word can be represented as a number. This is an analogy I found helpful.
The Table#
Picture a table covered in a random pile of ingredients — leafy greens, spices, meats, grains, bread, pasta, herbs, rice, tomatoes, roots, potatoes, seafood, flour.
Ask someone to organize it and, with no other instructions, patterns show up: greens near herbs, herbs near spices, flour near spices because both are powdery, potatoes near the roots, seafood near the meat, pasta near the potatoes. Ask a thousand people and you’d get a thousand slightly different tables, but the same clusters keep showing up. The process of organizing is “encoding” what’s similar and what’s different.
Coordinates#
Draw a grid on the table. 14 squares wide, 6 squares tall, say the bottom-left corner is (1, 1) and so on. Now every ingredient has “coordinates” assigned:
- chicken: (12, 5)
- steak: (13, 4)
- flour: (7, 5)
- kale: (2, 2)
Chicken isn’t just a word anymore, it has a meaningful location. Chicken and steak sit close because they’re both meat. This location, the coordinates themselves, is called an “embedding.”
The table in this analogy has two dimensions but these run out fast; chicken is also close to eggs, same animal, and pulling it toward eggs pulls it away from steak. You can’t satisfy both in two dimensions. By adding more dimensions the relationships stop fighting each other: chicken can be close to steak on one axis and close to eggs on another at the same time.
This table, which becomes multi-dimensional, is referred to as a “latent” space and it’s within a coordinate system like this that an LLM operates.
Feed a model a sentence, it maps every word to a location like this, does math on the locations, and reads back the location of whatever comes next.