Inside the Mind of an LLM: The Next-Word Machine

Inside the Mind of an LLM: The Next-Word Machine

If you type the word 'unbelievably' into an AI, it doesn't actually see the word the way you do. To a computer, raw human text is a messy, infinite wild west.

Instead, the AI's first step is to chop text into standardized puzzle pieces called tokens. Watch how 'unbelievably' gets sliced into three distinct subwords: 'un', 'believ', and 'ably'.

Each of these pieces is then mapped to a unique number in a massive dictionary. By using these subword tokens, the AI can easily understand millions of rare words without running out of memory.

Once words are chopped into token IDs, how does a computer actually understand them? It doesn't use definitions; instead, it translates each token into a unique coordinate in a giant, high-dimensional space.

In this space, similar concepts sit close together. A token like 'king' gets mapped to a specific coordinate, while 'queen' sits nearby, and 'apple' is placed far, far away.

While we can only draw two or three dimensions, LLMs use thousands of dimensions at once. This dense list of numbers is called an embedding, turning raw text into mathematical meaning.

If you see the word bank by itself, does it mean a river side, or a financial institution? To find out, your brain automatically looks at the surrounding words.

An LLM does this using self-attention. Every token vector acts like a little search engine, sending out queries to all other tokens to calculate a connection strength.

By pulling information from water and river, the vector for bank dynamically updates its values. It is no longer just a generic word; it gains a context-rich, mathematical meaning.