Inside the Mind of an LLM: The Next-Word Machine

Inside the Mind of an LLM: The Next-Word Machine

If you type the word 'unbelievably' into an AI, it doesn't actually see the word the way you do. To a computer, raw human text is a messy, infinite wild west.

Instead, the AI's first step is to chop text into standardized puzzle pieces called tokens. Watch how 'unbelievably' gets sliced into three distinct subwords: 'un', 'believ', and 'ably'.

Each of these pieces is then mapped to a unique number in a massive dictionary. By using these subword tokens, the AI can easily understand millions of rare words without running out of memory.