How does an LLM actually work?

It predicts one token at a time. Given the text so far, the model produces a probability distribution over every possible next token, samples one, appends it, and repeats — feeding its own output back in until the response is complete. There is no lookup of stored answers; the reply is generated token by token from patterns learned during training.

Why do LLMs hallucinate?

Because a model predicts plausible-sounding text, not verified truth. It has no built-in sense of whether a statement is correct — only of how likely the words are to follow, given its training. When the most probable continuation happens to be wrong, the model states it just as confidently as a fact. Grounding answers in retrieved sources (retrieval-augmented generation) is the common way to reduce this.

A token is the chunk of text a model reads and writes in — roughly a word-piece. Common words are often a single token, while longer or rarer words are split into several. Models don't operate on letters or whole words directly; they turn text into tokens, predict tokens, and turn tokens back into text. Both the input a model can hold (its context window) and the cost of using it are measured in tokens.

Glossary · How AI reads content

Large Language ModelLLM

An AI model trained on vast amounts of text to predict the next token, which lets it understand and generate human-like language.

Updated August 20266 min readReviewed by PageLens.ai

How a language model predicts

The text so far

“The best CRM for a small team is ▮”

predict the next token

Probability of each next token

yourbrand0.42
Acme0.28
Salesforce0.19
…0.11

At its core an LLM predicts the next token — sampling from a probability distribution over what comes next.

A large language model (LLM) is the neural network behind today’s chatbots and AI search. Trained on vast amounts of text, it does one deceptively simple thing extremely well: given the words so far, it predicts the next one — over and over — to produce fluent, human-like language.

What an LLM actually is

An LLM is a neural network — almost always a transformer, the architecture introduced in the 2017 paper “Attention Is All You Need” — trained on very large amounts of text. The models you interact with by name are all LLMs: OpenAI’s GPT, Google’s Gemini, Anthropic’s Claude, Meta’s Llama.

Training is self-supervised: no one hand-labels the data. The model learns by repeatedly predicting the next token across enormous corpora of text, and correcting itself when it’s wrong. Through billions of these small predictions it absorbs grammar, facts, and patterns — not as a database it looks things up in, but as weights that shape what it predicts next.

How it generates text

At inferencetime — when it’s actually answering you — the model produces text one token at a time. From the text so far it computes a probability distribution over every possible next token, samples one from that distribution, appends it, and feeds the whole thing back in to predict the token after that. Fluent paragraphs are just this loop, run hundreds of times.

The vocabulary, defined plainly

Token. The chunk of text a model reads and writes in — roughly a word-piece. Common words are one token; longer or rarer ones get split into several.
Parameters.The billions of learned weights inside the network. They are what training adjusts, and they encode everything the model “knows.”
Context window. How much text the model can consider at once — the prompt plus its own reply so far — measured in tokens. Anything beyond it falls out of view.
Training vs. inference. Training is the one-time, expensive process of learning the weights from data. Inference is using the finished model to generate text — what happens every time you send a prompt.
Hallucination. Confident but wrong output. Because the model predicts plausible-sounding text rather than verified truth, it can state something false with the same fluency as a fact.

Training vs. inference

These two phases are easy to blur, but they’re very different jobs — one builds the model, the other uses it.

Training

Inference

Learns the parameters from data

Uses the fixed parameters to answer

Happens once, up front, at great cost

Happens every time you send a prompt

Adjusts weights by predicting next tokens

Predicts and samples next tokens, one by one

Produces the model

Produces the response you read

Why this matters for AI visibility

LLMs are the engines behind the chatbots and AI search tools buyers now ask for recommendations. When one names a brand, that name is a prediction drawn from patterns in its training — which is exactly why an LLM can also state something confidently wrong about you.

That tendency to hallucinate is the reason retrieval-augmented generationmatters: by grounding an answer in real sources fetched at answer time, rather than the model’s memory alone, it makes the reply both more accurate and more likely to cite you. Understanding how the model predicts is the first step to understanding why being a clear, retrievable source is the work of AEO and GEO.

Frequently asked

How does an LLM actually work?: It predicts one token at a time. Given the text so far, the model produces a probability distribution over every possible next token, samples one, appends it, and repeats — feeding its own output back in until the response is complete. There is no lookup of stored answers; the reply is generated token by token from patterns learned during training.
Why do LLMs hallucinate?: Because a model predicts plausible-sounding text, not verified truth. It has no built-in sense of whether a statement is correct — only of how likely the words are to follow, given its training. When the most probable continuation happens to be wrong, the model states it just as confidently as a fact. Grounding answers in retrieved sources (retrieval-augmented generation) is the common way to reduce this.
What is a token?: A token is the chunk of text a model reads and writes in — roughly a word-piece. Common words are often a single token, while longer or rarer words are split into several. Models don't operate on letters or whole words directly; they turn text into tokens, predict tokens, and turn tokens back into text. Both the input a model can hold (its context window) and the cost of using it are measured in tokens.