Every capability an LLM shows — answering questions, writing code, summarizing documents — emerges from one deceptively simple training objective: predict the next token in a sequence. Understanding this foundation demystifies both the impressive behavior and the strange failures you will encounter throughout this course.
Tokens Not Words
Models do not read words; they read tokens, subword chunks produced by a tokenizer. Common words map to single tokens while rare words split into pieces, which explains classic quirks like weak letter-counting and arithmetic on unusual numbers. Token counts also drive API pricing and context limits, so thinking in tokens is a practical skill, not trivia.
Prediction as Compression of Knowledge
To predict the next token well across trillions of examples, a model is forced to internalize grammar, facts, reasoning patterns, and stylistic conventions. Knowledge is stored implicitly in billions of learned weights rather than in a retrievable database, which is why models can articulate a fact fluently one moment and confabulate a similar-sounding falsehood the next.
Sampling and Temperature
At inference the model outputs a probability distribution over its vocabulary, and a sampling strategy picks the token. Temperature scales the distribution: low values make output focused and repeatable, high values make it diverse and creative. Parameters like top-p further trim unlikely tokens. Generation is this loop repeated, one token at a time.
Action Step
Paste a paragraph into a public tokenizer playground and study how it splits into tokens, noting where words fragment. Then call an LLM API twice with temperature 0 and temperature 1 on the same prompt and compare the outputs.
Educational content only. The AI/ML tooling landscape changes quickly — verify current library versions, APIs, and best practices in official documentation. This course does not certify anyone.