Token
Level 1
A small chunk of text that a model reads and writes, usually part of a word.
Term 2 of 8 in Language models
In plain language
Models do not see letters or words. Text is chopped into tokens, roughly three quarters of a word each. The model reads tokens and produces tokens.
Think of it like this
Like reading with Lego bricks instead of letters. "Unhelpful" might arrive as three separate bricks.
Why it matters
Tokens are the unit of both pricing and limits. Almost every AI cost and cap is measured in tokens.
Next terms in Language models
- Large language modelA model trained on enormous amounts of text to predict what words come next.
- PromptThe instruction and information you give a model.
- Context windowThe maximum amount of text a model can consider at one time.
- TemperatureA setting that controls how predictable or varied a model's output is.