Token (AI)
A token is a unit of representation that a model uses to process information, such as a word, part of a word or a symbol
What it means, how it is used and where it comes up, with its related concepts.
What it is and how it works
How text is split depends on the tokenisation system and the type of content. Context limits and part of the cost of some services are expressed in tokens. A token does not correspond to a fixed word or number of characters.
Also known as
Language token
Topics