Token
Token
A unit used by AI to process text, or data that proves access permissions to a service
In Simple Terms
A token is a "meaningful unit of data" or "marker" widely used in the world of IT. In AI, it refers to a fragment of text broken down to make processing easier — and there's a limit to how many can be processed at once. In security, it refers to a device or app that generates one-time passwords, or data used to prove access permission to a service. Depending on the context, a token can represent a "text boundary" or serve as "proof of identity or permission" — its role shifts with the situation.
Behind the Name
Token. Originally, the word means "sign," "symbol," or "substitute currency." Casino chips and coin-style subway tickets are also called tokens. In IT, the word is used to mean "a small meaningful unit" or "proof of permission."
Take a Closer Look!
A token is a "meaningful unit of data" or "marker" widely used in the world of IT. What it refers to changes depending on the context.
In AI — especially generative AI — it refers to "data fragments" that text is broken down into.
AI doesn't read text as-is; instead, it splits text into pieces using its own rules to make processing easier.
The way text is split differs from one AI to another, so the token count for the same text can vary.
The reason you pay attention to "token count" when using AI is that there's a limit to how many of these fragments can be processed at once.
In security, tokens have two main uses.
The first is a small device or app that generates one-time passwords — this is called a token. It's commonly used in online banking and similar services.
The second is data used to prove access permission to a service (an access token). For example, when an app wants to read your Google Calendar, it uses an access token issued by Google to show the system "this operation has been authorized."
Both are "tools for proof" related to security, but a one-time password token is mainly used to verify your identity, while an access token is used to confirm "this operation is permitted."