Tokens Explained: How LLMs Read and Generate Text
Large Language Models (LLMs) such as GPT, Claude, Gemini, and other modern AI systems work with text in a way that is different from how humans read sentences.
When we see:
Artificial intelligence is changing the world.
we understand it as a sequence of words.
An LLM does not directly process the sentence as ordinary words. Instead, the text is converted into smaller pieces called tokens.
Tokens are one of the fundamental building blocks of modern language models. Understanding tokens helps explain how LLMs process text, calculate context length, estimate API usage, and generate responses.
In this article, we will explore:
What tokens are
How tokenization works
Words vs tokens
Characters vs tokens
How text is converted into numbers
Token IDs
Input and output tokens
Context windows
Token probabilities
How LLMs generate text
Token counting with Python
Why tokenization matters for AI applications
What Is a Token?
A token is a piece of text that an LLM processes as an individual unit.
A token can represent:
A complete word
Part of a word
A punctuation mark
A space combined with text
Numbers
Special characters
For example, the sentence:
I love programming.
might be divided into pieces conceptually similar to:
I
love
programming
.
The exact tokenization depends on the tokenizer used by the particular model.
Therefore, a token is not always equal to one word.
Words Are Not the Same as Tokens
One of the most common misconceptions about LLMs is:
1 word = 1 token
This is not generally true.
Consider:
Artificial intelligence
Depending on the tokenizer, this could be represented by several tokens.
A longer or less common word may be split into multiple pieces.
For example:
unbelievable
could conceptually be represented as:
un
believ
able
The exact result depends on the tokenizer.
This approach allows language models to handle words they may not have seen as complete units during training.
Why Do LLMs Use Tokens?
If an LLM processed every possible word as a unique item, its vocabulary would become extremely large.
Instead, tokenizers divide language into reusable pieces.
For example:
play
player
playing
played
may share token components.
This makes it possible to represent many different words using a relatively manageable vocabulary.
Tokenization also helps models process:
New words
Technical terms
Names
URLs
Programming code
Numbers
Different languages
From Text to Tokens
The first major step in processing text is tokenization.
Consider:
Hello, how are you?
A tokenizer converts the text into a sequence of tokens.
Conceptually:
Text
↓
Tokenizer
↓
Token Pieces
↓
Token IDs
For example:
"Hello, how are you?"
could become something conceptually similar to:
["Hello", ",", " how", " are", " you", "?"]
The actual tokens and token IDs depend on the tokenizer.
What Is a Token ID?
LLMs do not directly process token strings such as:
Hello
world
computer
Instead, each token is mapped to a numerical identifier.
For example, a tokenizer might produce:
Hello → 15496
world → 995
These numbers are called token IDs.
The actual IDs differ between tokenizers.
The overall process is:
"Hello world"
↓
Tokenizer
↓
["Hello", " world"]
↓
Token IDs
↓
[15496, 995]
The model processes these numerical representations.
Tokenization Example Using Python
One popular tokenizer library for working with OpenAI-compatible tokenization is tiktoken.
Install it with:
pip install tiktoken
Then:
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = "Hello, how are you?"
tokens = encoding.encode(text)
print(tokens)
The output will be a list of integers representing token IDs.
You can also decode them:
decoded = encoding.decode(tokens)
print(decoded)
Output:
Hello, how are you?
The encoding and tokenization used by a specific production model can differ, so use the tokenizer associated with that model when exact token accounting matters.
Display Individual Tokens
We can inspect the token pieces directly.
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = "Hello, how are you?"
tokens = encoding.encode(text)
for token in tokens:
piece = encoding.decode([token])
print(token, repr(piece))
Example output might look conceptually like:
9906 'Hello'
11 ','
1268 ' how'
527 ' are'
499 ' you'
30 '?'
The important point is that the token ID is a number, while the decoded token is the text fragment represented by that number.
Token Count
We can calculate how many tokens are contained in a piece of text.
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = """
Large language models process text
using tokens instead of raw words.
"""
tokens = encoding.encode(text)
print("Token count:", len(tokens))
This is useful when building AI applications because models have limits on how much input they can process.
Tokens and Characters
Tokens are also different from characters.
Consider:
Hello
There are:
5 characters
But the tokenizer may represent it using one or more tokens.
Likewise:
internationalization
contains many characters and may be represented using multiple tokens.
Therefore:
Characters ≠ Words ≠ Tokens
These are different measurements.
Why Some Words Use More Tokens
Tokenizers are generally designed to represent frequently occurring text efficiently.
Common words may often be represented using fewer tokens.
Rare or unusual words may require more tokens.
For example:
computer
may be represented efficiently.
But a long technical term, unusual name, or random character sequence could require multiple tokens.
This is one reason that estimating token usage based only on word count is unreliable.
Tokens and Programming Code
Tokens are not limited to natural language.
LLMs can also process programming languages.
Consider:
def calculate_total(price, tax):
return price + (price * tax)
The tokenizer divides the code into pieces.
These may represent:
def
calculate
_total
(
price
,
tax
)
:
return
...
The exact tokenization depends on the model's tokenizer.
This is important because large source-code files can consume many tokens.
Tokens and Numbers
Numbers can also be split into tokens.
For example:
123456789
may not necessarily be represented as one token.
The tokenizer may divide it into multiple pieces.
This is one reason LLMs can sometimes struggle with certain numerical tasks: the model is processing token representations rather than treating every number as a native mathematical integer.
For precise arithmetic, external calculators or programmatic tools are often more reliable.
Special Tokens
Some tokenizers use special tokens for specific purposes.
Examples include tokens representing concepts such as:
Beginning of sequence
End of sequence
Padding
Unknown token
Special instructions
The exact special tokens depend on the model architecture and tokenizer.
In chat-based systems, the model may also receive structured information representing roles such as:
system
user
assistant
The underlying implementation determines exactly how these are encoded.
Input Tokens
When you send a prompt to an LLM, the text you provide becomes part of the model's input.
For example:
Explain machine learning in simple terms.
This is converted into tokens.
Conceptually:
User Prompt
↓
Tokenizer
↓
Input Tokens
↓
LLM
The model processes these tokens to determine what should come next.
Output Tokens
When an LLM generates a response, it produces tokens sequentially.
For example:
Machine learning is a branch of AI...
The response is generated as a sequence of tokens.
Conceptually:
Input Tokens
↓
LLM
↓
Output Token 1
↓
Output Token 2
↓
Output Token 3
↓
...
The generated tokens are eventually decoded back into readable text.
How LLMs Generate Text
One of the most important ideas behind LLMs is next-token prediction.
Suppose the model receives:
The sky is
The model predicts possible next tokens.
For example:
blue
cloudy
clear
dark
...
Each possible next token can have an associated probability.
Conceptually:
blue → 0.65
cloudy → 0.15
clear → 0.10
dark → 0.05
other → 0.05
The model selects a token according to the generation strategy.
Then that token is added to the sequence.
For example:
The sky is
becomes:
The sky is blue
The model then predicts the next token again.
Autoregressive Generation
This process is called autoregressive generation.
Consider:
The cat is
The model predicts:
sleeping
Now the sequence becomes:
The cat is sleeping
The model predicts the next token:
on
Now:
The cat is sleeping on
Then:
the
Then:
sofa
Eventually:
The cat is sleeping on the sofa.
The process can be visualized as:
Prompt
↓
Predict next token
↓
Add token
↓
Predict next token
↓
Add token
↓
Repeat
↓
Final response
Token Probabilities
At each generation step, the model produces scores for possible next tokens.
For example:
Prompt:
"The capital of France is"
Possible predictions:
Paris → high probability
London → low probability
Berlin → low probability
Madrid → low probability
The model selects a likely continuation.
This process happens repeatedly until the model reaches a stopping condition or generation limit.
What Is Temperature?
Temperature is a generation parameter that can influence how predictable or diverse the output is.
A lower temperature generally makes generation more focused and deterministic.
A higher temperature generally allows more variation.
Conceptually:
Low Temperature
↓
More predictable
↓
Less variation
and:
High Temperature
↓
More variation
↓
More diverse output
The exact behavior depends on the model and API.
Temperature does not change the underlying tokenizer.
Token Limit and Context Window
LLMs have a maximum amount of information they can process within a single context.
This is called the context window.
It is measured in tokens.
For example:
Context Window
↓
┌─────────────────────────┐
│ System Instructions │
│ Conversation │
│ User Prompt │
│ Documents │
│ Previous Messages │
│ Model Response │
└─────────────────────────┘
All of these can consume context.
A model with a larger context window can process longer conversations or documents in a single request.
The exact maximum depends on the model.
Why Context Windows Matter
Imagine uploading a very large document to an AI application.
The document may contain:
100,000+ tokens
If the model's available context is smaller than the required input plus output, the entire document cannot be processed in one request.
Developers can solve this using techniques such as:
Chunking
Summarization
Retrieval
Embeddings
RAG
Token Chunking
Large documents can be divided into smaller chunks.
For example:
Large Document
↓
┌──────────────┐
│ Chunk 1 │
├──────────────┤
│ Chunk 2 │
├──────────────┤
│ Chunk 3 │
├──────────────┤
│ Chunk 4 │
└──────────────┘
Each chunk can be processed separately.
Python example:
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = """
This is a long document that needs
to be divided into smaller pieces.
"""
tokens = encoding.encode(text)
chunk_size = 100
chunks = [
tokens[i:i + chunk_size]
for i in range(0, len(tokens), chunk_size)
]
print("Number of chunks:", len(chunks))
This is a simple token-based chunking example.
Production systems often use more sophisticated chunking strategies that preserve paragraphs, sentences, or semantic boundaries.
Tokens and RAG
Retrieval-Augmented Generation, commonly called RAG, uses external information together with an LLM.
A simplified RAG pipeline looks like:
User Question
↓
Search / Retrieval
↓
Relevant Documents
↓
Tokenization
↓
LLM
↓
Answer
For example:
Question:
"What is our refund policy?"
The system retrieves the relevant company policy.
Only the relevant sections are added to the model context.
This helps reduce unnecessary token usage and allows applications to work with large document collections.
Tokens and API Costs
Many commercial LLM APIs calculate usage based on tokens.
A request may contain:
Input Tokens
+
Output Tokens
For example:
Input: 2,000 tokens
Output: 1,000 tokens
--------------------
Total: 3,000 tokens
The actual pricing varies by model and provider.
Therefore, token optimization can help reduce API costs.
How to Reduce Token Usage
Developers can reduce unnecessary token consumption by:
Removing Repeated Instructions
Avoid sending the same large text unnecessarily.
Summarizing Conversations
Instead of sending an entire conversation history, store a compact summary when appropriate.
Chunking Documents
Process only the relevant portions of large documents.
Using Retrieval
Retrieve relevant information rather than sending an entire knowledge base.
Avoiding Excessive Output
Set appropriate generation limits when long responses are unnecessary.
Tokenization of Different Languages
Token usage can vary significantly between languages.
The same meaning can require different numbers of tokens depending on the tokenizer and language.
For example:
English
Spanish
Tamil
Chinese
Japanese
Arabic
may have different tokenization characteristics.
This matters when building multilingual applications because token usage and context consumption can vary between languages.
Always benchmark with the actual languages your application will support.
Tokenization and Emojis
Emojis and special Unicode characters can also be represented by one or multiple tokens.
For example:
😀
🚀
❤️
may have different token representations.
This demonstrates that a visible character does not necessarily correspond to one token.
Tokenization of URLs
URLs can also consume multiple tokens.
For example:
https://example.com/products/artificial-intelligence
may be divided into multiple pieces.
This is important when processing:
Web pages
Search results
Documentation
Source code
API responses
Tokenization of Large Documents
Suppose you have a document containing:
50,000 words
It would be incorrect to assume:
50,000 words = 50,000 tokens
The actual token count depends on:
Language
Vocabulary
Tokenizer
Word frequency
Formatting
Numbers
Punctuation
Code
URLs
The correct way to determine token usage is to run the text through the appropriate tokenizer.
Build a Token Counter in Python
Here is a reusable token counting script:
import tiktoken
def count_tokens(text):
encoding = tiktoken.get_encoding(
"cl100k_base"
)
tokens = encoding.encode(text)
return len(tokens)
text = """
Large language models use tokens
to process and generate text.
"""
count = count_tokens(text)
print("Token count:", count)
This can be useful when developing applications that send large amounts of text to an LLM.
For exact production accounting, use the tokenizer and usage information associated with the specific model/provider.
Build a Simple Token Analyzer
We can also display token IDs and their corresponding pieces.
import tiktoken
encoding = tiktoken.get_encoding(
"cl100k_base"
)
text = "Artificial intelligence is powerful."
tokens = encoding.encode(text)
print("Total tokens:", len(tokens))
print("\nTokens:")
for token_id in tokens:
token_text = encoding.decode(
[token_id]
)
print(
f"ID: {token_id:<8} "
f"Text: {token_text!r}"
)
This is a useful way to understand what the tokenizer is actually doing.
Tokens in a Chatbot
A chatbot may process a conversation like this:
System Instructions
↓
Previous Messages
↓
User Message
↓
Tokenization
↓
LLM
↓
Generated Tokens
↓
Assistant Response
For example:
User:
What is Python?
Assistant:
Python is a popular programming language...
Both the user's message and the assistant's generated response consist of tokens internally.
If the conversation becomes very long, the total context can grow significantly.
Token-Based AI Architecture
A typical LLM application can be represented as:
USER
↓
TEXT INPUT
↓
TOKENIZER
↓
TOKEN IDs
↓
┌──────────────┐
│ LLM │
└──────────────┘
↓
NEXT TOKEN
↓
NEXT TOKEN
↓
NEXT TOKEN
↓
TOKENIZER
↓
DECODE TOKENS
↓
TEXT RESPONSE
↓
USER
The tokenizer sits at the boundary between human-readable text and the numerical representation processed by the model.
Tokens vs Embeddings
Tokens and embeddings are related but different.
A token is a discrete text unit:
"computer"
A token ID might be:
12345
An embedding is a vector representation containing many numerical values.
Conceptually:
Token
↓
Token ID
↓
Embedding Vector
↓
Neural Network
An embedding might look like:
[
0.12,
-0.45,
0.78,
0.21,
...
]
The actual vectors are much larger and are learned representations used by the model.
Why Tokens Are Important for Developers
Understanding tokens is useful when building:
AI chatbots
RAG systems
Document processing systems
AI coding assistants
Summarization tools
Translation applications
AI search systems
Content generation platforms
Customer support bots
Token awareness helps developers design systems that stay within context limits and use resources efficiently.
Common Token Misconceptions
Misconception 1: One Word Equals One Token
False.
A word may be one token or several tokens.
Misconception 2: One Character Equals One Token
False.
A token can represent multiple characters or a fragment of text.
Misconception 3: Token IDs Have Meaningful Numerical Order
Not necessarily.
A token ID is primarily an index used by the tokenizer's vocabulary. Token ID 100 is not inherently more meaningful than token ID 200.
Misconception 4: More Tokens Always Mean Better Answers
False.
More tokens simply mean more text is being processed or generated.
Good prompt design is often more important than simply increasing token count.
Misconception 5: Tokens Only Contain Words
False.
Tokens can represent:
Words
Word fragments
Punctuation
Spaces
Numbers
Symbols
Code
Unicode characters
Practical Example
Consider the prompt:
Explain how machine learning works in simple terms.
The process is approximately:
TEXT
↓
"Explain how machine..."
↓
TOKENIZER
↓
Token IDs
↓
┌──────────────────┐
│ LLM │
└──────────────────┘
↓
Probability scores
↓
Next token
↓
Next token
↓
Next token
↓
...
↓
Generated text
The LLM does not simply search for a stored paragraph and return it. It generates a sequence of tokens based on the input context and learned model parameters.
Real-World Token Optimization Strategy
Suppose an AI application processes a large company knowledge base.
A naive approach might send:
Entire Knowledge Base
↓
LLM
This can consume a large amount of context.
A better architecture is:
Knowledge Base
↓
Document Chunking
↓
Embeddings / Index
↓
User Question
↓
Retrieval
↓
Relevant Chunks
↓
LLM
↓
Answer
This approach reduces irrelevant context and can make the system more efficient.
Final Summary
Tokens are the fundamental text units processed by modern language models.
The overall process can be summarized as:
Human Text
↓
Tokenization
↓
Token IDs
↓
Neural Network
↓
Next-Token Prediction
↓
Generated Token IDs
↓
Decoding
↓
Human-Readable Text
The most important concepts to remember are:
Words ≠ Tokens
Characters ≠ Tokens
Token IDs = Numerical identifiers
Input Tokens = Text sent to the model
Output Tokens = Text generated by the model
Context Window = Maximum context measured in tokens
Tokenization = Converting text into model-readable pieces
Understanding tokens is essential for anyone building applications with Large Language Models. It helps developers reason about context limits, prompt size, document processing, RAG architectures, generation behavior, and API usage.
Once you understand how text becomes tokens and how tokens are generated one after another, the internal workflow of an LLM becomes much easier to understand.