How do I count tokens?

Use tokeniser tools: OpenAI's tiktoken library (Python), online tokeniser tools, or approximate with the 4 characters per token rule. Different models use different tokenisers, so counts may vary.

Why are tokens used instead of words?

Tokens balance vocabulary size with coverage. A word-based vocabulary would be huge and miss new words. A character-based approach would be inefficient. Subword tokens handle any text efficiently.

Do input and output tokens cost the same?

Usually no - output tokens typically cost more (2-4x) because generation is more compute-intensive than encoding. Check specific model pricing. Both contribute to total cost.

How can I reduce token usage?

Strategies: concise prompts, summarise long context, use system prompts efficiently, limit max_tokens appropriately, cache common queries, and choose smaller models for simple tasks.

Book Free Assessment Calculator

Tokens

The basic units of text that LLMs process. Roughly 1 token = 4 characters or 0.75 words in English. Both input and output are measured in tokens.

In-Depth Explanation

Tokens are the fundamental units that LLMs process. Rather than working with characters or words directly, models break text into tokens - subword units that balance vocabulary size with expressiveness.

How tokenisation works:

Text is split into tokens using a learned vocabulary
Common words often become single tokens
Rare words split into multiple tokens
Punctuation and spaces are also tokens
Different models use different tokenisers

Token rules of thumb (English):

1 token ≈ 4 characters
1 token ≈ 0.75 words
100 tokens ≈ 75 words
1000 words ≈ 1300 tokens

Why tokens matter:

Pricing: API costs are per-token
Limits: Context windows measured in tokens
Computation: Processing time scales with tokens
Output control: max_tokens limits generation length

Tokenisation quirks:

"Hello" = 1 token, "Hello!" = 2 tokens
Numbers can be surprising (384 might be 2 tokens)
Non-English text often uses more tokens
Code tokenises differently than prose

Business Context

Token usage directly determines API costs. A typical 1000-word document is about 1300 tokens. Monitor token usage to control expenses.

How Clever Ops Uses This

We help Australian businesses understand and optimise token usage. Efficient prompting and smart caching can reduce AI costs by 50% or more while maintaining quality.

Example Use Case

"The word "hamburger" is 3 tokens: "ham", "bur", "ger". The word "the" is 1 token. Understanding this helps predict costs."

Frequently Asked Questions

Learn More

Large Language Models Explained: Complete Business Guide

Understand how LLMs work, compare GPT-4, Claude, Gemini, and Llama, and learn to choose the right model for your business needs. Complete guide to capabilities, limitations, and practical applications.

Read article

Tokenization Total Cost of Ownership