Context Window
The maximum amount of text (measured in tokens) that an LLM can process in a single request. This includes both the input prompt and the generated output.
In-Depth Explanation
The context window is one of the most important specifications of a language model, defining the maximum amount of information it can consider when generating a response. Think of it as the model's working memory.
Context window sizes have grown dramatically:
- GPT-3 (2020): 4K tokens (~3,000 words)
- GPT-4 (2023): 8K-128K tokens
- Claude 3 (2024): Up to 200K tokens (~150,000 words)
- Gemini 1.5 Pro: Up to 1M tokens
Within the context window, everything competes for space:
- System instructions and prompts
- Retrieved documents in RAG systems
- Conversation history in chat applications
- The generated output itself
Managing context effectively is crucial because:
- Costs scale with context usage (input tokens are charged)
- Longer contexts can dilute relevance and attention
- Exceeding the context window causes errors
- Very long contexts may have degraded recall in the middle
Business Context
Larger context windows (128K-1M tokens) enable processing entire documents but cost more. Choose models based on your typical document sizes and budget.
How Clever Ops Uses This
We help clients choose appropriate context window sizes for their use cases. Often, effective chunking and retrieval strategies outperform simply using larger contexts.
Example Use Case
"GPT-4 Turbo has a 128K token context window, roughly equivalent to 300 pages of text, enabling analysis of entire contracts or reports."