Chunking

Chunking is the process of dividing large documents into smaller segments for processing by AI systems. It's a critical step in RAG pipelines where chunk quality directly impacts retrieval and answer quality.

Why chunking matters:

• Models have context window limits
• Smaller chunks enable precise retrieval
• Embedding quality degrades for very long texts
• Allows returning most relevant portions

Chunking strategies:

• Fixed size: Split every N characters/tokens
• Sentence-based: Split at sentence boundaries
• Paragraph-based: Maintain paragraph structure
• Semantic: Use AI to find natural breaks
• Recursive: Hierarchical splitting with overlap

Key parameters:

• Chunk size: How large each piece is (typically 200-1000 tokens)
• Chunk overlap: How much consecutive chunks share (typically 10-20%)
• Separators: What constitutes a break point

Common mistakes:

• Chunks too small: lose context
• Chunks too large: dilute relevance
• No overlap: miss information at boundaries
• Ignoring structure: break mid-sentence/thought