Batching

Batching is the practice of grouping multiple items together for processing in a single operation, rather than handling each individually. This fundamental optimisation technique can dramatically improve AI system efficiency.

Why batching matters:

• Cost reduction: API calls often have per-request overhead
• Throughput increase: More items processed per unit time
• Resource efficiency: Better GPU/CPU utilisation
• Rate limit management: Fewer requests within limits

Types of batching:

• Request batching: Multiple inputs in one API call
• Embedding batching: Generate multiple embeddings together
• Inference batching: Process multiple prompts simultaneously
• Database batching: Bulk inserts/queries

Batching strategies:

• Fixed size: Wait for N items, then process
• Time-based: Process accumulated items every X seconds
• Hybrid: Whichever threshold is reached first
• Dynamic: Adjust batch size based on load

Trade-offs to consider:

• Larger batches = better efficiency but higher latency
• Smaller batches = lower latency but more overhead
• Memory constraints limit maximum batch size