Inference
Using a trained model to make predictions or generate outputs on new data. This is the "runtime" phase of AI, as opposed to training.
In-Depth Explanation
Inference is the process of using a trained AI model to produce outputs from new inputs. While training teaches the model, inference puts that learning to work on real-world data.
The inference process:
- Receive input data (text prompt, image, etc.)
- Preprocess and tokenise the input
- Pass through the model's neural network
- Generate output predictions
- Post-process into the final result
Key inference considerations:
- Latency: Time from input to output
- Throughput: Requests processed per second
- Cost: Compute resources and API fees
- Accuracy: Quality of outputs
- Reliability: Consistency and uptime
Inference optimisation techniques:
- Model quantisation (reduce precision)
- Batching multiple requests
- Caching common queries
- Using smaller models where appropriate
- Streaming outputs progressively
- Speculative decoding for speed
Business Context
Inference costs are your ongoing AI expenses. Optimising inference speed and efficiency directly impacts operational costs and user experience.
How Clever Ops Uses This
We help Australian businesses optimise inference costs and performance. Smart model selection and caching strategies can reduce AI costs by 50-80% while maintaining quality.
Example Use Case
"When a customer asks your chatbot a question, the model performs inference to generate a response in real-time."