Parameters
The learned values (weights and biases) in a neural network that determine its behavior. LLMs have billions of parameters.
In-Depth Explanation
Parameters in machine learning are the learnable values that define a model's behavior. For neural networks, parameters primarily consist of weights (connection strengths) and biases (offset values) that are adjusted during training.
Understanding parameter counts:
- 7B model: 7 billion parameters
- 70B model: 70 billion parameters
- GPT-4: Estimated 1.7+ trillion parameters
- More parameters: Generally more capability, more cost
What parameters represent:
- Encoded knowledge from training data
- Patterns and relationships learned
- The "intelligence" of the model
- Everything the model can do
Parameters vs hyperparameters:
- Parameters: Learned during training (weights, biases)
- Hyperparameters: Set before training (learning rate, batch size)
Resource implications:
- Memory: ~2-4 bytes per parameter (varies by precision)
- Compute: More parameters = slower inference
- Cost: Training and running scales with size
Business Context
Parameter count roughly indicates model capability. GPT-4 has ~1.7T parameters; smaller models with 7-70B can still be very capable for many tasks.
How Clever Ops Uses This
We help Australian businesses choose models with appropriate parameter counts, balancing capability against deployment cost and speed.
Example Use Case
"A 70B parameter model offers a good balance of capability and deployment cost for many enterprise applications."