QLoRA (Quantized LoRA)
An even more efficient fine-tuning technique that combines quantisation with LoRA, enabling fine-tuning of large models on consumer hardware.
In-Depth Explanation
QLoRA (Quantized LoRA) extends LoRA by loading the base model in 4-bit quantised form, dramatically reducing memory requirements. This enables fine-tuning of models that would otherwise require expensive enterprise GPUs.
How QLoRA works:
- Load base model with 4-bit quantisation (NF4)
- Add LoRA adapters (trained in higher precision)
- Compute gradients through frozen quantised weights
- Update only the LoRA parameters
- Optionally merge adapters with base model
Key innovations:
- 4-bit NormalFloat: Optimal quantisation for normally distributed weights
- Double quantisation: Further compress quantisation constants
- Paged optimizers: Handle memory spikes gracefully
- Full fine-tuning quality: Despite extreme compression
Memory requirements (example):
- 7B model full precision: ~28GB
- 7B model QLoRA: ~6GB
- 70B model full precision: ~280GB
- 70B model QLoRA: ~48GB
Business Context
QLoRA makes custom model training possible on a single GPU, dramatically lowering the barrier to custom AI development.
How Clever Ops Uses This
QLoRA enables us to fine-tune large models for Australian businesses without requiring expensive cloud GPU clusters, making custom AI accessible.
Example Use Case
"Fine-tuning a 70B parameter model on a single consumer GPU using QLoRA, creating a highly capable custom model affordably."