LoRA (Low-Rank Adaptation)
An efficient fine-tuning technique that trains only a small number of additional parameters, dramatically reducing compute and storage requirements.
In-Depth Explanation
LoRA (Low-Rank Adaptation) is a fine-tuning technique that dramatically reduces the resources needed to customise large language models. Instead of updating all model weights, LoRA trains small adapter matrices that modify model behaviour.
How LoRA works:
- Freezes the original model weights
- Adds small trainable matrices (adapters) to each layer
- These adapters are low-rank decompositions
- Only adapter parameters are trained
- At inference, adapters can be merged with base weights
Benefits of LoRA:
- Efficiency: 1000x fewer trainable parameters
- Speed: Much faster training
- Cost: Can train on consumer GPUs
- Storage: Adapters are tiny (MBs vs GBs)
- Flexibility: Easy to swap adapters for different tasks
Typical LoRA parameters:
- Rank (r): Usually 4-64, controls adapter capacity
- Alpha: Scaling factor, often set to r or 2×r
- Target modules: Which layers to adapt (attention, all)
Business Context
LoRA makes fine-tuning 10x cheaper and faster, making custom model training accessible for most businesses.
How Clever Ops Uses This
We use LoRA extensively for Australian business fine-tuning projects. It enables custom models without the massive compute costs of full fine-tuning.
Example Use Case
"Fine-tuning Llama with LoRA on a single GPU instead of expensive multi-GPU clusters, creating a custom model for your specific use case."