Module 4 of 6 · Expert · 9 min read
Fine-tuning and distillation
When prompting is not enough: supervised fine-tuning, preference tuning, RL and distillation.
- 1Exhaust prompting and retrieval before fine-tuning.
- 2Fine-tuning teaches format, style and narrow skills; RAG supplies knowledge.
- 3Distillation trains a small model to imitate a large one on your task.
01Decide in this order
Each step costs more than the last, in money and maintenance.
- Better prompts and examples.
- Retrieval for knowledge the model lacks.
- Fine-tuning for consistent format, tone or specialised behaviour.
- Training from scratch: almost never.
02Kinds of tuning
Supervised fine-tuning (SFT) trains on input-output pairs. Preference tuning (such as DPO) trains on pairs of better and worse answers. Reinforcement fine-tuning optimises against a grader, useful when correct answers can be checked automatically.
Parameter-efficient methods such as LoRA adapt a small number of extra weights, making tuning cheap and swappable.
03Distillation
Generate high-quality outputs from a frontier model on your task, then fine-tune a small, fast model on them. You can often keep most of the quality at a fraction of the cost and latency.
Finished reading?
Mark it complete to track your progress, then test yourself with the LLM Internals quiz.