A full alignment pipeline is less mysterious than it sounds: prepare data, run supervised fine-tuning, evaluate, then apply preference optimization — measuring at each stage so you know which step earned which gain.
The stages
Why DPO after SFT rather than instead of it
SFT can teach a model the shape of a good answer. It cannot teach which of two well-formed answers a human would actually send — there is no signal for that in a single-target dataset. Preference pairs carry exactly that signal, and the best source is corrections real users or agents made in production.
What to expect
On a narrow task with a few thousand clean examples, this sequence can move task accuracy from around 61% to the low 80s, and tool-call validity from under 50% to the mid 90s. Output length often drops substantially too, which shows up directly in serving cost.
