RLHF vs. DAGGER: The robustness debate shaping next-gen LLM fine-tuning

The hot debate in LLM fine-tuning is RLHF with reward modeling and DPO versus multi-step SFT with DAGGER-style rollouts. The claim: RLHF gives a value signal that helps a model learn why answers are better, not just what correct looks like. ^[1]

Why RLHF delivers robustness RLHF creates a reward model that encodes human preferences, helping the model generalize to unseen prompts and balance exploration with exploitation. ^[1]

DAGGER in LLM training today DAGGER is classic imitation learning: rollouts annotated by an oracle to improve trajectories. In LLMs, it's praised for structured tasks but may miss the internal 'goodness' signal and can be brittle when contexts drift; it's also noted as underexplored. ^[1]

DPO and the offline path Some discussions point to DPO—training that says one response is better than another—as a way to beat plain SFT without looping in reinforcement learning. ^[1] There's a note that DPO can outperform SFT on positive labels, and that this offline path could sidestep the RL loop. ^[1]

Overall, the consensus leans toward RLHF for robustness in multi-step training, while DAGGER remains a niche for structured tasks. ^[1]

References