Prebuttal

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

NeurIPS · simulated review · 2026-10-27T07:30:28+00:00 · seed 42
backends: R1: agent-claude · R2: agent-codex · R3: api-gemini
Unofficial demo: an AI-simulated review of arXiv:2305.18290 by Rafael Rafailov, Archit Sharma, Eric Mitchell, et al., licensed CC BY 4.0. Summaries and short quotes below were generated by AI models from the paper and may contain errors; generated only to show what Prebuttal outputs. Not written, reviewed or endorsed by the authors, and not an assessment of the work's merit. Authors who want this page removed: open an issue.
4.33/6
estimated · σ 0.47 · range 4–5 · n=3
Three independent AI reviewers, isolated contexts, plus an area chair. This is a pre-submission stress test, not a prediction.
R1 senior_kind 4/6
agent-claude · sonnet · conf 4

The paper proposes Direct Preference Optimization (DPO), which reparameterizes the reward in the KL-constrained RLHF objective as r(x,y)=β log(π(y|x)/πref(y|x)) (Eq. 5). Because the partition function cancels in the Bradley-Terry likelihood, the policy can be trained directly with a binary cross-entropy loss (Eq. 7), without a separate reward model or RL sampling loop. Section 5 argues the reparameterization loses no generality (Theorem 1, Lemmas 1-2) and attributes PPO instability to the missing normalizer.…

Top weaknesses
  • Evidence for superiority over PPO on real tasks is thin and rests mainly on a single GPT-4 judged setup. In TL;DR (Section 6.2), DPO ~61% vs PPO ~57% is a ~4-point gap with no confidence intervals, seeds, or…
  • The PPO baseline is weakly established outside the sentiment task. For dialogue (Section 6.2), the PPO model is an off-the-shelf checkpoint (reciprocate/ppo_hh_pythia-6B) that the authors could not make beat base…
  • The headline sentiment result is on a synthetic setup where preferences are generated by the same classifier that defines the reward (Section 6, Tasks; Appendix C.1: 6 pairs per prefix from 4 completions with…
R2 adversarial 4/6
agent-codex · gpt-6-luna · conf 4

The paper introduces Direct Preference Optimization (DPO), which reparameterizes a reward model in terms of a policy and reference policy so that the KL-constrained RLHF objective can be optimized directly with a preference classification loss. It proves that, under Bradley-Terry or Plackett-Luce preferences and stated support assumptions, this reparameterization preserves reward equivalence classes and recovers the corresponding optimal policy. Experiments on sentiment control, Reddit summarization, and…

Top weaknesses
  • The evidence for broad real-world preference alignment is narrower than the abstract's claims of matching or improving response quality in summarization and dialogue. Section 6 evaluates only Reddit TL;DR and one-turn…
  • The main summarization advantage is modest and incompletely quantified. Section 6.2 reports approximately 61% for DPO at temperature 0 versus 57% for PPO at its best temperature, but provides no confidence intervals or…
  • The human validation is limited in scope and population. Section 6.4 and Appendix D.3 use 25 Stanford-affiliated volunteers, only two human judgments per comparison for DPO and PPO-1, and one for SFT. This is weak…
R3 detail_obsessive 5/6
api-gemini · gemini-3.8-flash · conf 5

This paper presents Direct Preference Optimization (DPO), an algorithm for aligning language models to human preferences without explicit reward modeling or reinforcement learning. By analytically expressing the optimal policy for the standard KL-constrained reward maximization objective in closed form and substituting this parameterization into the Bradley-Terry preference model, the authors show that the optimal policy can be learned directly using a binary cross-entropy loss over preference pairs. The approach…

Top weaknesses
  • Lack of statistical dispersion and confidence intervals: No standard deviations, standard errors, or binomial confidence intervals are reported for the win rates in Table 1 (e.g., 0.36 vs. 0.26), Table 2 (e.g., 58% vs.…
  • Compromised PPO baseline in Anthropic-HH dialogue: In Section 6.2, the authors note that they were unable to get an existing public PPO model (reciprocate/ppo_hh_pythia-6B) to outperform the base Pythia-2.8B model, and…
  • Evaluation on out-of-distribution CNN/DailyMail (Table 1) reveals low absolute win rates against reference summaries for all models (DPO: 0.36 at temp 0, 0.31 at temp 0.25; PPO: 0.26 at temp 0, 0.23 at temp 0.25),…

Meta-review

AC estimate: 5/6

## Summary The submission proposes Direct Preference Optimization (DPO). The paper uses the closed-form optimum of the KL-constrained reward objective (Eq. 4) to write the reward as β log π/π_ref + β log Z(x) (Eq. 5). The partition function cancels in the Bradley-Terry likelihood (Eq. 6), so the policy can be trained with a binary cross-entropy loss (Eq. 7), with no explicit reward model and no RL sampling loop. Section 5.1 (Lemmas 1-2, Theorem 1, Proposition 1 in App. A.5-A.6) argues the reparameterization loses no generality over reward equivalence classes, given π_ref>0 and β>0. Experiments cover: - IMDb sentiment with a ground-truth classifier reward (reward-KL frontier, 22 runs, Fig. 2 left); - Reddit TL;DR with GPT-J (GPT-4 win rates, Fig. 2 right); - single-turn Anthropic-HH with Pythia-2.8B (Fig. 3); - a CNN/DailyMail transfer test (Table 1); - a human study validating GPT-4 as judge (Table 2). ## Assessment of the contribution All three reviewers agree the core idea is elegant and correctly derived. I agree. The derivations in App. A.1-A.2 follow directly, and the gradient analysis in Section 4 gives a concrete mechanistic reading. The paper links the implicit-reward weighting to the degeneration of unweighted unlikelihood (Table 3). The theory answers the obvious expressivity question: every equivalence class has a unique representative of the DPO form (Proposition 1). The practical payoff is a few lines of loss code (App. B) replacing a multi-model PPO pipeline. That is a substantial contribution in its own right. The controlled sentiment experiment is well designed and is the cleanest evidence for the claim that DPO optimizes the same objective better. Ground-truth reward is available, and DPO's frontier lies above PPO and even PPO-GT. R1 and R2 rightly…

Revision plan

  1. must Dialogue comparison has no matched PPO baseline (Sec. 6.2). The public PPO checkpoint fails to beat base Pythia-2.8B, and Best-of-128 is used as a 'rough proxy for PPO-level performance.' Yet the abstract claims DPO…
    → Either train an in-house PPO on Anthropic-HH with the same base model, data, prompts and decoding settings, reporting its tuning budget, or rewrite the abstract and Sec. 6.2 to state that the dialogue comparison is… R1, R2, R3
  2. must No uncertainty is reported for any win rate: TL;DR ~61% vs ~57% (Fig. 2 right), Table 1 (0.36/0.26, 0.31/0.23), Table 2 (human 58% on 272 judgments), Fig. 3. No seed variance and no number of test prompts are given.
    → Add bootstrap or Wilson confidence intervals to all win-rate figures and tables. State the number of evaluation prompts. Report variance across at least 3 seeds for DPO and PPO on TL;DR and sentiment. Discuss whether… R1, R2, R3
  3. should The efficiency claim ('computationally lightweight', 'substantially simpler', abstract and Sec. 1) is not measured.
    → Report GPU-hours, wall-clock time and peak memory for DPO vs PPO at the same model size (e.g., GPT-J on TL;DR), including reference-model forward passes. Otherwise, phrase the claim qualitatively. R1, R2
  4. should β sensitivity and tuning fairness are unclear. β is swept only in sentiment, TL;DR uses β=0.5 (App. B) while being described as 'not meaningfully tuned,' and the PPO tuning budget on TL;DR is not described.
    → Add a β sweep on TL;DR (and HH if possible). Document the number of runs and hyperparameters tried for PPO vs DPO. Explain how β=0.5 was chosen. R1, R2
  5. should Possible length bias in GPT-4 win rates. Sec. 6.4 notes GPT-4 (S) prefers longer, repetitive summaries, and Tables 7, 9 and 10 show verbose DPO outputs hitting maximum length. No length statistics are reported.
    → Report average output length per method and task. Add a length-controlled or length-binned win-rate analysis for TL;DR and HH. R1
  6. should The sentiment result (Sec. 6.1) uses classifier-generated preferences (App. C.1), so it tests optimization rather than robustness to noisy human labels. The learned reward model's accuracy for PPO is not reported.
    → Scope the Sec. 6.1 conclusion to optimization efficiency. Report the reward model's validation accuracy. Optionally, add a label-noise ablation, e.g., flipping a fraction of pairs. R1, R2
  7. could The Sec. 5.2 instability explanation (missing normalizer f(rφ, π_ref, β)) is not empirically tested. Sec. 5.2 also cites 'Eq. 7' where Eq. 4 is meant.
    → Fix the equation reference. Either add a PPO run with an estimated soft-value baseline, or label Sec. 5.2 explicitly as an interpretation rather than a demonstrated cause. R1
  8. could The human study is small: 25 Stanford-affiliated volunteers, single annotation for SFT, and the judgment counts in Table 2 (272/122/199) are not reconciled in the text. Over-optimization (Fig. 3 right) is only…
    → Explain the reduction from 300/125/200 judgments to the reported counts (ties and the non-responding volunteer, footnote 7). Acknowledge the rater-pool limitation. Use the sentiment setting's ground-truth reward to… R2, R3