Model Extrapolation Expedites Alignment (original) (raw)

Chujie Zheng1,2 Ziqi Wang3 Heng Ji3 Minlie Huang1 Nanyun Peng222footnotemark: 2
1The CoAI Group, DCST, BNRist, Tsinghua University
2University of California, Los Angeles 3University of Illinois Urbana-Champaign
chujiezhengchn@gmail.com aihuang@tsinghua.edu.cn violetpeng@cs.ucla.edu
Work done during Chujie’s visit to UCLA. Project repository: <github.com/chujiezheng/LLM-Extrapolation>. Corresponding authors.

Abstract

Given the high computational cost of preference alignment training of large language models (LLMs), exploring efficient methods to reduce the training overhead remains an important and compelling research problem. Motivated by the observation that alignment training typically involves only small parameter changes without injecting new knowledge into models, we propose a straightforward method called ExPO (model extrapolation) to expedite LLMs’ alignment with human preferences. Given a partially-trained model and its initial SFT checkpoint, ExPO improves the implicit optimization objective of alignment training by simply amplifying the parameter change based on a first-order approximation, without any additional training overhead. Through controlled experiments, we demonstrate that ExPO boosts a DPO model trained with only 20% steps to outperform the fully-trained one. Moreover, we show that ExPO notably improves existing open-source LLMs (ranging from 1.8B to 70B parameters) on the leading AlpacaEval 2.0 and MT-Bench benchmarks, which highlights ExPO’s broader utility in efficiently enhancing LLM alignment.

Model Extrapolation Expedites Alignment

Chujie Zheng1,2††thanks: Work done during Chujie’s visit to UCLA. Project repository: <github.com/chujiezheng/LLM-Extrapolation>. Ziqi Wang3 Heng Ji3 Minlie Huang1††thanks: Corresponding authors. Nanyun Peng222footnotemark: 2 1The CoAI Group, DCST, BNRist, Tsinghua University 2University of California, Los Angeles 3University of Illinois Urbana-Champaign chujiezhengchn@gmail.com aihuang@tsinghua.edu.cn violetpeng@cs.ucla.edu

1 Introduction

After conventional unsupervised pre-training on massive textual corpora and supervised fine-tuning (SFT) on high-quality demonstration data, large language models (LLMs) usually require a dedicated training stage to align with human preferences (OpenAI, 2022, 2023; Bai et al., 2022), as exemplified by the well-known Reinforcement Learning from Human Feedback (RLHF; Ouyang et al. 2022; Schulman et al. 2017) and Direct Preference Optimization (DPO; Rafailov et al. 2023). However, alignment training still requires expensive computational resources (Ji et al., 2024; Meng et al., 2024), particularly for the larger-sized LLMs (e.g., 70B parameters). This underscores the significance of exploring more efficient alignment methods to reduce the training overhead.

Our work is first motivated by the observation that preference alignment training typically does not inject new knowledge into models, thereby likely inducing only small changes of model parameters. We support this hypothesis through three arguments.First, mainstream alignment algorithms like RLHF and DPO incorporate a constraint term (e.g., the KL divergence term) to prevent excessive deviation from the initial SFT checkpoint.Second, in recent open-source LLM alignment projects (Tunstall et al., 2023; Wang et al., 2023; Ivison et al., 2023), preference alignment training usually adopts smaller learning rates (e.g., 5e-7) and fewer training steps (e.g., 400~500 steps) than SFT.Third, we take the zephyr-7b-dpo model (Tunstall et al., 2023) trained by HuggingFace as a specific instance. For any two among the pre-trained, SFT, and DPO checkpoints and for any corresponding parameter tensors 𝐏1subscript𝐏1\mathbf{P}_{1}bold_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and 𝐏2subscript𝐏2\mathbf{P}_{2}bold_P start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT, we compute the Frobenius norm ‖𝐏1−𝐏2‖normsubscript𝐏1subscript𝐏2\left\|\mathbf{P}_{1}-\mathbf{P}_{2}\right\|∥ bold_P start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - bold_P start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥ (and a normalized variant)111The Frobenius norm of tensor 𝐏𝐏\mathbf{P}bold_P is defined as: ‖𝐏‖=∑iPi2norm𝐏subscript𝑖superscriptsubscriptP𝑖2\left\|\mathbf{P}\right\|=\sqrt{\sum_{i}\mathrm{P}_{i}^{2}}∥ bold_P ∥ = square-root start_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_P start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, while the normalized variant is defined as: ‖𝐏‖=1|𝐏|⁢∑iPi2norm𝐏1𝐏subscript𝑖superscriptsubscriptP𝑖2\left\|\mathbf{P}\right\|=\sqrt{\frac{1}{|\mathbf{P}|}\sum_{i}\mathrm{P}_{i}^{% 2}}∥ bold_P ∥ = square-root start_ARG divide start_ARG 1 end_ARG start_ARG | bold_P | end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_P start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, where |𝐏|𝐏|\mathbf{P}|| bold_P | denotes the element number of 𝐏𝐏\mathbf{P}bold_P.. In Table 1, we show that the parameter change of alignment training (i.e., from SFT to DPO) is fairly small, whose absolute value of normalized Frobenius distance is merely 6.348×10−66.348superscript1066.348\times 10^{-6}6.348 × 10 start_POSTSUPERSCRIPT - 6 end_POSTSUPERSCRIPT, and is also significantly smaller than that of SFT (i.e., from Pre-trained to SFT).Therefore, in this work we hypothesize that preference alignment training usually involves only small parameter changes.

Table 1: Parameter changes of zephyr-7b-dpo.

Based on this hypothesis, we formally apply a first-order approximation to the implicit optimization objective of alignment training. We empirically justify the soundness of this approximation with open-source LLMs, where we show that an interpolated model between the DPO/RLHF model and the initial SFT checkpoint generally exhibits intermediate alignment performance compared to the original models. Building upon the first-order approximation, we propose a straightforward method called ExPO (model extrapolation) to expedite LLMs’ alignment with human preferences.ExPO amplifies the parameter change of alignment training to improve the implicit optimization objective, thus bypassing the additional training overhead to achieve better alignment performance.

We conduct controlled experiments to validate ExPO’s effectiveness. We show that ExPO notably boosts the DPO models using fewer training steps (e.g., only 20%) to outperform the fully-trained one, with the improvement of up to 8.4% length-controlled win rate on AlpacalEval 2.0 (Li et al., 2023). We then conduct ablation studies to identify several key factors influencing ExPO’s efficacy, including training data quality, training hyperparameters, and optimizer. Furthermore, we extend ExPO’s application to twelve open-source LLMs ranging from 1.8B to 70B parameters, which have undergone varied alignment training such as offline DPO, iterative DPO, or online RLHF. We show that ExPO consistently improves these LLMs by up to 4.5% on AlpacaEval 2.0 and 0.37 on MT-Bench (Zheng et al., 2023b), suggesting that ExPO can also serve as a practical and efficient means to compensate for potential training inadequacy of existing, already-aligned LLMs. In summary, our work demonstrates the efficacy of model extrapolation in enabling efficient LLM alignment, which can inspire follow-up studies and broader applications in future work.

2 Methodology

2.1 Formulation

We denote the language model’s parameter space as 𝚯𝚯\bm{\Theta}bold_Θ and suppose that the alignment performance can be quantified by a continuous scalar function ω:𝚯→ℝ:𝜔→𝚯ℝ\omega:\bm{\Theta}\to\mathbb{R}italic_ω : bold_Θ → blackboard_R, where the higher ω⁢(𝜽)𝜔𝜽\omega({\bm{\theta}})italic_ω ( bold_italic_θ ) indicates the better alignment with human preferences. In other words, ω⁢(𝜽)𝜔𝜽\omega({\bm{\theta}})italic_ω ( bold_italic_θ ) is the implicit optimization objective of alignment training. Note that ω⁢(𝜽)𝜔𝜽\omega({\bm{\theta}})italic_ω ( bold_italic_θ ) may not have an analytic form. In practice, we can employ a reward model as a proxy to compare the relative values of ω⁢(𝜽)𝜔𝜽\omega({\bm{\theta}})italic_ω ( bold_italic_θ ) by calculating the expected reward score on a development set of instructions. We suppose that the model ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (parameterized by 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT) has undergone moderate alignment training, and denote its SFT checkpoint as ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (parameterized by 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT), which is used for initializing ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and satisfies ω⁢(𝜽0)<ω⁢(𝜽1)𝜔subscript𝜽0𝜔subscript𝜽1\omega({\bm{\theta}}_{0})<\omega({\bm{\theta}}_{1})italic_ω ( bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) < italic_ω ( bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ).

2.2 First-order Approximation

Based on the aforementioned observation, we suppose that the parameter change from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, denoted as ‖𝜽1−𝜽0‖=‖Δ⁢𝜽‖normsubscript𝜽1subscript𝜽0normΔ𝜽\left\|{\bm{\theta}}_{1}-{\bm{\theta}}_{0}\|=\|\Delta{\bm{\theta}}\right\|∥ bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥ = ∥ roman_Δ bold_italic_θ ∥, is small. We can formally perform a Taylor Expansion of ω𝜔\omegaitalic_ω at 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and retain the first-order term:

where we define γ∈[0,1]𝛾01\gamma\in[0,1]italic_γ ∈ [ 0 , 1 ] to ensure that ‖γ⁢Δ⁢𝜽‖norm𝛾Δ𝜽\left\|\gamma\Delta{\bm{\theta}}\right\|∥ italic_γ roman_Δ bold_italic_θ ∥ remains small. In particular, setting γ=1𝛾1\gamma=1italic_γ = 1 gives:

Thus, the first-order approximation (Equation 1) essentially predicts that ω⁢(𝜽0+γ⁢Δ⁢𝜽)𝜔subscript𝜽0𝛾Δ𝜽\omega({\bm{\theta}}_{0}+\gamma\Delta{\bm{\theta}})italic_ω ( bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_γ roman_Δ bold_italic_θ ) will improve as γ∈[0,1]𝛾01\gamma\in[0,1]italic_γ ∈ [ 0 , 1 ] increases.

Figure 1: Interpolated models usually exhibit intermediate performance between the original DPO/RLHF models and the SFT checkpoints, while their performance improves with increasing γ𝛾\gammaitalic_γ in Equation 1.

To verify this, we conduct experiments using several open-source DPO/RLHF LLMs (Tunstall et al., 2023; Cai et al., 2024; Zhu et al., 2023). We vary γ𝛾\gammaitalic_γ within [0,1]01[0,1][ 0 , 1 ] and construct interpolated models parameterized by 𝜽0+γ⁢Δ⁢𝜽=(1−γ)⁢𝜽0+γ⁢𝜽1subscript𝜽0𝛾Δ𝜽1𝛾subscript𝜽0𝛾subscript𝜽1{\bm{\theta}}_{0}+\gamma\Delta{\bm{\theta}}=(1-\gamma){\bm{\theta}}_{0}+\gamma% {\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_γ roman_Δ bold_italic_θ = ( 1 - italic_γ ) bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_γ bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. Their alignment performance is evaluated on the UltraFeedback (Cui et al., 2023) development set using two open-source reward models: RM-Mistral-7B and FsfairX-LLaMA3-RM-v0.1 (detailed experimental setups are described in Section 3.1). Notably, when γ=0𝛾0\gamma=0italic_γ = 0 or 1111, the constructed models degenerate to the original SFT checkpoint ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and the DPO/RLHF model ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, respectively. The results in Figure 1 show that the interpolated models constructed via 𝜽0+γ⁢Δ⁢𝜽subscript𝜽0𝛾Δ𝜽{\bm{\theta}}_{0}+\gamma\Delta{\bm{\theta}}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_γ roman_Δ bold_italic_θ can generate fluent and coherent responses. Moreover, their alignment performance always lies between the original SFT model ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and the DPO/RLHF model ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, and improves with increasing γ𝛾\gammaitalic_γ, which is consistent with the predictions of the first-order approximation. We thereby empirically justify the soundness of the first-order approximation.

2.3 ExPO: Model Extrapolation

In the above first-order approximation, we constrain γ∈[0,1]𝛾01\gamma\in[0,1]italic_γ ∈ [ 0 , 1 ] to maintain the approximation’s validity along the straight-line path between 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. We now consider extending this approximation to the “extension” of the line connecting 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT beyond 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. Let γ>1𝛾1\gamma>1italic_γ > 1 and define α=γ−1>0𝛼𝛾10\alpha=\gamma-1>0italic_α = italic_γ - 1 > 0, denoting 𝜽2=𝜽0+γ⁢Δ⁢𝜽=𝜽0+(1+α)⁢Δ⁢𝜽subscript𝜽2subscript𝜽0𝛾Δ𝜽subscript𝜽01𝛼Δ𝜽{\bm{\theta}}_{2}={\bm{\theta}}_{0}+\gamma\Delta{\bm{\theta}}={\bm{\theta}}_{0% }+(1+\alpha)\Delta{\bm{\theta}}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_γ roman_Δ bold_italic_θ = bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + ( 1 + italic_α ) roman_Δ bold_italic_θ. By choosing appropriate α𝛼\alphaitalic_α such that ‖(1+α)⁢Δ⁢𝛉‖norm1𝛼Δ𝛉\left\|(1+\alpha)\Delta{\bm{\theta}}\right\|∥ ( 1 + italic_α ) roman_Δ bold_italic_θ ∥ remains small, we can reformulate the first-order approximation as:

According to Equation 3, we approximately have ω⁢(𝜽2)>ω⁢(𝜽1)𝜔subscript𝜽2𝜔subscript𝜽1\omega({\bm{\theta}}_{2})>\omega({\bm{\theta}}_{1})italic_ω ( bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) > italic_ω ( bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ). This suggests that, starting from a partially-aligned model ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and its SFT checkpoint ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, by selecting appropriate α>0𝛼0\alpha>0italic_α > 0, we can construct a new model ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT parameterized by 𝜽2subscript𝜽2{\bm{\theta}}_{2}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT through amplifying the parameter change Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ:

such that ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT achieves better alignment performance than ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. Consequently, we improve the implicit optimization objective ω⁢(𝛉)𝜔𝛉\omega({\bm{\theta}})italic_ω ( bold_italic_θ ) of alignment training without requiring additional training.

Since the process of Equation 6 essentially “extrapolates” the parameters of ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT along the line connecting 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, we refer to the procedure defined by Equation 6 as ExPO (model extrapolation). Figure 2 illustrates the ExPO method, where the orange curve from 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT indicates the actual training trajectory from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, and the straight orange line from 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to 𝜽2subscript𝜽2{\bm{\theta}}_{2}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT denotes the extrapolation from ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT. In practice, the hyperparameter α𝛼\alphaitalic_α in Equation 6 (controlling the extrapolation length) can be tuned using inference-level computational resources. For example, hyperparameter search for a 7B model requires only a single A10 24GB GPU, while a 70B model needs two A100 80GB GPUs. As high-performance LLM inference frameworks like vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2023c) continue to rapidly develop, the costs of hyperparameter search will keep decreasing.

Figure 2: The orange curve indicates the training trajectory from 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, while the orange line denotes the extrapolation from 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT along Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ, thus producing 𝜽2subscript𝜽2{\bm{\theta}}_{2}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT.

Connection to Model Averaging/Interpolation

It is worth noting that the idea of “model averaging” has been explored in prior work. Specifically, previous work has discovered that deep neural networks often exhibit mode connectivity (Garipov et al., 2018; Entezari et al., 2022; Zhao et al., 2020; Frankle et al., 2020). This property implies that between two local optima in the parameter space, there typically exists a path where model performance (e.g., validation accuracy or loss) does not degrade significantly during traversal. Empirical studies (Izmailov et al., 2018; Lin et al., 2024; Wortsman et al., 2022) have shown that even with simple linear interpolation paths between two local optima, the loss along the path remains low, and performance often lies between the original models, which is consistent with our observations in Figure 1. Recent LLM research (Lin et al., 2023; Yu et al., 2024; Akiba et al., 2024; Goddard et al., 2024) has further explored interpolation across multiple fine-tuned models (i.e., models initialized from the same pre-trained checkpoint but fine-tuned on different data) to create new models with combined capabilities. Note that Equation 6 can be rewritten as: 𝜽2=(1−γ)⁢𝜽0+γ⁢𝜽1subscript𝜽21𝛾subscript𝜽0𝛾subscript𝜽1{\bm{\theta}}_{2}=(1-\gamma){\bm{\theta}}_{0}+\gamma{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = ( 1 - italic_γ ) bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_γ bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, which means ExPO can be viewed as a generalized form of model interpolation with weights exceeding 1. Hence, the hypothesis we formulated based on the characteristics of preference alignment (i.e., small parameter changes) and the derived ExPO method essentially extend the weight range of traditional model interpolation (from [0,1]01[0,1][ 0 , 1 ] to (1,+∞)1(1,+\infty)( 1 , + ∞ )).

In the following sections, we will conduct extensive experiments to validate the effectiveness of ExPO in reducing the computational costs of preference alignment training.

3 Controlled Experiments

3.1 Setup and Evaluation Protocol

Models and Training Recipe

Our controlled experiments are based on the training recipe of the zephyr-7b-dpo model. Specifically, we use the UltraFeedback (Cui et al., 2023) dataset for model training, which contains diverse instruction-response pairs with GPT-4-annotated preference labels and is split into 61K and 1K data as the training and development sets, respectively. For DPO training, we use zephyr-7b-dpo’s SFT checkpoint for model initialization and as the reference model. We adopt the global batch size of 128, the learning rate of 5e-7, and the AdamW optimizer (Loshchilov and Hutter, 2019). Note that while zephyr-7b-dpo is trained for 478 steps in total (i.e., one epoch), in § 3.2 we will vary the training steps, or equivalently, the training data size. We train the models on 8 A100 80GB GPUs.

Inference Details

We employ the vLLM (Kwon et al., 2023) library for high-throughput model inference. We use top-k𝑘kitalic_k (k=40𝑘40k=40italic_k = 40) and nucleus sampling (Holtzman et al., 2020) (p=0.9𝑝0.9p=0.9italic_p = 0.9) with a temperature of 0.7. To avoid repetition in generated texts, we set both the factors of presence penalty and frequency penalty to 0.1. We set the sampling random seed to 42.

Hyperparameter Search

To determine the optimal α𝛼\alphaitalic_α value in ExPO, we use a combination of binary search and grid search with manually tuned intervals (see Appendix B for details). We select the α𝛼\alphaitalic_α giving the highest expected reward on the UltraFeedback development set (1K instructions), as calculated by the reward model RM-Mistral-7B.

Evaluation Protocol

We resort to AlpacaEval 2.0 (Li et al., 2023) for model evaluation, which is a leading benchmark that assesses LLMs’ instruction-following ability and their alignment with human preferences. It contains a fixed set of 805 instructions chosen to be representative of real user cases. For each instruction, it calculates the probability that a GPT-4 Turbo evaluator prefers the output of the evaluated model over the GPT-4 baseline, thus providing an affordable and replicable alternative to human annotation. The win rate over the GPT-4 baseline is computed as the expected preference probability, while the length-controlled (LC) win rate (Dubois et al., 2024) alleviates the length bias of the GPT-4 Turbo evaluator (i.e., the prior preference toward longer responses).

In § 3.2, we report both the raw and LC win rates, as well as the expected reward score over the 805 instructions calculated. For subsequent experiments, unless otherwise stated, we report the expected reward score on the UltraFeedback development set (1K instructions) for ease of analysis.

Table 2: Evaluation results on AlpacaEval 2.0 of applying ExPO to DPO models trained with varying steps (ℳ1∗superscriptsubscriptℳ1\mathcal{M}_{1}^{*}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT).

Figure 3: Reward distribution on UltraFeedback (development set) for the extrapolated models in Table 2.

Figure 4: ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT’s reward scores and response lengths on UltraFeedback (development set) varying with α𝛼\alphaitalic_α (x-axis) for the partially-trained DPO models in § 3.2. Dashed vertical lines correspond to the optimal α𝛼\alphaitalic_α values.α=0𝛼0\alpha=0italic_α = 0 indicates that ExPO is not applied (i.e., ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT).

3.2 Analysis of Varying Training Steps

We first investigate whether ExPO can enhance LLMs with limited alignment training. Given that the full training of zephyr-7b-dpo consists of 478 steps (one epoch over the UltraFeedback training data), we initialize from the same SFT checkpoint (ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT) and use the aforementioned training configuration to train DPO models (ℳ1∗superscriptsubscriptℳ1\mathcal{M}_{1}^{*}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT) with 10%, 20%, and 40% of the full training steps. We directly use zephyr-7b-dpo as the 100%-step (full-training) model ℳ1100%superscriptsubscriptℳ1percent100\mathcal{M}_{1}^{100\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT. For these DPO models, we apply ExPO to derive extrapolated models ℳ2∗superscriptsubscriptℳ2\mathcal{M}_{2}^{*}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT.

Main Results

As shown in Table 2, while fewer training steps generally yield lower alignment performance, ExPO effectively bridges the gap caused by reduced training steps. For example, ExPO boosts ℳ110%superscriptsubscriptℳ1percent10\mathcal{M}_{1}^{10\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT’s LC win rate from 10.4% to ℳ210%superscriptsubscriptℳ2percent10\mathcal{M}_{2}^{10\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT’s 16.3% and ℳ120%superscriptsubscriptℳ1percent20\mathcal{M}_{1}^{20\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT from 12.9% to ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT’s 21.3%, enabling these extrapolated models to match or even surpass the fully-trained ℳ1100%superscriptsubscriptℳ1percent100\mathcal{M}_{1}^{100\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT.

Hyperparameter Search Analysis

The optimal α𝛼\alphaitalic_α values for ℳ210%superscriptsubscriptℳ2percent10\mathcal{M}_{2}^{10\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT, ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT, ℳ240%superscriptsubscriptℳ2percent40\mathcal{M}_{2}^{40\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 40 % end_POSTSUPERSCRIPT, and ℳ2100%superscriptsubscriptℳ2percent100\mathcal{M}_{2}^{100\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT are 8.0, 2.5, 0.5, and 0.3, respectively. Figure 3 illustrates the reward distributions of these extrapolated models, showing that their response distributions shift toward higher reward regions compared to the original ℳ1∗superscriptsubscriptℳ1\mathcal{M}_{1}^{*}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∗ end_POSTSUPERSCRIPT models. In Figure 4, we show that increasing α𝛼\alphaitalic_α within a reasonable range consistently improves alignment performance. However, excessively large α𝛼\alphaitalic_α causes sharp performance drops and abnormal response length increases (e.g., generating gibberish or failing to terminate). This indicates that overly large α𝛼\alphaitalic_α violates the first-order approximation (Equation 4) as ‖(1+α)⁢Δ⁢𝜽‖norm1𝛼Δ𝜽\left\|(1+\alpha)\Delta{\bm{\theta}}\right\|∥ ( 1 + italic_α ) roman_Δ bold_italic_θ ∥ becomes too large. Additionally, since more training steps lead to larger ‖Δ⁢𝜽‖normΔ𝜽\left\|\Delta{\bm{\theta}}\right\|∥ roman_Δ bold_italic_θ ∥, smaller α𝛼\alphaitalic_α values are required for models with more training steps (e.g., ℳ1100%superscriptsubscriptℳ1percent100\mathcal{M}_{1}^{100\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT) to maintain the validity of Equation 4, which is consistent with our hyperparameter search results.

Table 3: Ablation results on UltraFeedback (development set) of adjusting training data quality. “N/A” denotes that the reward score does not improve after applying ExPO with the smallest α=0.1𝛼0.1\alpha=0.1italic_α = 0.1.

Computational Cost Analysis

The fully-trained model ℳ1100%superscriptsubscriptℳ1percent100\mathcal{M}_{1}^{100\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT requires about 12 GPU hours (A100 80GB). In contrast, ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT’s hyperparameter search takes about 0.5 GPU hour, and combined with ℳ120%superscriptsubscriptℳ1percent20\mathcal{M}_{1}^{20\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT’s about 2.5-hour training, the total cost is about 3 GPU hours, leading to a 75% reduction compared to full training while achieving comparable or better alignment performance. Moreover, ExPO’s hyperparameter search, which only involves model inference, also significantly reduces hardware requirements, e.g., a 7B model requires only a single A10 24GB GPU for search, whereas training typically needs 8 A100 80GB GPUs. The above results reaffirm the soundness of the first-order approximation and demonstrate ExPO’s effectiveness in reducing computational costs for LLM alignment.

Other Observations

We also observe two other noteworthy phenomena: (1) Extrapolated alignment performance does not strictly increase with training steps. For example, ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT outperforms ℳ2100%superscriptsubscriptℳ2percent100\mathcal{M}_{2}^{100\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT, suggesting ExPO’s efficacy depends on factors like training data and hyperparameters. We will explore these factors in § 3.3 and 3.4. (2) Even fully trained models like ℳ1100%superscriptsubscriptℳ1percent100\mathcal{M}_{1}^{100\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT benefit from ExPO (LC win rate increases by 2.8%), indicating that existing already-aligned models may not be fully optimized, and ExPO can fill this gap. We will apply ExPO to more existing, already-aligned models in § 4.1.

3.3 Analysis of Training Data Quality

In the previous section, we observed that alignment performance after model extrapolation does not strictly improve with increased training steps. We conjecture that this occurs because more training makes the model more prone to learning spurious features from data, such as length bias222In the UltraFeedback training set, preferred and non-preferred responses have average lengths of 319 and 277 tokens, respectively. (Park et al., 2024). According to Equation 6, under our controlled experimental setup where all ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT are initialized from the same SFT model ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and 𝜽0subscript𝜽0{\bm{\theta}}_{0}bold_italic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, the highest achievable performance of the extrapolated model ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT is uniquely determined by Δ⁢𝛉Δ𝛉\Delta{\bm{\theta}}roman_Δ bold_italic_θ. Hence, ExPO’s effectiveness requires Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ to indicate the direction that genuinely improves alignment performance. Learning spurious features like length bias degrades the “quality” of Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ, thus undermining the extrapolation performance. Figure 5 illustrates this phenomenon: as training steps increase (from 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to 𝜽1′superscriptsubscript𝜽1′{\bm{\theta}}_{1}^{\prime}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT), the model can learn spurious features from training data, leading to the degraded alignment performance of extrapolated models (e.g., 𝜽2′superscriptsubscript𝜽2′{\bm{\theta}}_{2}^{\prime}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT underperforms 𝜽2subscript𝜽2{\bm{\theta}}_{2}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT).

Figure 5: Increasing training steps (from 𝜽1subscript𝜽1{\bm{\theta}}_{1}bold_italic_θ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to 𝜽1′subscriptsuperscript𝜽′1{\bm{\theta}}^{\prime}_{1}bold_italic_θ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT) can make the model more prone to learning spurious features from training data, such as length bias. This consequently impairs the direction of Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ and the achievable performance of ExPO (e.g., 𝜽2′subscriptsuperscript𝜽′2{\bm{\theta}}^{\prime}_{2}bold_italic_θ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT underperforms 𝜽2subscript𝜽2{\bm{\theta}}_{2}bold_italic_θ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT).

Table 4: Ablation results of the training epochs, learning rate, and optimizer on UltraFeedback (development set).

To analyze how training data quality affects ExPO’s effectiveness in a controlled manner, we take length bias as an example and manually inject length bias into the training data. Unlike the random sampling in § 3.2, we sort the UltraFeedback training data by the length difference between preferred and non-preferred responses in descending order. We then train models on the sorted samples orderly so that models will prioritize learning from samples with larger length differences. From Table 3, while introducing length bias temporarily boosts reward scores (ℳ110%,bsuperscriptsubscriptℳ1percent10b\mathcal{M}_{1}^{10\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % , roman_b end_POSTSUPERSCRIPT and ℳ120%,bsuperscriptsubscriptℳ1percent20b\mathcal{M}_{1}^{20\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % , roman_b end_POSTSUPERSCRIPT outperform ℳ110%superscriptsubscriptℳ1percent10\mathcal{M}_{1}^{10\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT and ℳ120%superscriptsubscriptℳ1percent20\mathcal{M}_{1}^{20\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT), extrapolated models consistently underperform (ℳ210%,bsuperscriptsubscriptℳ2percent10b\mathcal{M}_{2}^{10\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % , roman_b end_POSTSUPERSCRIPT and ℳ220%,bsuperscriptsubscriptℳ2percent20b\mathcal{M}_{2}^{20\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % , roman_b end_POSTSUPERSCRIPT are worse than ℳ210%superscriptsubscriptℳ2percent10\mathcal{M}_{2}^{10\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT and ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT). Moreover, the optimal α𝛼\alphaitalic_α values for ℳ210%,bsuperscriptsubscriptℳ2percent10b\mathcal{M}_{2}^{10\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % , roman_b end_POSTSUPERSCRIPT and ℳ220%,bsuperscriptsubscriptℳ2percent20b\mathcal{M}_{2}^{20\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % , roman_b end_POSTSUPERSCRIPT are 0.2 and 0.4, which are far smaller than those for ℳ210%superscriptsubscriptℳ2percent10\mathcal{M}_{2}^{10\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT (8.0) and ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT (2.5). For ℳ140%,bsuperscriptsubscriptℳ1percent40b\mathcal{M}_{1}^{40\%,\mathrm{b}}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 40 % , roman_b end_POSTSUPERSCRIPT, ExPO even fails to yield any improvement. These results demonstrate that training on biased or low-quality data (e.g., with length bias) causes Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ to fail to indicate the direction that genuinely improves alignment performance, thereby diminishing the benefits of model extrapolation.

3.4 Analysis of Training Configurations

Next, we analyze how specific training hyperparameters influence ExPO’s effectiveness. Since ExPO amplifies the parameter change Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, we investigate whether ExPO is equivalent to directly increasing the magnitude of parameter changes, such as by raising the training epochs or learning rate. Additionally, since the training trajectory from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (and the resulting Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ) is closely tied to the gradient descent algorithm, we also explore the impact of the optimizer on ExPO’s effectiveness. All experiments use the model trained with 20% steps in § 3.2 as the baseline and follow the default training data and configurations.

Training Epochs and Learning Rate

We increase the training epochs or learning rate for ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. Table 4 shows that while both adjustments improve ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT’s performance, they also reduce the benefits of model extrapolation (lower ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT performance) and yield smaller optimal α𝛼\alphaitalic_α values. Meanwhile, the ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT models trained with more epochs or larger learning rates generate significantly longer responses compared to the default setup. This suggest that both adjustments also make models prone to learning the length bias in training data, thereby degrading Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ’s quality and the gains from ExPO. Notably, when training epochs are set to 3, ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT cannot benefit from ExPO, likely because the first-order approximation (Equation 4) no longer holds as ‖Δ⁢𝜽‖normΔ𝜽\left\|\Delta{\bm{\theta}}\right\|∥ roman_Δ bold_italic_θ ∥ becomes too large.

Optimizer

We train ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT using three popular optimizers: AdamW (Loshchilov and Hutter, 2019) (default), AdaGrad (Duchi et al., 2011), and RMSprop (Hinton, 2012). Table 4 shows that while AdaGrad converges slowest (lowest ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT performance), it achieves the highest extrapolated alignment performance (ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT), slightly surpassing AdamW. Conversely, RMSprop, while yielding the best ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT performance, results in the poorest ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT performance. AdamW, as the dominant optimizer in modern LLM training, strikes a balance between convergence efficiency and extrapolated performance. These results highlight that different optimizers significantly affect Δ⁢𝜽Δ𝜽\Delta{\bm{\theta}}roman_Δ bold_italic_θ’s quality and extrapolation outcomes.

Table 5: Evaluation results on AlpacaEval 2.0 and MT-Bench of applying ExPO to existing DPO/RLHF LLMs.

4 Extended Applications of ExPO

4.1 Applying ExPO to More Existing, Already-aligned LLMs

In § 3.2, we observed that ExPO also brings noticeable performance improvements to the fully-trained zephyr-7b-dpo. This motivates us to apply ExPO to more existing, already-aligned LLMs. As hypothesized in § 1, the normally-trained models should also satisfy the first-order approximation premise, i.e., ‖Δ⁢𝜽‖normΔ𝜽\left\|\Delta{\bm{\theta}}\right\|∥ roman_Δ bold_italic_θ ∥ is small. We select twelve open-source models from HuggingFace for experiments (see Appendix C for their model IDs):

• •
  Five models trained via offline DPO, including zephyr-7b-alpha/beta (Tunstall et al., 2023) and tulu2-7/13/70b (Wang et al., 2023);
• •
  Two models trained via iterative DPO, including snorkel-7b-iter (Tran et al., 2023) and llama3-8b-iter (Dong et al., 2024);
• •
  Five models trained via online RLHF, including starling-7b-alpha/beta (Zhu et al., 2023) and internlm2-1.8/7/20b (Cai et al., 2024).

These models cover a diverse range of model sizes (from 1.8B to 70B) and span three mainstream alignment algorithms widely used in practice.

Based on our hyperparameter search experience for zephyr-7b-dpo in § 3.2 (Appendix B), for the twelve models above, we conduct a simple grid search for the optimal α𝛼\alphaitalic_α, using the interval of 0.1 within [0.1, 0.5]. In addition to AlpacaEval 2.0, we also evaluate these models on MT-Bench (Zheng et al., 2023b), another leading benchmark for assessing instruction-tuned LLMs’ general and multi-turn ability. It contains a set of challenging multi-turn open-ended questions covering topics such as writing, role-playing, math, coding, and more. The model-generated answers are judged by GPT-4 via a scalar score (from 1 to 10).

In Table 5, we show that ExPO consistently improves the evaluated LLMs, with notable improvements of up to 10.1% win rate and 4.5% LC win rate on AlpacaEval 2.0 (for internlm2-20b and tulu2-70b, respectively) and 0.37 on MT-Bench (for llama3-8b-iter). This suggests that existing, already-aligned LLMs may still not have been trained to optimality or “saturation”.ExPO offers a practical and efficient means to compensate for potential inadequate training of existing LLMs (or, squeeze more alignment performance out of these models), as it only requires inference-level hardware resources and bypasses the costly additional training overhead.

Table 6: Evaluation results on UntraFeedback of applying ExPO to models trained via different algorithms.

4.2 Applying ExPO to More Alignment Algorithms

So far, we have primarily applied ExPO to models trained via the dominant DPO or RLHF algorithms (§ 3 and 4.1). Since ExPO does not assume the specific training method for ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, we expect that ExPO can be applied to models trained via other algorithms than DPO or RLHF. To this end, we use a series of Mistral/LLaMA-3 models released by Meng et al. (2024), which are trained via various alignment algorithms and are all initialized from the same SFT checkpoints. These algorithms include: RRHF (Yuan et al., 2023),SLiC-HF (Zhao et al., 2023a), IPO (Azar et al., 2024), CPO (Xu et al., 2024), KTO (Ethayarajh et al., 2024), R-DPO (Park et al., 2024), and SimPO (Meng et al., 2024). We refer readers to Meng et al. (2024) for elaboration on these algorithms’ optimization objectives as well as the models’ training configurations. Following the previous experience, we search the optimal α𝛼\alphaitalic_α value within the range of [0.1, 0.5] with the interval of 0.1.

As shown in Table 6, ExPO effectively complements various alignment training algorithms. While these models have been carefully tuned according to Meng et al. (2024), they still benefit from model extrapolation. This indicates that ExPO does not rely on specific alignment algorithms but instead generalizes across diverse methods, showcasing its broad compatibility and practical utility.

4.3 Discussion on Failure Cases

Finally, we discuss the failure cases we encountered when applying ExPO to more various models. (1) ExPO supposes ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is an SFT model and ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT is one that further undergoes alignment training. However, when we attempted with a pre-trained model as ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and an SFT one as ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, we found that model extrapolation usually cannot improve alignment performance and can even lead to model collapse (e.g., the extrapolated model struggles to generate the EOS token or mistakenly generates special tokens). We speculate that this is because SFT typically adopts a larger learning rate and more training steps, and serves to adapt models to the chat templates (Zheng, 2024), so new knowledge is actually injected into models. (2) Another type of failure cases is also related to model overfitting. For example, the Storm-7B model (Liu et al., 2024a) is trained via iterative DPO for three iterations. When experimenting with this model, we found that applying ExPO with even the very small α=0.1𝛼0.1\alpha=0.1italic_α = 0.1 results in severe model collapse, probably because the model overfits to its employed reward model during iterative DPO training.

In both cases, ExPO’s underlying first-order approximation can become invalidated as the resulting ‖Δ⁢𝜽‖normΔ𝜽\left\|\Delta{\bm{\theta}}\right\|∥ roman_Δ bold_italic_θ ∥ is too large. Therefore, we suggest that more deliberate strategies are needed when applying ExPO to models with large parameter changes, e.g., by leveraging the intermediate checkpoints. We note that recent work has made promising exploration (Lin et al., 2025) and expect more follow-up studies in future work.

5 Conclusion

This work demonstrates the efficacy of the ExPO (model extrapolation) method in enabling more efficient LLM alignment with human preferences.ExPO builds upon the hypothesis that alignment training typically involves only small changes of model parameters. Given a partially-trained model ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and its initial SFT checkpoint, ExPO improves the implicit optimization objective of alignment training by simply amplifying the parameter change based on a first-order approximation, thus directly achieving better alignment performance without additional training overhead. We empirically validate ExPO’s effectiveness through controlled experiments, where we show that the DPO model trained with 20% steps can be boosted to outperform the fully-trained one. Furthermore, we extend ExPO’s application to twelve existing, already-aligned LLMs, showing that ExPO consistently improves their performance on the mainstream LLM benchmarks AlpacaEval 2.0 and MT-Bench. This suggests that ExPO can also serve as a practical and efficient means to compensate for potential inadequate alignment training of existing LLMs. Overall, our work highlights the utility of model extrapolation in efficient LLM alignment, which can inspire future research in this direction.

6 Limitations

Hyperparameter Search

The current ExPO adopts the simplest form of uniform extrapolation and requires manual hyperparameter search for α𝛼\alphaitalic_α. Future work could explore how to determine the optimal α𝛼\alphaitalic_α automatically and adaptively (i.e., using different α𝛼\alphaitalic_α values for different model modules). For example, the information from optimizer states and parameter gradients during the later phase of alignment training could be useful for this purpose.

Alignment Tax

While ExPO makes substantial improvements in instruction-following ability and alignment with human preferences, this seems not “free” and can instead incur an additional alignment tax, a widely observed issue in human preference optimization algorithms (Ouyang et al., 2022; Dong et al., 2024; Meng et al., 2024), which indicates the possible fluctuations or drops in downstream task performance after alignment training. We evaluate the models in § 3.2 and 4.1 on the six downstream tasks (Clark et al., 2018; Zellers et al., 2019; Hendrycks et al., 2021; Lin et al., 2022; Sakaguchi et al., 2021; Cobbe et al., 2021) from the Open LLM Leaderboard333We employ the evaluation implementation of Eleuther’s lm-evaluation-harness (version 0.4.4). Note that the mismatch of input templates used for chat-style evaluations (e.g., AlpacaEval 2.0 and MT-Bench) and for these downstream task evaluations could also contribute to the observed alignment tax, as discussed in Meng et al. (2024). (v1; Beeching et al. 2023). We find that in most cases, ExPO amplifies the alignment tax introduced by the alignment training (from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT). For example, for the partially-trained models in § 3.2 (Appendix D, Figure 6), the original DPO models (ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT) show improvements over the initial SFT model (ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT) on TruthfulQA and declines on GSM8K, while applying ExPO (ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT) leads to further improvements or declines, respectively. For the existing, already-aligned LLMs in § 4.1, the amplification of the alignment tax by ExPO is usually smaller as shown in Figure 7 in Appendix D, suggesting a trade-off between the alignment training overhead (from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT) and the additional alignment tax brought by ExPO (from ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT).

Acknowledgements

We thank Sidi Lu, Yufei Tian, Zi-Yi Dou, and other members of the UCLA PlusLab & NLP group as well as anonymous reviewers for their constructive feedback and discussions.

This work was supported by an Amazon AGI Research Award through UCLA-Amazon Science Hub and a National Science Foundation CAREER award #2339766. This work was also supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and China Scholarship Council (with No. 202306210211).

References

• Akiba et al. (2024) Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187.
• Azar et al. (2024) Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR.
• Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
• Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard.
• Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.
• Cai et al. (2024) Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297.
• Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
• Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
• Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377.
• Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863.
• Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
• Dubois et al. (2024) Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.
• Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121–2159.
• Entezari et al. (2022) Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2022. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations.
• Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.
• Frankle et al. (2020) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259–3269. PMLR.
• Garipov et al. (2018) Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, volume 31.
• Gemma et al. (2024) Team Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.
• Goddard et al. (2024) Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee’s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257.
• He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
• Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
• Hinton (2012) Geoffrey Hinton. 2012. Rmsprop: Divide the gradient by a running average of its recent magnitude. https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf. Coursera Lecture 6e of Neural Networks for Machine Learning.
• Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
• Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702.
• Izmailov et al. (2018) P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. 2018. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876–885.
• Ji et al. (2024) Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. 2024. Aligner: Achieving efficient alignment through weak-to-strong correction. In Advances in Neural Information Processing Systems.
• Ji et al. (2023) Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852.
• Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
• Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.
• Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
• Lin et al. (2025) Yiguan Lin, Bin Xu, Yinghao Li, and Yang Gao. 2025. Extrapolation merging: Keep improving with extrapolation and merging. arXiv preprint arXiv:2503.04834.
• Lin et al. (2023) Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2023. Mitigating the alignment tax of rlhf.
• Lin et al. (2024) Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. 2024. Spurious feature diversification improves out-of-distribution generalization. In International Conference on Learning Representations.
• Liu et al. (2021) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691–6706, Online. Association for Computational Linguistics.
• Liu et al. (2024a) Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Han-Sen Zhong, and Wanli Ouyang. 2024a. Iterative length-regularized direct preference optimization: A case study on improving 7b language models to gpt-4 level. arXiv preprint arXiv:2406.11817.
• Liu et al. (2024b) Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares-López, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. 2024b. Decoding-time realignment of language models. In International Conference on Machine Learning.
• Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
• Lu et al. (2024) Sidi Lu, Hongyi Liu, Asli Celikyilmaz, Tianlu Wang, and Nanyun Peng. 2024. Open-domain text evaluation via contrastive distribution modeling. In International Conference on Machine Learning.
• Meng et al. (2024) Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems.
• OpenAI (2022) OpenAI. 2022. https://chat.openai.com.chat.
• OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
• Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744.
• Park et al. (2024) Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pages 4998–5017, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
• Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
• Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.
• Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
• Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
• Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
• Tran et al. (2023) Hoang Tran, Chris Glaze, and Braden Hancock. 2023. Iterative dpo alignment. Technical report, Snorkel AI.
• Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944.
• Wang et al. (2023) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning on open resources. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
• Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965–23998. PMLR.
• Xu et al. (2024) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In International Conference on Machine Learning.
• Yu et al. (2024) Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning.
• Yuan et al. (2023) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback. In Advances in Neural Information Processing Systems.
• Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
• Zhao et al. (2020) Pu Zhao, Pin-Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. 2020. Bridging mode connectivity in loss landscapes and adversarial robustness. In International Conference on Learning Representations.
• Zhao et al. (2023a) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023a. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425.
• Zhao et al. (2023b) Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2023b. Calibrating sequence likelihood improves conditional language generation. In International Conference on Learning Representations.
• Zheng (2024) Chujie Zheng. 2024. Chat templates for huggingface large language models. https://github.com/chujiezheng/chat_templates.
• Zheng et al. (2023a) Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. 2023a. Click: Controllable text generation with sequence likelihood contrastive learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1022–1040, Toronto, Canada. Association for Computational Linguistics.
• Zheng et al. (2023b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623.
• Zheng et al. (2023c) Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2023c. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104.
• Zhu et al. (2023) Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. 2023. Starling-7b: Improving llm helpfulness & harmlessness with rlaif.

Appendix A Related Work

LLM Alignment

Modern large language models (LLMs) are first pre-trained on massive textual corpora with the unsupervised language modeling objective (Brown et al., 2020; Touvron et al., 2023b; Dubey et al., 2024), and then fine-tuned to learn to follow human instructions (OpenAI, 2022, 2023; Ji et al., 2023). The current fine-tuning paradigm typically contains two steps: supervised fine-tuning (SFT) and human preference optimization. Our work focuses on the later step, which aims to adjust the model’s response distribution to better align with human preferences. In this process, the model is usually trained on preference data (“A is better than B”; Zhao et al. 2023b; Zheng et al. 2023a), thus learning to assign higher probabilities to human-preferred responses over the disfavored ones. Common implementations for human preference optimization include Reinforcement Learning from Human Feedback (RLHF; Ouyang et al. 2022; Schulman et al. 2017), Direct Preference Optimization (DPO; Rafailov et al. 2023), and many other DPO’s variants or competitors (Azar et al., 2024; Xu et al., 2024; Ethayarajh et al., 2024; Park et al., 2024; Meng et al., 2024). Given LLMs’ gigantic parameters, the processes from pre-training to SFT and the alignment training still require expensive computational resources. Therefore, exploring more efficient alignment methods to reduce training overhead has always been an important and compelling research challenge (Ji et al., 2024). To address this challenge, we propose the ExPO method, which has demonstrated promising efficacy in expediting LLM alignment.

There is another line of work that attempts to bypass the expensive alignment training by blending multiple models’ token predictions during the inference time (Liu et al., 2021; Lu et al., 2024; Liu et al., 2024b), usually referred to as inference-time alignment methods. In comparison to ExPO, these inference-time methods often require more complex and varied implementations of model inference, which are not typically supported by existing high-performance LLM inference infrastructures (e.g., vLLM). This inconvenience not only reduces the practical efficiency of model inference but also significantly increases the cost of their hyperparameter search processes. In contrast, ExPO only involves regular inference of a single model, which can be seamlessly supported by existing infrastructures, thereby inheriting the merit in inference efficiency.

Model Averaging/Interpolation

Model averaging/interpolation is a commonly used technique in machine learning. It utilizes multiple models trained with different random initializations or data subsets and interpolates the weights of these models to obtain a new model with stronger out-of-distribution generalization (Izmailov et al., 2018; Lin et al., 2024; Wortsman et al., 2022; Lin et al., 2023). This technique is based on the mode connectivity of neural networks (Garipov et al., 2018; Entezari et al., 2022; Zhao et al., 2020; Frankle et al., 2020). Specifically, prior work found that multiple local optima in the parameter space can often be connected by low-loss (linear) paths, particularly for models with residual connection structures (He et al., 2016). This can explain why model interpolation can produce new, functional models when applied to LLMs (as our observations in Figure 1), as residual connection has become a dominant choice of architecture design in modern LLMs like LLaMA (Touvron et al., 2023a). We notice that recent LLMs have widely adopted model interpolation, as exemplified by Gemma-2 (Gemma et al., 2024) and LLaMA-3 (Dubey et al., 2024), possibly also for further enhancement in out-of-distribution generalization.

Appendix B Hyperparameter Search Details

We use the experiments in Table 2 as an example to illustrate how we conduct hyperparameter search.

Starting with ℳ210%superscriptsubscriptℳ2percent10\mathcal{M}_{2}^{10\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 10 % end_POSTSUPERSCRIPT:

(1) First, with an interval of 5, we tried α=5𝛼5\alpha=5italic_α = 5 and α=10𝛼10\alpha=10italic_α = 10. We found that both significantly outperformed ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, but (α=5)>(α=10)𝛼5𝛼10(\alpha=5)>(\alpha=10)( italic_α = 5 ) > ( italic_α = 10 ). (2) Then, setting the search range to [5,10]510[5,10][ 5 , 10 ] with an interval of 1, we applied binary search and tried α=7𝛼7\alpha=7italic_α = 7 and α=8𝛼8\alpha=8italic_α = 8. We found that (α=8)>(α=7)𝛼8𝛼7(\alpha=8)>(\alpha=7)( italic_α = 8 ) > ( italic_α = 7 ). We then tried α=9𝛼9\alpha=9italic_α = 9 and found (α=8)>(α=9)𝛼8𝛼9(\alpha=8)>(\alpha=9)( italic_α = 8 ) > ( italic_α = 9 ). (3) We thus determined α=8𝛼8\alpha=8italic_α = 8 as optimal.

Note that smaller search intervals might yield better results, but we deem this unnecessary in practice.

Then, for ℳ220%superscriptsubscriptℳ2percent20\mathcal{M}_{2}^{20\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT:

(1) With previous experience, we first tried α=2𝛼2\alpha=2italic_α = 2 and α=4𝛼4\alpha=4italic_α = 4 with an interval of 2. We found that α=2𝛼2\alpha=2italic_α = 2 significantly outperformed ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, but α=4𝛼4\alpha=4italic_α = 4 performed worse than ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. (2) Then, setting search ranges to [1,2]12[1,2][ 1 , 2 ] and [2,4]24[2,4][ 2 , 4 ] with an interval of 1, we applied binary search and tried α=1𝛼1\alpha=1italic_α = 1 and α=3𝛼3\alpha=3italic_α = 3. We found that (α=2)>(α=3)>(α=1)𝛼2𝛼3𝛼1(\alpha=2)>(\alpha=3)>(\alpha=1)( italic_α = 2 ) > ( italic_α = 3 ) > ( italic_α = 1 ). (3) Next, with an interval of 0.5 in [2,3]23[2,3][ 2 , 3 ], we tried α=2.5𝛼2.5\alpha=2.5italic_α = 2.5 and found (α=2.5)>(α=2)𝛼2.5𝛼2(\alpha=2.5)>(\alpha=2)( italic_α = 2.5 ) > ( italic_α = 2 ). (4) We thus determined α=2.5𝛼2.5\alpha=2.5italic_α = 2.5 as optimal.

This took 5 searches in total, each taking about 5min (using one A100 80GB, including inference on development set and reward model scoring), totaling about 0.5 GPU hours.

Next, for ℳ240%superscriptsubscriptℳ2percent40\mathcal{M}_{2}^{40\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 40 % end_POSTSUPERSCRIPT:

(1) Based on previous experience, we first tried α=0.5𝛼0.5\alpha=0.5italic_α = 0.5 and found it outperformed ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT. (2) Then with an interval of 0.1, we applied grid search and tried α=0.6𝛼0.6\alpha=0.6italic_α = 0.6 and α=0.4𝛼0.4\alpha=0.4italic_α = 0.4. We found that α=0.6𝛼0.6\alpha=0.6italic_α = 0.6 performed worse than ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, while (α=0.5)>(α=0.4)𝛼0.5𝛼0.4(\alpha=0.5)>(\alpha=0.4)( italic_α = 0.5 ) > ( italic_α = 0.4 ). (3) We thus determined α=0.5𝛼0.5\alpha=0.5italic_α = 0.5 as optimal.

Note that the search experience for ℳ240%superscriptsubscriptℳ2percent40\mathcal{M}_{2}^{40\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 40 % end_POSTSUPERSCRIPT is a key motivation for us to use [0.1, 0.5] as search range with 0.1 interval for ℳ2100%superscriptsubscriptℳ2percent100\mathcal{M}_{2}^{100\%}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT and models in § 4.1.

Summary

Overall, we (and in practice) do not search blindly, but flexibly combine binary search, grid search, and dynamically adjusted search intervals.These strategies are simple, practical, and represent consensus in practice.It is also noteworthy that the above search only requires inference-level GPU hardware (e.g., A10 24GB). Therefore, compared to the reduced training overhead (from 12 GPU hours for ℳ1100%superscriptsubscriptℳ1percent100\mathcal{M}_{1}^{100\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 100 % end_POSTSUPERSCRIPT to 2.5 GPU hours for ℳ120%superscriptsubscriptℳ1percent20\mathcal{M}_{1}^{20\%}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 20 % end_POSTSUPERSCRIPT) and training-level GPU hardware (from eight A100 80GB to one A10 24GB), the α𝛼\alphaitalic_α search process in ExPO is more economical and efficient.

Table 7: Hyperparameter search results for α𝛼\alphaitalic_α in § 3.2 and 4.1.

Appendix C HuggingFace Models

Appendix D Supplementary Experimental Results of Alignment Tax (§ 6)

Figure 6: Evaluation results for the models in § 3.2 on downstream tasks. The x-axis denotes the proportions of training steps. As the “cost” of simply improving instruction-following ability and alignment with human preferences, ExPO can also amplify the alignment tax introduced by the alignment training.

Figure 7: Evaluation results for the LLMs in § 4.1 on downstream tasks. For these already-alighed models, the additional alignment tax brought by ExPO is usually smaller, suggesting a trade-off between the alignment training overhead (from ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT) and the additional alignment tax brought by ExPO (from ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT to ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT).