Mix Your Base and RLVR Weights:
A Training-Free Knob for Diversity

Youngseong Kim

Independent Researcher

RLVR makes a model better at its first try but worse at exploring. One number, α, mixes the weights back toward the base model, with no training and one model to serve.

Paused · click to play
Click the figure to pause, or a chapter to jump. Chapter 1 shows real samples (MATH500, problem 204); token probabilities in chapter 3 are the worked example of Section 3.2; curves in chapter 4 are measured (Minerva).

Abstract

As test-time scaling (TTS) emerges as a new scaling axis, a model's sample diversity (pass@k at large k) has become as important as its single-sample accuracy (pass@1) in the many applications that combine a verifier with multiple reasoning traces. Reinforcement learning with verifiable rewards (RLVR), however, improves pass@1 while narrowing this diversity (Yue et al., 2025), which makes test-time scaling less sample-efficient. We apply WiSE-FT-style weight interpolation, W(α) = Wbase + α(WRL − Wbase), as a single training-free knob that trades pass@1 for pass@k. Under the experimental setup of Yue et al. (Qwen2.5-7B and SimpleRL-Zoo-7B on six math benchmarks), α = 0.25 recovers the base model's large-k coverage (pass@128 of 70.4 vs. 70.3 for the base model and 63.8 for RL, averaged over the two non-saturated benchmarks) while keeping 64–80% of the RL pass@1 gain, and it reaches the RL model's pass@128 with 2.3–7.1× fewer samples. α = 0.5 strikes a balance (pass@1 −1.2 and large-k pass@k +1.6 points relative to RL), and α = 0.75 is nearly a free lunch (RL-level pass@1 with OlympiadBench pass@128 +2.5). The coverage reached at α = 0.25 exceeds what the RL model reaches at any sampling temperature reported for this pair.

Results

Interactive versions of the paper's main results. Hover over any line, point, or bar for exact values.

What collapse looks like

Real final answers from 128 samples per model, on hard problems where the RLVR policy never answers correctly. Each color is one distinct answer; green is correct.

Problem
Correct answer:

pass@k along the path from base to RLVR

RLVR wins at k = 1 and loses at large k. Interpolated models sit in between, and α = 0.25 tracks the upper envelope at large k.

Benchmark
View

Minerva and OlympiadBench are the main benchmarks; on GSM8K and MATH500 every model is above 94% at k = 128. The dotted vertical line marks k = 32, where estimates switch to a second sampling stage (paper, Appendix A).

The trade-off as a knob

Each model is a point: pass@1 against pass@128. Press play to turn α from the base model to the RLVR policy.

Benchmark
Interpolation path Mixture of base and RLVR samples RLVR at higher T (Yue et al., 2025)

The dashed curve mixes samples of the two endpoints in every proportion at 128 samples in total. The dotted line is the RLVR policy's pass@128 at an entropy-matched higher temperature, read from Yue et al. (2025), Fig. 18.

Samples needed to match RLVR

With a verifier, coverage costs samples. Smallest number of samples per problem at which each model reaches the RLVR policy's pass@k.

Target

Read the ratios with care: the RLVR policy's pass@k curve is flat at large k, so small differences in pass@k become large differences in the required k. The base model also needs few samples, but on OlympiadBench, Minerva, and MATH500 its pass@1 is 8 to 13 points lower than that of α = 0.25. AMC23 and AIME24 are omitted because of their size (40 and 30 problems).

Why mixing is not the same as heating

To first order, the interpolated model's next-token distribution is a weighted product of the two models, πα ∝ π01−α π1α. Temperature rescales the RLVR policy alone, π11/T. Move the sliders on the worked example from the paper.

Base π0 RLVR π1 Interpolated πα RLVR at temperature T

A toy distribution over three tokens (paper, Section 3.2), not measured outputs. The geometric mixture is a first-order approximation; its error vanishes at both endpoints.

Which α should I use?

Recommendations from the 7B pair. On a new pair, treat them as a starting point to confirm.

0.25

Verifier, many samples

Base-level pass@128 with 64–80% of the RLVR pass@1 gain; RLVR's pass@128 with 2.3–7.1× fewer samples.

0.5

Balance

Pass@1 −1.2 and large-k pass@k +1.6 points relative to RLVR.

0.75

Small gain at no cost

Pass@1 −0.2, pass@k +1.0; OlympiadBench pass@128 +2.5.

1 or 0.75

No verifier

Under majority voting, interpolation does not improve on the RLVR policy.

Show full results table

pass@1 / pass@k at the largest k (%), math-verify grading. The largest k is 128, except 1024 for AMC23 and AIME24.

Try it on your own model pair

Any RLVR model and the base model it was trained from (same architecture and tokenizer). No GPU needed to build the mix.

zsh — rlvr-interp-passk
# CPU is fine: tensors are mixed one at a time in fp32 and saved in fp16
$ git clone https://github.com/Kim-Ai-gpu/rlvr-interp-passk && cd rlvr-interp-passk
$ pip install torch safetensors huggingface_hub

$ python -m interp_passk.interpolate \
    --base Qwen/Qwen2.5-7B --rl hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo \
    --alpha 0.25 0.75 --out models          # -> models/a25, models/a75

$ vllm serve models/a25 --dtype float16      # serve like any other checkpoint

What it does not do

Stated up front in the paper.

Citation

@article{202611.2034,
	doi = {10.20944/preprints202611.2034.v1},
	url = {https://doi.org/10.20944/preprints202611.2034.v1},
	year = 2026,
	month = {November},
	publisher = {Preprints},
	author = {Youngseong Kim},
	title = {Mix Your Base and RLVR Weights: A Training-Free Knob for Diversity},
	journal = {Preprints}
}