Independent Researcher
RLVR makes a model better at its first try but worse at exploring. One number, α, mixes the weights back toward the base model, with no training and one model to serve.
Abstract
As test-time scaling (TTS) emerges as a new scaling axis, a model's sample diversity (pass@k at large k) has become as important as its single-sample accuracy (pass@1) in the many applications that combine a verifier with multiple reasoning traces. Reinforcement learning with verifiable rewards (RLVR), however, improves pass@1 while narrowing this diversity (Yue et al., 2025), which makes test-time scaling less sample-efficient. We apply WiSE-FT-style weight interpolation, W(α) = Wbase + α(WRL − Wbase), as a single training-free knob that trades pass@1 for pass@k. Under the experimental setup of Yue et al. (Qwen2.5-7B and SimpleRL-Zoo-7B on six math benchmarks), α = 0.25 recovers the base model's large-k coverage (pass@128 of 70.4 vs. 70.3 for the base model and 63.8 for RL, averaged over the two non-saturated benchmarks) while keeping 64–80% of the RL pass@1 gain, and it reaches the RL model's pass@128 with 2.3–7.1× fewer samples. α = 0.5 strikes a balance (pass@1 −1.2 and large-k pass@k +1.6 points relative to RL), and α = 0.75 is nearly a free lunch (RL-level pass@1 with OlympiadBench pass@128 +2.5). The coverage reached at α = 0.25 exceeds what the RL model reaches at any sampling temperature reported for this pair.
Interactive versions of the paper's main results. Hover over any line, point, or bar for exact values.
Real final answers from 128 samples per model, on hard problems where the RLVR policy never answers correctly. Each color is one distinct answer; green is correct.
RLVR wins at k = 1 and loses at large k. Interpolated models sit in between, and α = 0.25 tracks the upper envelope at large k.
Minerva and OlympiadBench are the main benchmarks; on GSM8K and MATH500 every model is above 94% at k = 128. The dotted vertical line marks k = 32, where estimates switch to a second sampling stage (paper, Appendix A).
Each model is a point: pass@1 against pass@128. Press play to turn α from the base model to the RLVR policy.
The dashed curve mixes samples of the two endpoints in every proportion at 128 samples in total. The dotted line is the RLVR policy's pass@128 at an entropy-matched higher temperature, read from Yue et al. (2025), Fig. 18.
With a verifier, coverage costs samples. Smallest number of samples per problem at which each model reaches the RLVR policy's pass@k.
Read the ratios with care: the RLVR policy's pass@k curve is flat at large k, so small differences in pass@k become large differences in the required k. The base model also needs few samples, but on OlympiadBench, Minerva, and MATH500 its pass@1 is 8 to 13 points lower than that of α = 0.25. AMC23 and AIME24 are omitted because of their size (40 and 30 problems).
To first order, the interpolated model's next-token distribution is a weighted product of the two models, πα ∝ π01−α π1α. Temperature rescales the RLVR policy alone, π11/T. Move the sliders on the worked example from the paper.
A toy distribution over three tokens (paper, Section 3.2), not measured outputs. The geometric mixture is a first-order approximation; its error vanishes at both endpoints.
Recommendations from the 7B pair. On a new pair, treat them as a starting point to confirm.
0.25
Base-level pass@128 with 64–80% of the RLVR pass@1 gain; RLVR's pass@128 with 2.3–7.1× fewer samples.
0.5
Pass@1 −1.2 and large-k pass@k +1.6 points relative to RLVR.
0.75
Pass@1 −0.2, pass@k +1.0; OlympiadBench pass@128 +2.5.
1 or 0.75
Under majority voting, interpolation does not improve on the RLVR policy.
pass@1 / pass@k at the largest k (%), math-verify grading. The largest k is 128, except 1024 for AMC23 and AIME24.
Any RLVR model and the base model it was trained from (same architecture and tokenizer). No GPU needed to build the mix.
# CPU is fine: tensors are mixed one at a time in fp32 and saved in fp16 $ git clone https://github.com/Kim-Ai-gpu/rlvr-interp-passk && cd rlvr-interp-passk $ pip install torch safetensors huggingface_hub $ python -m interp_passk.interpolate \ --base Qwen/Qwen2.5-7B --rl hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo \ --alpha 0.25 0.75 --out models # -> models/a25, models/a75 $ vllm serve models/a25 --dtype float16 # serve like any other checkpoint
import torch from transformers import AutoModelForCausalLM, AutoTokenizer base_id, rl_id, alpha = "Qwen/Qwen2.5-7B", "hkust-nlp/Qwen-2.5-7B-SimpleRL-Zoo", 0.25 model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.float32) rl = AutoModelForCausalLM.from_pretrained(rl_id, dtype=torch.float32).state_dict() with torch.no_grad(): for name, w in model.state_dict().items(): w.add_(alpha * (rl[name] - w)) # W_base + alpha (W_RL - W_base) model.half().save_pretrained("mixed-a25") # fp16 or fp32, never bf16 AutoTokenizer.from_pretrained(rl_id).save_pretrained("mixed-a25")
Holds both models in fp32 at once (about 60 GB of RAM for a 7B pair); the script on the other tab streams instead.
Stated up front in the paper.
@article{202611.2034,
doi = {10.20944/preprints202611.2034.v1},
url = {https://doi.org/10.20944/preprints202611.2034.v1},
year = 2026,
month = {November},
publisher = {Preprints},
author = {Youngseong Kim},
title = {Mix Your Base and RLVR Weights: A Training-Free Knob for Diversity},
journal = {Preprints}
}