I tried to look into the giant debate around what all the different methods of RLVR actually does to the LLM's output distribution and internal representations. Papers are arguing whether "RL truly expands a model’s reasoning capabilities or merely amplifies high-reward outputs already latent in the base model’s distribution. And whether continually scaling up RL compute reliably leads to improved reasoning performance." <https://arxiv.org/abs/2505.24864> Answer to that question is different across different papers because the different papers use different KL-divergence formulations etc. in RLVR methods, different evaluation methodologies, different amount of compute. For example, this "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" paper <https://arxiv.org/abs/2504.13837> says: "In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric." "Surprisingly, we find that the current RLVR training setup does not elicit fundamentally new reasoning patterns." "Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model." "Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. But this "Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration" paper <https://arxiv.org/abs/2510.03865v1> tries to theoretically explain why, and fix it: "However, a fundamental limitation remains: as the sampling budget increases, the advantage of RLVR-trained models over their pretrained bases often diminishes or even vanishes, revealing a strong dependence on the base model's restricted search space." "We attribute this phenomenon to the widespread use of the reverse Kullback-Leibler (KL) divergence regularizer, whose mode-seeking behavior keeps the policy trapped inside the base model's support region and hampers wider exploration." "To address this issue, we propose RAPO (Rewards-Aware Policy Optimization), an algorithm to promote broader yet focused exploration. Our method (i) utilizes the forward KL penalty to replace the reverse KL penalty for out-of-distribution exploration, and (ii) reweights the reference policy to facilitate adaptive in-distribution exploration" "Notably, RAPO enables models to surpass the base model's performance ceiling and solves previously intractable problems, advancing the frontier of RLVR for challenging reasoning tasks" This "ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models" paper <https://arxiv.org/abs/2505.24864>: "We challenge prevailing assumptions by demonstrating: prolonged RL (ProRL) training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling" "We introduce ProRL, a novel training methodology that incorporates KL divergence control, reference policy resetting, and a diverse suite of tasks" "Our empirical analysis reveals that RL-trained models consistently outperform base models across a wide range of pass@k evaluations, including scenarios where base models fail entirely regardless of the number of attempts." "We further show that reasoning boundary improvements correlates strongly with task competence of base model and training duration, suggesting that RL can explore and populate new regions of solution space over time." "These findings offer new insights into the conditions under which RL meaningfully expands reasoning boundaries in language models and establish a foundation for future work on long-horizon RL for reasoning." If for example the last ProRL paper's claims that "prolonged RL (ProRL) training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling" "RL-trained models consistently outperform base models across a wide range of pass@k evaluations, including scenarios where base models fail entirely regardless of the number of attempts" is true and there isn't some issue in the methodology, especially evaluation, then it seems like properly setup RLVR can actually potentially do some expanding of model's reasoning capabilities. They seem to do 256 samples for both base and ProRL model, and say they go from 0% to 100% on Reasoning Gym tasks, which seems to underly the "base models fail entirely regardless of the number of attempts" claim, but maybe if they did more samples for the base model, it wouldnt be 0% anymore. And the RAPO paper has 2048 samples for both base and RAPO model evaluated on subset AIME problems that base models couldnt solve, that makes me sceptical of the generalizing claims. I wanna see much more samples than this. If GRPO LLM RLVR it surfaces and reinforces existing capabilities (by mostly sharpening the distribution) present in base model, or if it makes novel reasoning patterns beyond the base model emerge, is still an open problem in open science. I would argue that one did any good, scientifically rigorous, large scale experiments, and reverse engineering, with transparent data, with ablations, with controlling confounders, and everything, at sufficient scale, in open science, yet. But I think we definitely know that preLLM RL can definitely find "novel reasoning patterns" (or how to call it) without any supervised data like nonLLM neural nets playing games using RL only, with all the famous examples of reward hacking included. And since LLMs are currently solving open math problems in novel ways, by nontrivially finding using machinery from one abstract math field in another abstract math field like in the Unit distance conjecture disproof as I described in [[Explosion of AI generated mathematics with counterexamples to conjectures and the future of AI mathematics with Artificial Grothendieck Intelligence]], that makes me think that they already do some kind of novel composing of already learned strategies, but there may be more than that. But we have to be careful, since in the limit everyone is "just" composing already learned words from the dictionary, unless you're inventing new languages, lol.