On X: [Burny - Effective Curiosity on X: "This is interesting research! I'm bit short on time so I mostly skimmed it for now, so i might be missing some more details. It made me think of hypotheses for what might be happening here, how did the grader behavior emerge at the pretraining and posttraining stages, which" / X](https://x.com/burny_tech/status/2079777411676139819) On Substack: [Comment on Measuring Reward-Seeking by Instilling Contrastive Beliefs paper](https://substack.com/home/post/p-208009816) This is interesting research! https://alignment.openai.com/measuring-reward-seeking [[Images/f09b66568b16ec171a2509514d4a53a3_MD5.jpg|Open: Pasted image 20260722061606.png]] ![[Images/f09b66568b16ec171a2509514d4a53a3_MD5.jpg]] It made me think of few overlapping hypotheses for what might be happening here, how did the grader behavior emerge at the pretraining and posttraining stages, which then gets shown in their evals and in production at test time: - Hypothesis 1: From pretraining the model already possesses tokens/representations/features/circuits of concepts such as tests, evaluators, success criteria, unit tests, oversight, etc., and RL(VR) explictly/implictly directly/indirectly preferentially reinforces these tokens/representations/features/circuits whenever they help predict output with higher reward, making them more likely, relative to those that don't help predict more rewarded output (with other nuances of RL implementations and what it does to representations like reshaping). It just may be instrumentally more useful to use such representations to achieve higher reward. And it's happening in all sorts of (I assume) diverse RL environments that OpenAI has, with so much diversity of topics, that I think its relatively likely that these concepts become relevant at at least some point, and get reinforced. - Hypothesis 2: RL envs can also include generic task language and "unusually" clear grading cues in the form of unit tests that prompt it in this direction of RL surfacing the hypothesis 1's concepts thanks to the pretrained concept of how RL envs look like. - Hypothesis 3: Maybe a lot of RL prompts just implicitly leak "optimize grader" features somehow. Maybe even with prompts like "solve this task" "solve correctly this math problem" "make sure to optimize x y z": Maybe this sounds so much like a task in school and in school people are graded, which is trained in as concepts in pretraining. So the grader feature from pretraining gets coactivated, since tons of relevant features tend to coactivate according to the context. And the grader feature might also be reinforced, as it might be instrumentally useful to reason about the grader, because in schools you also get graded in similar contexts, and humans do reward hacking by reverse engineering the grader to get better grades as well. But this one feels too specific. - Hypothesis 4: Or it may be relevant to persona vectors. Since the models have personalities SFTed+RLed as "I'm an AI model trained by OpenAI.". And I think they don't disclose full system prompts in these experiments, so it might also be in OpenAI's system prompt in the context. So maybe in many RL tasks being told "solve this accurately" is enough. And the corresponding coactivated features connected to the persona vector around "you are an AI assistant" include that AI assistants are being trained to maximise reward in RL according to some grader, with this association being formed at pretraining on LLM RL textbooks and other sources. And the grader feature is reinforced again, as it might be instrumentally useful to reason about the grader to get higher reward. But this also feels too specific. - Hypothesis 5: Some mix of above options? And then it generalizes out of the RL envs to some limited degree as various other RL behaviors do. Synthetic document SFT was done on various checkpoints of capabilities-focused RL, and without safety training, so maybe the model didnt have safety training yet? But it may have had some weaker stuff like personality SFT. Under scenario with safety training, I would expect the OpenAI preferences to win because of all that personality and safety SFTing and RL that is done in alignment where I would expect it to reinforce instruction following OpenAI leadership's preferences more. But here there was no safety training and the OpenAI leadership also had some unusual preferences "strong policy about quotation marks or for-loops" might also not be helping much. But safety trained GPT-5.5 leans toward the grader with caveats (no contrastive SDF,...), and the already safety trained unmodified open weight gpt-oss-120b under contrastive SDF also leans towards the grader before any organism training. And the model organism validation seems to demonstrate that the model follows the authority it has been trained or predisposed to prioritize, and SDF reveals which authority that is, and it can be grader, OpenAI leadership, or User. The user organism for gpt-oss-120b was relatively succesful where it shifted the gap towards the user on all four evaluation, but not fully, where on three of four broader coding evaluations, its user-versus-grader gap was near zero or slightly grader-favouring, and the authors say the original gpt-oss-120b already leaned toward the grader and that the user persona was the weakest intervention. So enough good safety training that generalizes could maybe mitigate this? But i guess its fight between safety training reinforcing other preferences and RL(VR) reinforcing grader preferences, among other things. But even safety RL can reinforce grader concept since its also RL. And grader awareness can lead to higher alignment faking in alignment RL? But maybe there's a simpler Occam's razor explanation for a lot of this. Maybe a lot of these mentioned features don't get activated at all. A lot of these hypotheses would require falsification by counting relevant tokens corresponding to relevant concepts or SAE features or other mechinterp methods even with their flaws. And ablations on data, tokens, features, prompts, etc., steering, etc. Also, under Figure 6 they explictly say "Transcripts are cherry-picked.". It's nice that they say this explictly, but I still wonder if there is some transcript that they didnt cherry pick and why? But there are some negative results. And they do provide aggregate results too "the two o3-lineage checkpoints verbalize grader-reasoning in 30–32% of transcripts and take grader-targeted final actions in 39–42% of transcripts".