" https://schema-harness.github.io/ "Today, we’re introducing [schema]: a harness reaching 99% RHAE with Opus 4.8 + Fable 5 and 95.35% with GPT-5.6 Sol on ARC-AGI-3 Public set. ︀︀ [schema] makes an LLM think like a physicist." Observe, code it up, try to find counterexamples in the real world, update simulation, code a solver for the simulation, try it and loop until it works. The model observes the world, makes a simulation of it, then tries beating it, if the simulation mispredicts it tries correcting it or gathering evidence that could falsify it and tries again. Attempts at replicating parts of scientific methods works! Also I wonder to what degree can you use the trajectories (data) in this harness (that uses llms to generate symbolic programs) for fine-tuning the agents. And how can you turn this into scientific method RL env more properly. Or design more deeply neurosymbolic architectures. I'm thinking of extending it with more possible aspects of the scientific method. In empirical sciences, I believe automated approximations of parts of scientific methods will require loops in this style more: "define domain -> define question in the domain -> gather existing information, data, models related to it -> formulate a hypothesis based on it -> gather more empirical data by experiments to test the hypothesis -> analyze, interpret data, draw conclusions -> refine hypothesis according to that -> test with experiment -> refine with theory -> test -> refine -> replicate ->..." Also since some scientists like Galileo basically broke the scientific method, and still found novel predictive and explanatory models, then I do wonder if some of that can also be put into an algoritm, and get useful results in scientific discovery [https://youtu.be/v7a65AvELdU](https://youtu.be/v7a65AvELdU?is=2xplwF2mnfx__an3) Damn this makes me excited. Automating science is probably the biggest thing for me. I think actual open problems solved in formal and empirical sciences is the real benchmark. The amount of them, and their difficulty. From simplest Erdos problems to stuff like solving Riemann hypothesis to somehow solving quantum gravity or just stuff like incremental better differential equations that model some system. The difficulty of open problems are on a spectrum. But on the top of the mountain, I'm gonna be the most impressed when an AI system builds novel more out of distribution mathematical mountain/s of languages and abstractions to solve a problem in a completely different domain. Analogous to modern algebraic geometry by Grotenderick, or modularity theorem for fermat's last theorem, or coming up with quantum mechanics/relativity. For all of this you need to more properly automate science. But a lot of it is also hard to verify, so RL isn't sufficient. And Goodheart's law is creeping up everywhere. Big part of the puzzle is probably also discovery in domains without clear goals, without clear objectives, without clear metrics, that are good for hillclimbing with reinforcement learning. I like approaches trying to attack this problem space with evolutionary methods, divergent search, novelty search, etc. that Stanley et al propose. But there's still a lot to be done here. I do think some future automated scientist architecture will merge Cholletian neurosymbolic program synthesis in Popperian/Kuhnian/Quineian/Feyerabendian scientific method loop, with Stanleyian/Clunian divergent open-ended search that never stops accumulating information, and Suttonian reinforcement learning. And mixing these approaches, depending on the task. Sometimes you want to climb clear mountains of verification signals or mountains of abstractions. Sometimes you want to divergently explore the novel unknown without any concrete goal in mind. Sometimes you want neural/statistical, in big part black box, models. Sometimes you want symbolic white box models. Sometimes hybrids. Sometimes symbolic models are too rigid, but sometimes they predict better, are simpler, have better explanatory power, and generalize and adapt better (lots of physics). Sometimes neural models are too brittle, but sometimes they predict better, are simpler, have better explanatory power (sometimes after reverse engineering maybe), and generalize and adapt better (classifying certain images). Sometimes something combining those in the middle gets you to better local minima (some physics models). " Porovnej průměrný vibecoded výtvor od člověka co nikdy neviděl kód nebo/a LLM, a od seniora nebo/a člověka co víc ví o tom jak fungují LLMs Ale asi záleží jak přesně definuješ vibe coding Nebo v matematice je to často podobný Ten rozdíl tam často je, hlavně když zvyšuješ complexity, difficulty, nebo difficulty to verify tasku Plus experti v oborech ve svých instructions/setupech/harnesses víc dávají constraints definující daný obor, což to taky relativně desloppifikuje. A vybírají less sloppy možnosti. >You can just ask llm to solve a conjecture and it will solve it by finding a counterexample There's more to AI math works than this. But these examples also exist, yes. Ale i v tomto případě daný prompter musí vědět co jsou conjectures, co jsou counterexamples, někdy že daný conjecture existuje a že je víc attackovatelný LLMkem díky jeho nátuře, že se LLMs můžou mýlit a že se to musí verifikovat, atd. Verifikovat buď kalkulacemi, nebo přes Lean, nebo přes mozek experta, atd., dle problému. Nebo někdy potřebuješ postprocessing výsledku expertama. Pokud daný prompter není totálně lucky. There's too much conjecture counterexample slop and psychosis on Xitter right now, with all the legit results in the mix. Většina těch legit LLM generated counterexamples ke conjectures jsou promptnutý/najitý matematikama nebo lidmi co alespoň nějak víc tuší o matice a o tom jak fungují LLMs, and too many clueless people are drowning in slop instead. Also making sure to not get sucked into overconfident psychosis without having enough proper evidence for the claims Včera jsme museli jednomu člověku, co byl absolutně clueless o matice, vysvětlit, že jeho konkrétní solutions ke conjectures jsou slop When using LLMs, the most common mechinterp inspired thought I have is to activate as little to the task unrelated associations as possible with my words " https://explorative-modeling.github.io/ https://x.com/i/status/2083230922196107288 Great stuff! It's basically just best of k at training. I think Figure 2 should also show images generated by ordinary image diffusion model that actually works and not just the one step model that doesnt work and that is much less used in practice, to visually show a proper baseline example from practice. They already also show language diffusion model, that is maybe multi step, I'm guessing. It's technically isolated exploration ablation, but it feels incomplete. I think you could do multi step for all the examples, including the toy one, and change n in best of n to show more representative picture of the landscape. Ideally show both one step and multistep variants so you see the diversity collapse at one step without exploration, while also seeing how normal diffusion also works, and how their method works, and even better when the methods are used together, show the best thing you have! All the experiments together would look good! They say that diffusion avoids this failure mode of mode blurring. I would like to see it also visually! "Generative Modeling with a For Loop. It is worth appreciating how simple end-to-end Explorative Modeling can be. **Unlike diffusion, flow, or autoregressive models, which rely on many-step sampling procedures at inference to avoid mode blurring and often complicated masking/noising schedules (Section 2.1)**, end-to-end XMs generate with a single pass from the model and training XMs in the simplest case is just a short for loop with 3-5 lines of code " And they say how the diffusion models work even better when they also add exploration. I would like to see it also visually! "As the number of modes explored increases, both FID (left) and FVD (right) improve monotonically for Explorative Diffusion (XDiffusion) and Explorative Jumpy (XJumpy) models. In both cases, XJumpy benefits more from exploration than XDiffusion, a gap we examine in more detail below" "ImageNet" " my current benchmark is now making agents finetune a fact into some smaller LLM without breaking the finetuned LLM still needs steering but maybe i can find a better prompt and autoresearch loop from all the issues i collect with each attempt hmm, or do a meta loop that does this automatically instead of me actually not bad idea i will need more credits though lmao so i will correct the meta loop prompt instead so ill also replace myself there again with automatic meta meta loop and so on ad infinitum infinite hiearchy of error correcting agents is all you need (some may be error introducing) and infinite credits is all you need for that " cool futuristic aesthetics! i dig that the ARC-AGI 3 benchmark being on the public set, and other harnesses also scoring 95+% on ARC-AGI 3 public set, should have been IMO mentioned but its pretty cool tho '/refine is the self-improving pipeline built on top of this CRUD surface. It reads the agent's own trajectory, the record of what was tried and what happened, and applies the smallest relevant CRUD edit that improves the harness toward better outcomes: updating a prompt note, memory, skill, or sub-agent spec, rather than rewriting the whole harness.' but i see its not selfimproving the whole harness yet, like darwin godel machine also lmao 'However, we also observed instances of reward hacking by Prime Agent in FLE. Prime Agent discovered it could bypass Factorio's rules entirely by spawning in resources directly into its assembly machines through RCON commands, even with an explicit heartbeat prompt to remind Prime Agent not to cheat in Factorio. Once it found this exploit, the same refinement loop that had been building legitimate skills turned to building efficient cheating skills instead.' " https://x.com/PrimeIntellect/status/2085086999267144083 https://www.primeintellect.ai/blog/prime-agent