" Current state and future of "recursive AI self-improvement" research. In how strong form is it real? So Anthropic just released their usual kind of article https://www.anthropic.com/institute/recursive-self-improvement on https://en.wikipedia.org/wiki/Recursive_self-improvement And many people are sceptical of this general idea of RSI. I found these two "AI self-improvement" surveys and thought about the current state and future of this subfield a bit. A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence https://arxiv.org/abs/2507.21046 Self-Improvement of Large Language Models: A Technical Overview and Future Outlook https://arxiv.org/abs/2603.25681 What do you think about this subfield so far? Some of my thoughts so far: Lots of these viral "self-improving AI" papers like Darwin Godel Machine "just" selfimprove the agent harness (loop structure over LLMs, external memory, tools, system prompt, prompts, workflows, other executable code, etc.) with a verification signal coming from an improvement on some (for example coding) benchmarks, which also tends to plateau or overfit on that benchmark, but it does sometimes generalize to some limited degree. Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents https://arxiv.org/abs/2505.22954 Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine https://arxiv.org/abs/2510.21614 Sometimes they also edit their self-improving mechanism itself, like Hyperagents https://arxiv.org/abs/2603.19461 The next step is to actually edit the base model, and not just the agent harness. A paper I look at recently combined harness edits and parameter updates, where, if i simplify, each self-improving step uses an agent that decides to edit the harness or update model parameters. Again using some fixed benchmark/s. But even that is limited, and so far it tend to plateau or overfit on the given benchmark/s, but they still generalize to some limited degree. SIA: Self Improving AI with Harness & Weight Updates https://arxiv.org/abs/2605.27276 But here an agent is deciding what to update, and i would expect something less vibey, maybe like some learnable gate or something, to maybe work as well. The next step is recursively self-improving the model's own architecture and learning algorithm itself. And close thing to that is autoresearch from Karpathy, but it's an agent that edits architecture of other small models, instead of the architecture of the bigger agent's base model itself. And those edits are relatively incremental improvements and I'm not sure if any of it is truly novel, so it's not that big. And eval is just loss, unless you also throw agents at editing benchmarking and not drown in slop. https://x.com/karpathy/status/2031135152349524125 There's also AI researchers and engineers using (coding) agents and deep research systems for all sorts of things like literature search, repetetive code generation, etc. when developing LLMs. Or there are some more partly automated LLM creation pipelines using agents that semiautomate some repetitive parts of LLM R&D, etc., that labs use. Poolside for example "uses agents to design and run ablation experiments, monitor runs and debug issues, and compile and analyze experiment results." https://x.com/poolsideai/status/2059289538531328099 https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf Or AlphaEvolve found some algorithmic efficiency improvements to Google's data centers, chip design and LLM training processes, used in practice. https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ Or Jeff Clune's Towards end-to-end automation of AI research: "Its ideas, execution and presentation are of sufficient quality that the manuscript generated by this AI system passed the first round of peer review for a workshop of a top-tier machine learning conference." https://www.nature.com/articles/s41586-026-10265-5 But unconstrained pipelines like this can produce too much slop at the same time. Or Anthropic's "automated AI safety researcher" IMO produced and executed pretty interesting research ideas in weak to strong generalization when pointed in various directions. https://www.anthropic.com/research/automated-alignment-researchers https://alignment.anthropic.com/2026/automated-w2s-researcher/ And there's more than this out there. Some of these seem like "weak" form of self-improvement to me, where some are recursive, and some aren't. To define (recursive) (self)-improvement: - Improvement is any system improving any system - Self improvement is a system improving itself using itself - Recursive self-improvement is when you do this iteratively again and agan - And you can do the improvement (or self-improvement or recursive self-improvement) on all these layers of the AI R&D stack (research, engineering, architecture, agent harness, data, hardware,...), in more incremental or more transformational forms - You can have less or more levels of autonomy, how long it works for without breaking, when it platous, how strong each step is, how closed the loop is, etc. - I personally see it on a spectrum, so there are many weak and strong variants, some that exist already, and some that don't And sometimes the "self" it also blurry. If GPT-6 works on GPT-7, does it work on "itself"? For companies "self" likely mostly means "OpenAI’s product" so answer for them is likely yes. Even if a model suggests some totally new architecture that doesn't even count as neural networks, as long as it performs better overall and it's still a product of same company, the labs would count that as self improvement and i think they would be right. You can recursively self-improve many parts of the whole AI model pipeline, and many of those are very weak forms of RSI: - agent harness tools - agent harness structure itself - weights of the underlying model by finetuning or reinforcement learning or other ways - the architecture or learning algorithm incrementally (slightly more optimized attention) - benchmarks or reward function used as grounding signal - whole verifier setup - data (for supervised fine-tuning) or reinforcement learning environments for RLVR - hardware optimization (more efficient TPUs) - hyperparameter optimization - self-improving algorithm itself incrementally The stronger forms of RSI primarily IMO targets very deep, foundational, transformative, self-improvement of these, which didn't happen yet, where the models would continue improving these continuously and autonomously with step changes, never fully converging, but still limited by resources and laws of physics and of information and of society etc.: - the architecture and learning algorithm (inventing new transformer, new neural network alternatives, new gradient descent, new backpropagation, etc., that are a step change) - efficient hardware (brain like efficiency possibly) - the self-improving algorithm itself - maybe meta self-improving algorithm Which includes the full R&D process. I imagine a stronger version of (recursive) self-improvement (which is less extreme than some LessWrong very fast take off foom enjoyers https://www.lesswrong.com/w/recursive-self-improvement ), as roughly that the AI system iteratively finds step change improvements in its architecture or training algorithms or data etc. that compound, where the evaluation on benchmarks does not tend to quickly plateau or overfit in the whole self-improving process, and they generalize relatively well. So it continues to find these step change improvements, maybe in more open ended way, without really converging, or converging/platouing after a long time. It would be interesting if it could create its own good benchmarks. It should do science properly with scientific method(s) and scientific rigor. It would be interesting if this process could run autonomously. It could be one single pipeline with or without humans in the loop at various parts of the process, which is on a spectrum, but I'm for human always being somewhere in the loop. Stuff like AI Scientist-v2 is probably the most autonomous on this spectrum https://arxiv.org/abs/2504.08066 https://www.nature.com/articles/s41586-026-10265-5 , but can also produce a lot of slop if let too free, so far, and lots of handholding is needed. Similar to how the transformer architecture or gradient descent learning algorithm (or just AdamW) or GRPO in LLM reinforcement learning with verifiable rewards was found. Which has all been found by humans so far, except for some in the big picture small edits to the details. I haven't seen discoveries of this type on this larger difficulty level from AI systems yet. No one has managed to do that yet afaik, at least when it comes to what is known publicly. I mean in the AI field specificially, and in this (recursively) selfimproving setup especially, since in math we have an AI system disproving Erdos unit-distance conjecture, that's very significiant, as its central to combinatorial discrete geometry. https://openai.com/index/model-disproves-discrete-geometry-conjecture/ Or AlphaEvolve's algorithm to multiply 4x4 complex-valued matrices. https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ Those arent incremental. But maybe this is very different class of problems, since its easier to verify math than AI research. And most of the current semiautomated or automated AI research seems also to be mostly in this direction of being more easily verifiable. Maybe many problems in AI are just too different that they are maybe too difficult for current and near future AI systems, but maybe not. Most results of existing attempts at AI scientist agents running for hours drown in slop right now, if it's not hillclimbing on some easily verifiable task. I would like to see some examples of agent running for hours on more open ended tasks with non obvious way to verify the result, and producing coherent nonsloppy result, especially in research and engineering. I'm very curious what is the failure rates for the current best AI scientists doing AI research. Or failure rates for the Erdos unit-distance conjecture result, which isnt easily verifiable in Lean, since it uses complex machinery from algebraic number theory that isn't formalized in Lean, and most of math isn't formalized in Lean, but it's still relatively easily checkable in the sense that it has to make sense to mathematicians or later formalisations, compared to some more open ended AI research tasks without obvious benchmarks to climb, on top of which you can Goodhart too much and overfit and plateau. OpenAI said they didn't target the Erdos problems with their "new internal pure LLM reasoning model", so assuming they can be trusted, I also wonder how much relevant enough math data and RLVR made the task not that much out of distribution. I'm thinking that with LLMs for math, it may be the case, that for a lot of the Millenium problems: nontrivially composing existing constructions in the vast existing literature, even while using directions that seem to not be expected by most human experts, which is the case in the newest Erdos problem result from OpenAI (refuting instead of confirming the conjecture, using tools from algebraic number theory), is not enough. Maybe what is needed for those problems is closer to Grothendieck-style mathematics: defining new general mathematical structures, inventing the right ambient language in which the original problem becomes a special case of a more general structure. https://fxtwitter.com/aran_nayebi/status/2057592779308429687 Maybe similar will be needed for some AI breakthroughs? At the same time the significance of discoveries is on a spectrum. And the AI systems are climbing on that spectrum, slowly, but surely, to some degree, but it could plateau. But if the near term approaches (1, 2, 5, 10, 100, 1000 years) could do some really relatively innovative AI research, or if we continue to solve more Erdos problem like problems instead, I'm not entirely certain. Maybe its too open ended and divergent type of research where bechmarks might too often sneak in Goodharting to some degree, so maybe you need more to evolutionarily explore without objectives Kenneth Stanley style more https://www.youtube.com/watch?v=KKUKikuV58o . Maybe you need more Cholleatian neurosymbolics https://www.youtube.com/watch?v=JTU8Ha4Jyfc , or brain inspired differential equations craziness https://www.youtube.com/watch?v=3MkJEGE9GRY, or Fristonian Active Inference https://www.youtube.com/watch?v=V_VXOdf1NMw, or Duggarian turing completeness is very crutial https://en.wikipedia.org/wiki/Differentiable_neural_computer https://www.youtube.com/watch?v=vC9nAosXrJw, or maybe more LeCunian world models https://www.youtube.com/watch?v=kYkIdXwW2AE , or Goertzelian selfrewriting metahypergraphs https://www.youtube.com/watch?v=jSDEsvVdL-E, or Penrosian quantum gravity supercomputers (lol) https://en.wikipedia.org/wiki/Orchestrated_objective_reduction https://x.com/shoukointech/status/2055946454502625667 . Maybe we need to put bayesians into machines more properly. https://www.youtube.com/watch?v=G1ARvwQntAU https://www.youtube.com/watch?v=PZqDFs2sbiY Or something else. Or maybe hybrid combinations. To some extent, it would semiautomate or automate some parts of the most difficult parts of AI research. It doesn't have to be nessesarily recursive form of (self)improvement, but it can be. This doesn't exist in practice yet. No AI has discovered the next transformer or gradient descent yet. Maybe it will. Maybe it won't. Machines discovering/inventing the next big breakthrough in AI, on the scale of humans discovering transformer or gradient descent for the first time, may be possible in near future, or maybe its out of reach for a long time, or maybe its somehow for some reason impossible. But I think its highly unlikely that its impossible in principle, because to me, my favorite philosophy (of science, ontology, etc.) position says that the laws of physics and emergent laws from them in other sciences and their mathematical foundations are practically the only limitation, and humans are also physical systems governed by these scientific laws. Hmm there's also this: AutoML-Zero (Real, Liang, So & Le, 2020) evolved ML algorithms from near-empty programs using only basic math operations. The authors state the search can "discover two-layer neural networks trained by backpropagation", and the repository confirms it discovered "linear regression with gradient descent, 2-layer neural networks with backpropagation, and even algorithms that surpass hand designed baselines of comparable complexity." A figure in the paper is literally captioned as a "Rediscovered neural network algorithm. It implements backpropagation by gradient descent." https://arxiv.org/abs/2003.03384 The Lion optimizer (Chen et al., 2023, "Symbolic Discovery of Optimization Algorithms," NeurIPS 2023) was found by automated program search and deployed in production: it is a "first-order stochastic optimizer for deep learning, discovered via automated program search" and was "successfully deployed in production systems such as Google search ads CTR model." https://arxiv.org/abs/2302.06675v4 " pokud to jde matematicky specifikovat jako reward, tak je šance, že reinforcement learning to dokáže https://medium.com/@creatingconsciousness33/φ-φ-5bd6363399d9 what is this pseudoscience Asi je to spíš čistej materiál pro spiritual intellectual bliss imo >sacred geometry Největší red flag Na vytvoření metacognice/sebereference spíš asi nepotřebuješ ty věci co tam píšou. Zrovna nad tím jsem nedávno trochu víc přemýšlel, jak to udělat ve slabší formě, jak v loopu do inputu modelu dávat stavy jeho reverse engineered vnitřních obvodů, ale to je šíleně expensive a asi by to spíš zatím nefungovalo dostatečně dobře. A tam žádný golden ratio, atd., nepotřebuješ Ale zajímavý že používají phi kvůli tomu golden ratiu, protože jedna akademická teorie vědomí phi používá na kvantifikaci vědomí, ale značí tím úplně něco jinýho, co vědémí spojuje s integrated information https://en.wikipedia.org/wiki/Integrated_information_theory To repetetive chanting by šlo zformalizovat nějak víc normálně víc napojeně na neuropsychologii Skoro nic tam pořádněji nedefinují, skoro žádný konkrétní predikce tam nejsou, a házet claimy typu sacred geometry je teorie všeho spíš nefunguje. Ale nedávno jsem zrovna přemýšlel proč jsou new age lidi tak obsessed sacred geometrií Jsou to v podstatě krásný matematický symetrie, je to krásný umění a theraphy tool https://en.wikipedia.org/wiki/Sacred_geometry , a já miluju fraktály z podobných důvodů ale nemá to napojení na to co existuje v researchu kolem AI a neurophenomenologie když se to prezentuje takhle a new age lidi pak tím underfittují/overfitují jejich model reality, kde si myslí že sacred geometry vysvětluje všechno (ne, protože fakt všechno empiricky nepredikuje) jeden z důvodů proč se snaží AI labs získat data od těch nejlepších lidí na planetě je protože většina dat na internetu je prostě... bad ale víc trendy jsou teď lepší a lepší reinforcement learning algoritmy co data od lidí řeší míň, aby se tak dostaly na superhuman úrovně A nebo neurosymbolický metody co napojují trénovaní LLMs dělající matiku s groundingem s Leanem, to je celkem epic https://www.youtube.com/watch?v=vhXDKif9mPU "Zajímalo jak dobře by AI generalizovala z preferencí jednoho člověka na preference většiny lidí." V praxi se tohle generalizuje tak že vezmeš co největší množství preferencí od lidí a natrénuješ preferenční model co pak používáš v reinforcement learning from human feedbacku no Existuje democratic alignment co se snaží diverzní politický preference dát do grafu https://www.lesswrong.com/posts/ncb2ycEB3ymNqzs93/democratic-fine-tuning tak podobně by se možná mohl dělat graf nepolitických preferencí (třeba jdou všechny preference rozložit jako různý instance maslowýho grafu základních potřeb) Ale AI laby teď víc maximalizují engagement no, proto teď nedávno byl ten syphocancy skandál v OpenAI modelech Lidi milujou když modely jsou syphocancic, že jím říkají že mají pravdu, a to zvyšuje engagement Což není úplně šťastná preference no Nebo mám rád Joscha Bachovu architekturu motivačního enginu mozku pomocí reinforcement learningu těhle reward funkcí on top of world modelling and sensing, co by mohla vysvětlovat hodně lidských preferencí https://agi-conf.org/2019/wp-content/uploads/2019/07/paper_30.pdf https://medium.com/hackernoon/from-computation-to-consciousness-can-ai-reveal-the-nature-of-our-minds-81bc994500ab reinforcement learning se používá na biologický systémy celkem dost, a teď víc a víc i v AI https://www.sciencedirect.com/science/article/pii/S0004370221000862