" I find it interesting that here he compares humans to pure-ish reinforcement learning but with a lot of human reward function engineering, and to TAS with some brute force and without reinforcement learning or other machine learning, and the TAS with some brute force and without reinforcement learning or other machine learning still wins https://www.youtube.com/watch?v=zFLQU70QstY So the title is a bit clickbaitish, since TAS with some brute force and without reinforcement learning or other machine learning is the actual winner still. Same still holds for Valve games afaik. The next logical step is to try optimization in the form of specialized reinforcement learning models and coding LLM agents in a loop over symbolic and analytical TAS actions/tools. That trackmania video is actually starting this approach in hybrid way i think, but apparently the reinforcement learning system was still unable to beat the community TAS on its own, even after being augmented with an auto-drift script and segmented training, so primarily humans crafted TAS that in part had brute-force exploration ended up setting the fastest times. Also its interesting that the brute-force petrubation script also accidentally found a previously unnoticed physical gap/hole between two misaligned road blocks, so it wasnt machine learning that found something that i would call creative. Generally most of current top TASes involve no machine learning, and mostly analytically perfect TAS actions (like strafing in Valve games https://youtu.be/hx7kvTZLHYI , drifting in trackmania) and brute force after you narrow the space sufficiently enough (like pixel perfect brute force autooptimize one right turn in trackmania). Many tools in TASer's toolbox arent really brute force, as pure dumb brute force (trying fully random solutions one by one) would sometimes take longer than the age of the universe, and some of TAS tools can potentially reduce the solution space with combinatorial explosion of possibilities a lot by smarter forms of search/optimization. Optimization by reinforcement learning is an example of that, relative to just dumb brute force. But humans crafting TASes with the help of brute force are still winning, that's interesting to me... Most machine learning approaches are just worse atm for some reason, but maybe people didnt try enough. In TASing you have to do the glitch hunting, path finding, creating TASing tools, finding and composing known strats, etc., which often requires creativity, and can potentially be semiautomated to some degree. One could generally say that various key presses, TAS actions with laws (analytical maximizing strafing math), brute force, neural network models like the RL models, other optimization methods (evolution), higher level search using LLM agents, etc., are components that can be used together, composed and recombined, by the cooperation of humans and machines, into potentially mathematically optimal perfect full-game TAS in the end. I was thinking of giving a simplified game with planted hypermovement glitches to this Schema agent or similar type of an agent, and/or additional TAS tools, and just let it go wild and see what can it find. https://fxtwitter.com/HavenFeng/status/2077770348876247502 , https://schema-harness.github.io/ It could try to code additional TAS machinery, potentially something like trying to write code to try to analytically solve strafing stuff... In a loop. Extending/editing the game's code to test some interventions maybe. But somehow minimizing the chances of it collapsing into slopfest by good grounding verification signals. Maybe one could try some autoresearch loop on coding agent thats coding a specialized RL model as well (actions can be keyboard presses or TAS tool actions), with some explicit verification signal as grounding (time on a map executed by code that coding agent shouldnt edit). Trackmania people have to do pretty complex reward function engineering to make the RL model actually find hypermovement glitches in trackmania more easily. https://www.youtube.com/watch?v=NUl6QikjR04 But there's also this old school case of RL finding hypermovement glitch on its own without the reward function being engineered in a way to incentivize finding it more easily. https://www.youtube.com/watch?v=Lu56xVlZ40M So there's hope that you can make a simplified game with planted hypermovement glitches and the RL agent finding them without the reward function being explictly engineered to find it more easily. One perspective to see speedrunning is as kind of applied simulated digital physics. Analytically solving strafing or drifting math is kind of like finding a physical law in this simulated digital physics universe. And in physics you also often unlock many new problems after you solve one. And mathematically optimal full-game TAS is the endgame. :FeelsWOWMan: Giving all these TAS tools used here, and human found strategies used here, to a coding agent, or/and engineering it into a specialized RL agent's action space, and achieving superior performance, could be amazing. And at the same time, once some machine learning system with almost no engineered in, specialized, human found strategies, in its tools or reward function or architecture, beats this, then I'm gonna be very fascinated. https://www.youtube.com/watch?v=U92eaoVC7wY You use a coding agent with LLM as its core decider, not just LLM alone. You give the agent as much of an environment and context and goals as TASer has. Most common coding agents are Codex <https://github.com/openai/codex> , Claude Code and OpenCode <https://github.com/anomalyco/opencode> right now, and they already have preinstalled a lot of the functionality what I'm gonna mention now, where you add additional configuration: You have a coding agent, in an "observe, decide, act" loop, embedded in bash terminal with file system that can execute arbitrary code and install arbitrary programs, search the web, make plans in .MD files, sometimes additionally with the ability to do visual testing of games/apps using computer use. You give the agent access to game code (if available), map files, existing TAS software API and code, pointers to existing speedrunner and TASing knowledge bases with known glitches and routes and overall best practices, previous TAS run scripts, previous speeruns, etc. And let it generate more TAS scripts, or/and generate TAS software, or additional code for simulations (strafing math for example), or glitch hunting code to falsify glitch hypotheses, that it can run and verify by code or visually. You let it execute the code, run scripts, run games/apps, etc. to get feedback. Time on TAS going down is good feedback as a verification signal that it can ground itself with. But you have to figure out how to constrain it for the task as much as possible so that it won't start drowning in slopfest instead. Many such cases. Sometimes you can use other agent checking it, or have an agent orchestrator orchestrating subagents. Similar setups are used in for example software engineering or ML/AI engineering and research right now. "