" Could a major opportunity to improve representation in deep learning be hiding in plain sight? Check out our new position paper: Questioning Representational Optimism in Deep Learning I wonder if differently setup deel learning architecture and training algorithm and pipeline could get to similar beautiful representations https://fxtwitter.com/NickEMoran/status/1924888905523900892?t=AH_UBS0KbzFHD5amvp7JjQ&s=19 https://fxtwitter.com/kenneth0stanley/status/1924650134299939082?t=3WQ9qlaxJ_fuueRl57UE8A&s=19 And I also wonder if it's better to frame each type of representation as having different advantages and disadvantages. Both unified factored representations and entangled representations in superposition. ' - features trained on a single image suck, - using a method with implicit heavy regularization on the features makes them more smooth, Summarized that for you ' And this guy might have interesting point: ' I read the intro and not convinced about some of the fracturing arguments. Paragraphs seems to contradict, e.g. unified factored independent representations is just "if statement for every case". >change hair color might also cause the foliage in the background to change as well Isn't a fractured representation, its a unfiied represntation of coloring, just wrong and should have been fractured more. Creativity is a huge use case for fractured representations. You want to mix independent ideas. Steve jobs did LSD to boost his creativity. Repeating neuron representations is generally good for learning (e.g. dropout). Redundant circuits boost robustness. And redundant circuits early in training can specialist later on. And the biological argument that the brain is very fractured. Information is duplicated and stored through the entire brain. There are many redundant circuits that do the same thing. There is no unified representation. I have an intuition about this. In evolutionary processes, they're stochastic, so if you have really entangled representations, they're very delicate, and will be heavily disrupted by the introduction of any noise, so the representations simplify and map closer (or more directly) to the data. I guess. Sorry, I'm not explaining it super well but it makes sense in my head. But if my intuition is correct, any stochasticity will encourage this I forget which paper it was, but there was a paper recently that noted the performance of modern LLMs (per parameter used) was closely tied to the very entangling of representations that this paper argues against And they argued that we still have orders of magnitudes more performance per parameter if we can encourage the entalgement of concepts So...Depending on who you ask, either A) Stop using dropout B) Use a ton of dropout Yeah, superposition, sorry, that's what I meant ' "We found that when superposition is weak, meaning only the most frequent features are represented without interference, the scaling of loss with model size depends on the underlying feature frequency; if feature frequencies follow a power law, so does the loss. In contrast, under strong superposition, where all features are represented but overlap with each other, the loss becomes inversely proportional to the model dimension across a wide range of feature frequency distributions." Superposition Yields Robust Neural Scaling: https://arxiv.org/abs/2505.10465 ' But yeah, they both note that stochasticity in the training process seems to encourage simpler representations, or at least ones that are more robust to perturbation. Presumably, having a fairly linear relationship to the data helps with that ' On the other hand: https://fxtwitter.com/fabmilo/status/1924656105340469595?t=C5ot2mUyNmOkRrUaYzyS6w&s=19 Fabrizio Milo (@fabmilo) @kenneth0stanley @hardmaru The messy network reminds me of so many software architecture frameworks and stacks. Stratified evolution can’t be optimized globally it seems. https://fxtwitter.com/kenneth0stanley/status/1924656383259246972?t=7HwEpBdsIByzfYa2eX03xQ&s=19 Kenneth Stanley (@kenneth0stanley) @fabmilo @hardmaru Yes good observation! In fact, we note in the paper that this kind of fractured entangled representation is like poorly written code. It's a good metaphor. I would be curious how he sees this paper about superposition yielding robust neural scaling in relation to his work Maybe these software architectures reflect our cognition I have a feeling that there is some sweet spot that maximizes the advantages and minimizes disadvantages of both unified factored representations and entangled representations in superposition to get more robust generalizing circuits that could be studied using methods from mechanistic interpretability " pokud to jde matematicky specifikovat jako reward, tak je šance, že reinforcement learning to dokáže ten vědecký pokrok v AI díky tomu AI boomu je zatím mega sice strašně koncetrovaný na jeden typ AI systémů, ale i tak je tam pořád nějaká diverzita v pozadí co není moc vidět, co je ale díky tomu boomu taky podporovaná vznikají díky tomu nějaký pokusy o formalizace inteligence vznikají díky tomu pokusy aplikaci všech těhle technologií ve fyzice, biologii, zdravotnictví to je co mě interesuje nejvíc no děje se to taky, děje se toho tolik nebo to že dokážeme do latentního prostoru o trillionech dimenzí takhle relativně dobře zakódovat takový šílený množství vědomostí, a s tím pak dál fungujovat, je naprosto mindblowing They discuss how reinforcement learning helps in zooming in on the more likely correct solutions, and at the same time the more RL scales, the more novel patterns can emerge, like the new patterns that emerged in AlphaZero (superhuman chess strategies) https://youtu.be/64lXQP6cs5M?si=a7-Ly7xdd9MGyoXl Tady ještě podobný věci řeší v kontextu reinforcement learningu, že jednu věc čemu reinforcement learning pomáhá je zooming in na ty víc pravděpodobněji správnější možnosti, a zároveň čím víc se RL škáluje, tím novější vzory můžou vznikat, jako vznikly nový vzory v AlphaZero (superhuman chess strategies) 9:20 Different people do AI applications/engineering/research for combinations of different reasons. Some people do exploratory research out of curiosity with the need to understand intelligence itself and the structure of reality which I resonate with the most, some want to save the world from misaligned AS, or save the world with aligned ASI, then some people want to make trillions of dollars at all costs, some want power, or some create interesting things because they are interesting, or create helpful things because they help, cool things because they are cool, beautiful things (including artistic) because they are beautiful, some want their basic needs met using this technology, or some decentralized open source computing/training/inference AI initiatives are trying to break the oligopolistic dominance of big tech that is slowly and surely strengthening, etc. So many incentives! Extremely quality data and best reinforcement learning setups are currently the biggest moats. That's why Google started winning. They have the best history and access to both. And I also wonder if it's better to frame each type of representation as having different advantages and disadvantages. Both unified factored representations and entangled representations in superposition. Could a major opportunity to improve representation in deep learning be hiding in plain sight? Check out our new position paper: Questioning Representational Optimism in Deep Learning I wonder if differently setup deel learning architecture and training algorithm and pipeline could get to similar beautiful representations https://fxtwitter.com/NickEMoran/status/1924888905523900892?t=AH_UBS0KbzFHD5amvp7JjQ&s=19 https://fxtwitter.com/kenneth0stanley/status/1924650134299939082?t=3WQ9qlaxJ_fuueRl57UE8A&s=19 Superposition yielding robust neural scaling [[2505.10465] Superposition Yields Robust Neural Scaling](https://arxiv.org/abs/2505.10465) Maybe these software architectures reflect our cognition I have a feeling that there is some sweet spot that maximizes the advantages and minimizes disadvantages of both unified factored representations and entangled representations in superposition to get more robust generalizing circuits that could be studied using methods from mechanistic interpretability https://x.com/burny_tech/status/1903817268514971742 2024Q3: “Reasoning” will probably need non-neural search, like MuZero. ︀︀2024Q4: Oh… apparently you can just do thinking in the context window and it just *learns* to backtrack and so on? Huh. ︀︀2025Q1: Memory will probably need test-time backward-passes, like AlphaProof. 2025Q2: Test time adaptation goes mainstream? Or more neurosymbolic architectures? Neurally guided program synthesis? Combining with knowledge graphs? Generalizable world models? [https://www.youtube.com/watch?v=w9WE1aOPjHc](https://www.youtube.com/watch?v=w9WE1aOPjHc) [https://www.youtube.com/watch?v=mfbRHhOCgzs](https://www.youtube.com/watch?v=mfbRHhOCgzs) Davidad Bitter lessoned me [https://fxtwitter.com/davidad/status/1903834443225190721](https://fxtwitter.com/davidad/status/1903834443225190721 "https://fxtwitter.com/davidad/status/1903834443225190721") Will scaling inference time training be the next bitter lesson? https://arxiv.org/abs/2401.11504 Future is multiagentic reinforcement learning I wanna see more mechanistic interpretability for models doing math. will ai replace mathematicians? přenost furt není dostatečně univerzálně reliable, a v dost math kontextech nad llms vyhrává wolframalpha a podobný systémy, a sice už jsou nějaký nový out of distribution objevy, ale zatím jsou celkem weakly out of distribution, a musí být hodně guided lidmi, ještě se nevyřešily nějaký víc out of distribution objevy Exponential growth of emergent program synthesis paradigm https://x.com/ndea/status/1928612540030005426 Guided novelty search at inference time And then finetune on the most distant ones (distant and correct in verifiable domains) The future of AI is building world models from scratch without human biases " The new Apple reasoning paper nicely added fuel to the AI culture war. I think its polarization getting stronger and stronger. You have memetic culture war between supersceptics and superoptimists. If you give supersceptics evidence that LLMs are useful in some domain, or that something they didnt expect is happening inside the systems, they completely dismiss it. If you give superoptimists evidence that LLMs are bad in some domain, or that something they expected to happen inside the systems didnt end up happening, they completely dismiss it. Ideological confirmation bias / selection bias only primarily accepting what's aligned with their expectations and mostly ignoring everything else. But ofc there are countless people in the middle. But the extremes are the loudest. I personally dislike both of these extremes, because both completely dismiss some papers. To me ideal approach is to integrate all existing papers into a unified world view, and not deny any. But I'm definitely still a bit biased towards being more of an optimist on this spectrum, but I still attempt to take in account both the capabilities and limitations of the current systems as much as possible. There's also still so much unknown. And definitions of many concepts like intelligence and reasoning can often have so many perspectives, so I prefer to collect them all and see them as equally as possible, because each perspective scientifically gives different kinds of differently useful insights. " Reinforcement Pre-Training https://fxtwitter.com/qx_dong/status/1932268949238067482 https://arxiv.org/abs/2506.08007 the fact that diffusion models LEARN TO MAKE STRUCTURE OUT OF NOISE IS THE MOST MINDBLOWING THING EVER https://x.com/rabrg/status/1932991382173692263 AI creativity What is exactly your current view on creativity in LLMs or AI systems in general? How do you conceptualize move 37, or systems like AlphaEvolve, Robin, or maybe AlphaFold, etc.? Or the image generators sometimes producing pretty impressive art IMO (for example if the prompt is vague and you get nice surprise)? I think that they have some degree of mostly in distribution, sometimes weakly out of distribution, creativity that is in some different style from humans and more biased towards the average, it's less divergent, less coherent thanks to fuzzy spaghetti features and circuits not grounded in exact meaning enough. And strong out of distribution creativity hasn't been cracked. Moc nechápu proč ten paper tak v mainstreamu zvirálněl. Všichni z toho dělají strašný haló jak je to strašně nový groundbreaking result, ale v realitě spoustu podobných results už ve výzkumu víme dlouho, viz např papery od Subbarao. Asi protože ten titulek a jazyk je nastavenej tak aby to confirmovalo co lidi chtěj slyšet, tak to generuje hodně clicks. A v praxi se tak jak v tom paperu ty čistý deep learning modely ani většinou nepoužívaj, protože v praxi se teď používá tool use, víc směrem k neurosymbolice kterou furt zmiňuju, se kterým podobný tasky jsou mnohem menší problém, jako např ukazuje v tomhle videu, skoro každý LLM interface dělá různorodý tool use under the hood, jako např executing pythonu kódu, search atd., takže to ani není reprezentitivní co se týče toho jak většina lidí tyhle různý systémy používaj, takže spousta mainstream news articles o tomhle nedává smysl. A zrovna jsem další podobný projekt kde symbolika o něco víc integrovaná linknul. Navíc ten paper má v sobě hodně flaws, např nebrali v potaz kolik tokenů ty modely můžou maximálně outputnout pro některý tasks kde je potřeba víc tokenů, kde apparently userovi řekli ať použije ten exaktní algoritmus/tool. A některý ty tasky prej byly dokonce impossible? Prej když se to adjustne tak results vypadají jinak. Ale možná to bude souviset s tím že Apple totálně floppuje ten jejich Apple Intelligence. https://www.youtube.com/watch?v=wPBD6wTap7g (edited) New paper: World models + Program synthesis in games https://fxtwitter.com/ellisk_kellis/status/1933196127358386212 https://fxtwitter.com/ellisk_kellis/status/1933196173537693823 ︀︀topwasu.github.io/poe-world ︀︀They use gpt-4o I wonder how would performance get better with better base llms Or if you added some evolutionary features I should play with it https://github.com/topwasu/poe-world https://arxiv.org/abs/2505.11581 Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis I wonder if you could somehow automatically interpret the representations in the middle of the training of the network and somehow add into loss function some fractured representations term to somehow add incentives for unified representations I am not fundamentally pro LLMs. I am not fundamentally against LLMs. LLMs are great systems. There are other great systems for different purposes. There will be even greater all sorts of systems for all sorts of purposes. I want to maximize scientific empirical predictive power. all is thermodynamics diffusion-like methods will eat everything https://x.com/BasedBeffJezos/status/1933784135648125253 For me "knowing how something works" means that we can causally influence it. Just knowing the architecture of LLMs won't let you steer them on a more deeper level like we could steer Golden Gate Bridge Claude for example. This is what mechanistic interpretability is trying to solve. And there are still tons of unsolved problems. AI for scientific discovery requires robust neurosymbolic mathematical reasoning with strong out of distribution reasoning and evolutionary divergent novelty search AI x physics intersection is an endless rabbithole. You can try to: - make AI systems model physics better than other AI systems (ferminet, lagrangian neural networks https://arxiv.org/abs/2003.04630 latent space squeezing https://www.youtube.com/watch?v=XRL56YCfKtA ,...), - study existing AI systems using methods from physics (the principles of deep learning theory book https://arxiv.org/abs/2106.10165 statistical physics https://arxiv.org/abs/2506.04374 ,...), - design better AI architectures using insights and methods from physics (liquid neural networks https://arxiv.org/abs/2006.04439 ,...), - advance many AI architectures that are applied physics (diffusion models https://en.wikipedia.org/wiki/Diffusion_model#Non-equilibrium_thermodynamics ,...), - make neurosymbolic systems to do physics algebraically (DreamCoder https://arxiv.org/abs/2006.08381 , formal theorem proving https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/ https://arxiv.org/abs/2405.14333 ,...), - do machine designed physics experiments https://www.youtube.com/watch?v=T_2ZoMNzqHQ - help machine learning in CERN https://atlas.cern/Updates/Feature/Machine-Learning etc. Combine DeepSeek-Prover and Google's AlphaEvolve LLMs are systems that are competent enough to understand complex instructions and work at massive scale, but are unreliable if they're not grounded by a verifier or something similar I didn't realize how many normies don't do cognitively demanding tasks that AI can't do yet daily so they already offload all their thinking to ChatGPT because it's good enough for that "The real danger is not that AI is too nice to users. The real danger is that AI is too nice to nonsense." - Sabine Hossenfelder https://www.youtube.com/watch?v=oQI8W_XUmww Add curiosity to agentic LLMs In the domains you care about, use AI as an extension of yourself, not as a replacement Gwen Rld for code, Gwen Rld for math,... where is Gwen Rld for physics? DeepSeek-Prover but for physics with PhysLean when? https://physlean.com/ https://arxiv.org/abs/2504.21801 The LLM's RL Revelation We Didn't See Coming https://www.youtube.com/watch?v=z3awgfU4yno Papers discussed: Understanding R1-Zero-Like Training: A Critical Perspective https://arxiv.org/abs/2503.20783 Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model https://arxiv.org/abs/2504.13837 Reinforcement Learning Finetunes Small Subnetworks in Large Language Models https://arxiv.org/abs/2505.11711 Spurious Rewards: Rethinking Training Signals in RLVR https://arxiv.org/abs/2506.10947 Figuring out the God's reward function is all you need AlphaEvolve but the task is mutating differential equations modelling dynamical systems and evaluation function is how well the differential equations fit the ground truth data LLM sycophancy is overalignment problem Flow matching is the ultimate shapeshifter method for shapeshifting probability distributions https://www.youtube.com/watch?v=7NNxK3CqaDk