https://arxiv.org/abs/2412.01276 https://arxiv.org/abs/2410.13166 https://fxtwitter.com/SakanaAILabs/status/1866286131685498920?t=BnOxECG53IjwbSdBZwrTcg&s=19 Gradient descent x evolution https://arxiv.org/abs/1810.06773 https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012673 https://en.wikipedia.org/wiki/Neural_Turing_machine https://en.wikipedia.org/wiki/Differentiable_neural_computer deepseek https://x.com/reach_vb/status/1872000205954089011 o1 https://www.youtube.com/watch?v=6PEJ96k1kiw https://youtu.be/rJkTsNrnu8g?si=rXA27hRIV5gkVHiD https://www.youtube.com/watch?v=AfAmwIP2ntY https://fxtwitter.com/gm8xx8/status/1871814856904478798?t=9c0HsVIk_VQbAQQyTI8XZA&s=19 https://arxiv.org/abs/2412.18319 https://fxtwitter.com/gm8xx8/status/1871644052166418867?t=YNarMXqmUbhm7sEm_taDkQ&s=19 https://arxiv.org/abs/2412.17256 Kenneth Stanley https://en.wikipedia.org/wiki/Kenneth_Stanley https://arxiv.org/abs/2501.04682 https://arxiv.org/abs/2412.06769 https://x.com/NovaSkyAI/status/1877793041957933347 https://novasky-ai.github.io/posts/sky-t1/ https://arxiv.org/abs/2501.04519 https://x.com/ziv_ravid/status/1877736408191754487 https://arxiv.org/abs/2501.05707 https://llm-multiagent-ft.github.io/ https://x.com/du_yilun/status/1878851914307371440 Transformer^2 https://arxiv.org/abs/2501.06252 Cognitive agent architecture for multiagent AI civilizations: "PIANO" (Parallel Information Aggregation via Neural Orchestration) that "enables agents to interact with humans and other agents in real-time while maintaining coherence across multiple output streams" in Minecraft https://arxiv.org/abs/2411.00114 https://x.com/GuangyuRobert/status/1852397383939960926 inference time scaling of diffusion models https://fxtwitter.com/iScienceLuvr/status/1879094413588107319 https://arxiv.org/abs/2501.06848 "Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models." https://fxtwitter.com/behrouz_ali/status/1878859086227255347 https://arxiv.org/abs/2501.00663v1 Wake up babe new transformer killer dropped https://arxiv.org/abs/2501.07301 It enables visual thinking in MLLMs by generating image visualizations of their reasoning traces. https://arxiv.org/abs/2501.07542 https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2301.08028 >supervised fine-tuning (SFT) helps the model memorize and align with certain outputs, while reinforcement learning (RL) helps the model generalize and learn out-of-distribution (OOD) tasks SFT to tame initial instruction following, and then RL to generalize https://arxiv.org/abs/2501.17161 https://x.com/Hesamation/status/1884579088121073972 https://arxiv.org/abs/2411.13420 https://x.com/edwardfhughes/status/1887492625453793471?t=bXqO9PkNQOO_rgcbhCjGFw&s=19 Self play self driving I predict that reasoning in latent space will be the next set of breakthroughs Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach https://arxiv.org/abs/2502.05171 https://arxiv.org/abs/2502.05078 https://x.com/burny_tech/status/1890189798494941691 https://www.verses.ai/research-blog/variational-bayes-gaussian-splatting-a-bayesian-approach-for-continual-3d-learning https://arxiv.org/html/2410.03592v1 https://fxtwitter.com/DimitrisPapail/status/1889755872642970039?t=Vr-9NWmA1IG51D_LJzhA2Q&s=19 https://arxiv.org/abs/2502.01612 https://fxtwitter.com/omarsar0/status/1889681118913577345 https://arxiv.org/abs/2502.06049 memory-augmented Transformer architecture that incorporates a dynamic memory module large memory models reasoning papers https://fxtwitter.com/TheAITimeline/status/1888720075793793121?t=6pLwpwpjNBhPr2zJP96dLA&s=19 New Levin paper https://arxiv.org/abs/2411.13420 wordcels vs latentspacerotators https://x.com/burny_tech/status/1890194390066594120 https://arxiv.org/abs/2502.05171 https://x.com/takeru_miyato/status/1893538672533733601 https://takerum.github.io/akorn_project_page/ Artificial Kuramoto Oscillatory Neurons uses Kuramoto model which is also used in computational neuroscience https://en.wikipedia.org/wiki/Neural_oscillation#Kuramoto_model https://arxiv.org/abs/2502.17543 https://arxiv.org/abs/2502.21321 https://arxiv.org/abs/2502.09992 https://iliao2345.github.io/blog_posts/arc_agi_without_pretraining/arc_agi_without_pretraining.html - Transformers (Attention) - RNNs (LSTMs, GRUs) - [CNNs](https://arxiv.org/abs/1810.13118) - GNNs / KGs - Manifold-based learning (UMAP, t-SNE, autoencoders (VAEs)) - Spectral-based models (Fourier Neural Operators (FNOs)) - Mamba (SSMs) - Memory-Augmented NNs (NTMs, DNCs) - Energy-based & Variational models (EBMs) - KANs - Flow-based models (CNFs) - GANs - Diffusion models and Stable Diffusion - Liquid NNs / SNNs " "However, real scientific breakthroughs will come not from answering known questions, but from asking challenging new questions and questioning common conceptions and previous ideas. We're currently building very obedient students, not revolutionaries. This is perfect for today’s main goal in the field of creating great assistants and overly compliant helpers. But until we find a way to incentivize them to question their knowledge and propose ideas that potentially go against past training data, they won't give us scientific revolutions yet. If we want scientific breakthroughs, we should probably explore how we’re currently measuring the performance of AI models and move to a measure of knowledge and reasoning able to test if scientific AI models can for instance: Challenge their own training data knowledge Take bold counterfactual approaches Make general proposals based on tiny hints Ask non-obvious questions that lead to new research paths We don't need an A+ student who can answer every question with general knowledge. We need a B student who sees and questions what everyone else missed. https://x.com/Thom_Wolf/status/1897630495527104932 " https://arxiv.org/abs/2410.01131 https://arxiv.org/abs/2211.01233 enhanced knowledge graphs and reasoning LLMs https://arxiv.org/abs/2502.13025 https://youtu.be/W2uauk2bFjs?si=MVhlTpK2kbaxmt-a neural celluar automata https://distill.pub/2020/growing-ca/ https://google-research.github.io/self-organising-systems/difflogic-ca/ https://corticallabs.com/cl1.html it's the people connected to Friston who are behind the paper where they taught biological neurons to ping ping where they mention the free energy principle https://corticallabs.medium.com/in-vitro-neurons-learn-and-exhibit-sentience-when-embodied-in-a-simulated-gameworld-387ec3f2c870 https://arxiv.org/abs/2502.06034 "Traveling Waves Integrate Spatial Information Through Time In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network? Just as ripples in water carry information across a pond, traveling waves of activity in the brain have long been hypothesized to carry information from one region of cortex to another; but how can a neural network actually leverage this information? This paper introduces convolutional recurrent neural networks that learn to produce traveling waves in their hidden states in response to visual stimuli, enabling spatial integration. They made wave dynamics flexible by adding learned damping and natural frequency encoders, allowing hidden state dynamics to adapt based on the input stimulus. By then treating these wave-like activation sequences as visual representations themselves, they obtain a powerful representational space that outperforms local feed-forward networks on tasks requiring global spatial context. In particular, they observe that traveling waves effectively expand the receptive field of locally connected neurons, supporting long-range encoding and communication of information. They demonstrate that models equipped with this mechanism solve visual semantic segmentation tasks demanding global integration, significantly outperforming local feed-forward models and rivaling non-local U-Net models with fewer parameters. Video shows Tetris-like dataset and variants of MNIST to compare the semantic segmentation ability of these wave-based models with two relevant baselines: Deep CNNs w/ large receptive fields, and small U-Nets." https://arxiv.org/abs/2408.08435 automating factorio https://fxtwitter.com/bio_bootloader/status/1899147299546423507 https://jackhopkins.github.io/factorio-learning-environment/ https://x.com/renxyzinc/status/1899539629411270758 https://opensource.getren.xyz/ittm/8_usm.html Universal state machine >dynamically growing symbolic graphs Reminds me of https://arxiv.org/abs/2502.13025 https://youtu.be/W2uauk2bFjs?si=MVhlTpK2kbaxmt-a Abandoning Objectives: Evolution through the Search for Novelty Alone https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/lehman_ecj11.pdf Why Greatness Cannot Be Planned https://link.springer.com/book/10.1007/978-3-319-15524-1 #72 Prof. KEN STANLEY 2.0 - On Art and Subjectivity [UNPLUGGED] https://www.youtube.com/watch?v=DxBZORM9F-8 https://x.com/burny_tech/status/1894491541227671779 " Kenneth Stanley is my spirit animal rage against the predefined objectives embrace the fully divergent search full of novelty and accidental epiphany with serendipity https://www.youtube.com/watch?v=DxBZORM9F-8 https://www.youtube.com/watch?v=_2vx4Mfmw-w it's an evolutionary breeding process of images, but humans pick the images that should have offsprings idea: picbreeder but let multimodal LLMs instead of humans choose the next image in the evolutionary breeding process 🤔 https://www.youtube.com/watch?v=_2vx4Mfmw-w i need to look more into how those novelty/diversity algorithms that he's mentioning work, maybe they can be added into RL reward functions in LLM RL https://www.researchgate.net/publication/46424802_Abandoning_Objectives_Evolution_Through_the_Search_for_Novelty_Alone New research project: Lluminate - an evolutionary algorithm that helps LLMs break free from generating predictable, similar outputs. Combining evolutionary principles with creative thinking strategies can illuminate the space of possibilities. https://x.com/_joelsimon/status/1899884376172982392?t=Z4q0CZ2C5-9v8A-QJPnpNA&s=19 https://www.joelsimon.net/lluminate Metagradient Descent https://arxiv.org/abs/2503.13751 https://x.com/f14bertolotti/status/1902259983753842971 https://intoai.pub/p/chain-of-draft-cod-is-the-new-king https://arxiv.org/abs/2502.18600 https://x.com/METR_Evals/status/1902384481111322929?t=4SbjaExBNAyL4W-3w881lQ&s=19 https://arxiv.org/abs/2503.14499 https://arxiv.org/abs/2202.09467 https://www.youtube.com/watch?v=48GRiu-TMmg https://fxtwitter.com/ericzhao28/status/1901704339229732874?t=fJDIBbvCb2b_QIBxnsHiLw&s=19 https://arxiv.org/abs/2502.01839 https://x.com/eric_haibin_lin/status/1901662955307200974 https://x.com/Synced_Global/status/1901794723633025282?t=pIRCz8vUZxj92QQ9aMrBzg&s=19 https://dapo-sia.github.io/static/pdf/dapo_paper.pdf DAPO algorithm grpo uphrade https://www.youtube.com/watch?v=4KK2NjMrcjo https://arxiv.org/abs/2503.24322 https://www.youtube.com/watch?v=UMkCmOTX5Ow An Evolved Universal Transformer Memory [Paper] https://arxiv.org/abs/2410.13166 Memory Layers at Scale [Paper] https://arxiv.org/abs/2412.09764 Titans: Learning to Memorize at Test Time [Paper] https://arxiv.org/abs/2501.00663 https://www.youtube.com/watch?v=qhYQ20TbtJ8 CoCoNuT: Training Large Language Models to Reason in a Continuous Latent Space [Paper] https://arxiv.org/abs/2412.06769 Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach [Paper] https://arxiv.org/abs/2502.05171 biologically inspired algorithms, alternatives to backprop https://towardsdatascience.com/feedback-alignment-methods-7e6c41446e36/ https://arxiv.org/abs/2212.13345 https://medium.com/@reutdayan1/hebbian-learning-biologically-plausible-alternative-to-backpropagation-6ee0a24deb00 reinforcement learning approaches that allow language models to improve without additional data using self-play https://arxiv.org/abs/2509.07414 https://x.com/teortaxesTex/status/1965654111069876296