A recurrent neural network is like a looped decision-making process, where each decision considers not only the current information but also what was learned from previous data points, making it well-suited for tasks involving sequences like language or time series data. More formally, it's a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence, allowing it to exhibit dynamic temporal behavior and process sequences of inputs. ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Artificial neural networks -> Long short-term memory=== Long short-term memory (LSTM) network is a recurrent neural network (RNN), aimed to deal with the vanishing gradient problem present in traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, hidden Markov models and other sequence learning methods. It aims to provide a short-term memory for RNN that can last thousands of timesteps, thus "long short-term memory". [Long short-term memory - Wikipedia](https://en.wikipedia.org/wiki/Long_short-term_memory) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Artificial neural networks -> Convolutional neural network=== A convolutional neural network is a type of deep learning algorithm primarily used for processing data with a grid-like topology, such as images, by applying convolutional layers to extract features and patterns. For image classification, segmentation etc. [Convolutional neural network - Wikipedia](https://en.wikipedia.org/wiki/Convolutional_neural_network) [Convolutions in image processing | Week 1 | MIT 18.S191 Fall 2020 | Grant Sanderson - YouTube](https://www.youtube.com/watch?v=8rrHTtUzyZA&list=PLZHQObOWTQDMp_VZelDYjka8tnXNpXhzJ&index=1&pp=iAQB) [[Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions | ACM Computing Surveys](https://dl.acm.org/doi/full/10.1145/3563691) Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions] [https://www.sciencedirect.com/science/article/pii/S0952197622005966 An analysis of explainability methods for convolutional neural networks] ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Artificial neural networks -> Symmetries and Geometric deep learning=== Deep learning is hard. While universal approximation theorems show that sufficiently complex neural networks can in principle approximate “anything”, there is no guarantee that we can find good models. Great progress in deep learning has nevertheless been made by judicious choice of model architectures. These model architectures encode inductive biases to give the model a helping hand. One of the most powerful inductive biases is to leverage notions of geometry, giving rise to the field of geometric deep learning. Fundamentally, geometric deep learning invovles encoding a geometric understanding of data as an inductive bias in deep learning models to give them a helping hand. Traditional deep learning focuses on Euclidean data (like images, text, and audio), which can be represented in regular grids. In contrast, geometric deep learning deals with non-Euclidean data such as graphs and manifolds. While learning generic functions in high dimensions is a cursed estimation problem, many tasks are not uniform and have strong repeating patterns as a result of the low-dimensionality and structure of the physical world. Geometric Deep Learning unifies a broad class of ML problems from the perspectives of symmetry and invariance. These principles not only underlie the breakthrough performance of convolutional neural networks and the recent success of graph neural networks but also provide a principled way to construct new types of problem-specific inductive biases: Symmetry and invariance, Stability, Multiscale representations. [Geometric Deep Learning - Grids, Groups, Graphs, Geodesics, and Gauges](https://geometricdeeplearning.com/) [A Brief Introduction to Geometric Deep Learning | by Jason McEwen | Towards Data Science](https://towardsdatascience.com/a-brief-introduction-to-geometric-deep-learning-dae114923ddb) [GEOMETRIC DEEP LEARNING BLUEPRINT - YouTube](https://www.youtube.com/watch?v=bIZB1hIJ4u8) [AMMI 2022 Course "Geometric Deep Learning" - Lecture 1 (Introduction) - Michael Bronstein - YouTube](https://www.youtube.com/watch?v=5c_-KX1sRDQ&list=PLn2-dEmQeTfSLXW8yXP4q_Ii58wFdxb3C) Other attempts at working with symmetries in Machine Ĺearning: [[[2311.00212] A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learning]([[2311.00212] A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learning](https://arxiv.org/abs/2311.00212)) A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learning] [[[2301.05638] Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles](https://arxiv.org/abs/2301.05638) Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles] [[What Einstein Can Teach Us About Machine Learning | by Jason McEwen | Towards Data Science](https://towardsdatascience.com/what-einstein-can-teach-us-about-machine-learning-1661e26bef2c) What Einstein Can Teach Us About Machine Learning] [[Frontiers | Symmetry-Based Representations for Artificial and Biological General Intelligence]([Frontiers | Symmetry-Based Representations for Artificial and Biological General Intelligence](https://www.frontiersin.org/articles/10.3389/fncom.2022.836498/full)) Symmetry-Based Representations for Artificial and Biological General Intelligence] ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Artificial neural networks -> Geometric deep learning -> Graph neural networks=== A class of artificial neural networks for processing data that can be represented as graphs. [Graph neural network - Wikipedia](https://en.wikipedia.org/wiki/Graph_neural_network) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Artificial neural networks -> Geometric deep learning -> Manifold Learning=== Nonlinear dimensionality reduction, also known as manifold learning, refers to various related techniques that aim to project high-dimensional data onto lower-dimensional latent manifolds [Nonlinear dimensionality reduction - Wikipedia](https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Transformer=== This currently the most popular model in AI, where most of language models are build on top of it. It works by learning to what to pay attention to in its token window when predicting the next token. Much more accurate than bag of words or word wectors used before in natural language processing. All giant state of the art language models use giant stacks of those. [Illustrated Guide to Transformers Neural Network: A step by step explanation - YouTube](https://www.youtube.com/watch?v=4Bdc55j80l8) [What are Transformer Neural Networks? - YouTube](https://www.youtube.com/watch?v=XSSTuhyAmnI) [[[1706.03762] Attention Is All You Need]([[1706.03762] Attention Is All You Need](https://arxiv.org/abs/1706.03762)) Attention Is All You Need, introducing transformers] [Transformer Neural Networks, ChatGPT's foundation, Clearly Explained!!! - YouTube](https://www.youtube.com/watch?v=zxQyTK8quyY) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Nonsymbolic aka Connectionist -> Geoffrey Hinton's Forward-Forward Algorithm=== This new approach, known as the Forward-Forward algorithm, is inspired by our understanding of neural activations in the brain. brain doesn't seem to do back propagation used by traditional artificial neural networks. Instead of relying on the traditional forward and backward passes of backpropagation, this method utilizes two forward passes — one with positive, real data and the other with negative data. Each layer in the network has its own objective function, which is to have high “goodness” for positive data and low “goodness” for negative data. [[[2212.13345] The Forward-Forward Algorithm: Some Preliminary Investigations](https://arxiv.org/abs/2212.13345) The Forward-Forward Algorithm: Some Preliminary Investigations] [Forward-Forward Algorithm. Geoffrey Hinton, a renowned researcher… | by Barhoumi Mosbeh | Medium]([Forward-Forward Algorithm. Geoffrey Hinton, a renowned researcher… | by Barhoumi Mosbeh | Medium](https://medium.com/@Mosbeh_Barhoumi/forward-forward-algorithm-ac24d0d9ffd)) [Unpacking The Forward-Forward Algorithm with Geoffrey Hinton - YouTube](https://www.youtube.com/watch?v=eEFOhoRmhEQ) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Symbolic nonsymbolic hybrids=== ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Symbolic nonsymbolic hybrids -> Active Inference=== https://direct.mit.edu/books/oa-monograph/5299/Active-InferenceThe-Free-Energy-Principle-in-Mind [Active Inference ModelStream #007.1 ~ Conor Heins & Daphne Demekas ~ pymdp - YouTube]([Active Inference ModelStream #007.1 ~ Conor Heins & Daphne Demekas ~ pymdp - YouTube](https://www.youtube.com/watch?v=skf3sOM-7WI)) Physics and brain inspired model for machine learning based on creating generative model predicting data finetuned by bayesian mechanics: partially observed markov decision process. ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Symbolic nonsymbolic hybrids -> Yann LeCun's Image Joint Embedding Predictive Architecture=== [The first AI model based on Yann LeCun’s vision for more human-like AI]([The first AI model based on Yann LeCun’s vision for more human-like AI](https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/)) https://openreview.net/pdf?id=BZ5a1r-kVsf [JEPA - A Path Towards Autonomous Machine Intelligence (Paper Explained) - YouTube]([JEPA - A Path Towards Autonomous Machine Intelligence (Paper Explained) - YouTube](https://www.youtube.com/watch?v=jSdHmImyUjk)) Learns by creating an internal model of the outside world, a unified representation of different types of data, which compares abstract representations of images (rather than comparing the pixels themselves) The predictive model could be anything from a simple linear regressor to a complex deep neural network, depending on the application and the complexity of the task at hand. ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Symbolic nonsymbolic hybrids -> Yoshua Bengio's Generative Flow Networks=== [Generative Flow Networks - Yoshua Bengio]([Generative Flow Networks - Yoshua Bengio](https://yoshuabengio.org/2022/03/05/generative-flow-networks/)) [[2202.13903] Bayesian Structure Learning with Generative Flow Networks]([[2202.13903] Bayesian Structure Learning with Generative Flow Networks](https://arxiv.org/abs/2202.13903)) [Yoshua Bengio on Pausing More Powerful AI Models and His Work on World Models - YouTube]([Yoshua Bengio on Pausing More Powerful AI Models and His Work on World Models - YouTube](https://youtu.be/I5xsDMJMdwo?si=bUM7LWPKICSdF-RF)) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Diffusion model=== In machine learning, diffusion models, also known as diffusion probabilistic models or score-based generative models, are a class of generative models. The goal of diffusion models is to learn a diffusion process that generates the probability distribution of a given dataset. They are used to generate image by reversing the diffusion process. [Diffusion model - Wikipedia](https://en.wikipedia.org/wiki/Diffusion_model) [What are Diffusion Models? - YouTube](https://www.youtube.com/watch?v=fbLgFrlTnGU) [Diffusion models explained. How does OpenAI's GLIDE work? - YouTube](https://www.youtube.com/watch?v=344w5h24-h8) [Diffusion Models | Paper Explanation | Math Explained - YouTube](https://www.youtube.com/watch?v=HoKDTa5jHvg) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Selfreferential Meta-learning=== Meta learning is a subfield of machine learning where automatic learning algorithms are applied to metadata about machine learning experiments. AIs selfimproving themselves! [Meta-learning (computer science) - Wikipedia]([Meta-learning (computer science) - Wikipedia]([Meta-learning (computer science) - Wikipedia]([Meta-learning (computer science) - Wikipedia]([Meta-learning (computer science) - Wikipedia](https://en.wikipedia.org/wiki/Meta-learning_(computer_science)))))) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Quantum machine learning=== The most common use of the term refers to machine learning algorithms for the analysis of classical data executed on a quantum computer, i.e. quantum-enhanced machine learning [Quantum machine learning - Wikipedia]([Quantum - Wikipedia](https://en.wikipedia.org/wiki/Quantum)_machine_learning) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Selforganizing=== [The Future of AI is Self-Organizing and Self-Assembling (w/ Prof. Sebastian Risi) - YouTube](https://www.youtube.com/watch?v=_7xpGve9QEE) [Self-organizing map - Wikipedia](https://en.wikipedia.org/wiki/Self-organizing_map) ===Science -> Technology -> Artificial Intelligence -> Paradigms -> Thermodynamic AI=== https://twitter.com/ColesThermoAI/status/1625368583789334528 [[2302.06584] Thermodynamic AI and the fluctuation frontier](https://arxiv.org/abs/2302.06584) [The Normal Blog - A First Demonstration of Thermodynamic Matrix Inversion]([The Normal Blog - A First Demonstration of Thermodynamic Matrix Inversion]([The Normal Blog - A First Demonstration of Thermodynamic Matrix Inversion](https://blog.normalcomputing.ai/posts/2023-11-09-thermodynamic-inversion/thermo-inversion.html))) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Language=== [Language model - Wikipedia](https://en.wikipedia.org/wiki/Language_model) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Language -> Transformers=== Transformer based language models have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model. ===Science -> Technology -> Artificial Intelligence -> Modalities -> Language -> Language models -> Tranformers=== ===Science -> Technology -> Artificial Intelligence -> Modalities -> Language -> Language models -> Tranformers -> GPTx=== The most famous big LLMs are GPTx by OpenAI ===Science -> Technology -> Artificial Intelligence -> Modalities -> Language -> Language models -> Tranformers -> Claude=== By Anthropic backed up by Amazon and Google. ===Science -> Technology -> Artificial Intelligence -> Modalities -> Language -> Language models -> Tranformers -> LLama=== Biggest open source LLM backed by Meta (Facebook) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Image -> Diffusion model -> Dalle 3=== Dalle is currently the best image generator, based on the diffusion model architecture. it is used to generate images. [DALL-E 3 is better at following Text Prompts! Here is why. — DALL-E 3 explained - YouTube](https://www.youtube.com/watch?v=NTGRcTRlcE4) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Video -> Runway=== [Runway - Advancing creativity with artificial intelligence.](https://runwayml.com/) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Embodied -> Robotics -> Boston dynamics=== ===Science -> Technology -> Artificial Intelligence -> Modalities -> Embodied -> Robotics -> Eureka=== ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal=== ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> OpenAI's ChatGPT4 Turbo and GPTs=== The bets multimodal LLMs so far. Backed by Microsoft. [GPT 4 Turbo-Charged? Plus Custom GPTS, Grok, AGI Tier List, Vision Demos, Whisper V3 and more - YouTube](https://www.youtube.com/watch?v=RkresCXaVLE) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> Google's Gemini=== Gemini is a multimodal model 10 bigger than GPT4 inside Google that has been leaked multiple times already. [GOBI BY OPENAI VS GEMINI BY GOOGLE: The next generation of AI | by Shiza Akif | Medium]([GOBI BY OPENAI VS GEMINI BY GOOGLE: The next generation of AI | by Shiza Akif | Medium](https://medium.com/@shizaakif/gobi-by-openai-vs-gemini-by-google-the-next-generation-of-ai-3ed484be5086)) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> OpenAI's Gobi=== Gobi is a leaked model trained by OpenAI tat's also many times bigger than GPT4 [GOBI BY OPENAI VS GEMINI BY GOOGLE: The next generation of AI | by Shiza Akif | Medium]([GOBI BY OPENAI VS GEMINI BY GOOGLE: The next generation of AI | by Shiza Akif | Medium](https://medium.com/@shizaakif/gobi-by-openai-vs-gemini-by-google-the-next-generation-of-ai-3ed484be5086)) ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> Elon Musk's Grok=== Counterculture to OpenAI. Attempts to be more humorous and less "censored". https://twitter.com/xai/status/1721027348970238035 https://twitter.com/elonmusk/status/1722893370010313117 ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> Gato=== ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> Active Inference=== ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> Active Inference -> Verses Genius=== More explanable than transformers! [[- YouTube](https://www.youtube.com/watch?v=mIUcU5c-vEs) VERSES AI Introduces Genius™ - Youtube] ===Science -> Technology -> Artificial Intelligence -> Modalities -> Multimodal -> Active Inference -> Digital Gaia=== [The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium]([The Gaia Attractor. A planetary AI copilot network to overcome the Metacrisis | Rafael Kaufmann | Medium](https://rkauf.medium.com/the-gaia-attractor-41e5af33f3b7)))))))))))))))) [AI Infrastructure for Planetary Decision Intelligence]([AI Infrastructure for Planetary Decision Intelligence](https://www.digitalgaia.earth/))