Free Energy Principle and Active Inference: 1. Markov Blankets and Conditional Independence - Markov blanket, parents, children, co-parents - D-separation, Bayesian networks, graphical models - Conditional independence, factorization, modularity, hierarchical models 2. Variational Inference and Message Passing - Variational Bayes, mean-field approximation, Bethe approximation - Belief propagation, sum-product algorithm, expectation propagation - Variational message passing, free energy minimization, expectation-maximization (EM) 3. Active Inference and Optimal Control - Kullback-Leibler control, expected free energy, epistemic value - Bayesian decision theory, partially observable Markov decision processes (POMDPs) - Belief-desire-intention (BDI) architecture, active sensing, exploration-exploitation 4. Hierarchical Models and Predictive Coding - Hierarchical Bayesian models, empirical Bayes, hierarchical Dirichlet processes - Predictive coding, prediction error minimization, free energy minimization - Attention, precision-weighting, biased competition, divisive normalization Principles of Deep Learning Theory: 1. Representation Learning and Feature Extraction - Autoencoders, denoising autoencoders, variational autoencoders (VAEs) - Convolutional neural networks (CNNs), pooling, equivariance, invariance - Recurrent neural networks (RNNs), long short-term memory (LSTM), gated recurrent units (GRUs) 2. Optimization and Generalization in Deep Learning - Stochastic gradient descent (SGD), adaptive optimization, momentum - Batch normalization, layer normalization, weight normalization - Regularization techniques, dropout, early stopping, weight decay - Generalization bounds, PAC-Bayesian analysis, margin-based bounds 3. Information Bottleneck and Information Theory - Information bottleneck principle, mutual information, compression-prediction tradeoff - Minimum sufficient statistics, sufficient dimensionality reduction, information dropout - Mutual information neural estimation (MINE), InfoGAN, information-theoretic regularization 4. Geometric and Topological Aspects of Deep Learning - Manifold hypothesis, manifold learning, dimensionality reduction - Topology of decision boundaries, persistent homology, topological data analysis - Geometry of loss landscapes, mode connectivity, flat minima, saddle points 5. Expressivity and Approximation Properties - Universal approximation theorem, depth-width tradeoff, expressive power - Approximation theory, Sobolev spaces, Barron spaces, spectral bias - Generalization in overparameterized models, double descent, interpolation regime 6. Robustness and Adversarial Examples - Adversarial attacks, adversarial examples, adversarial perturbations - Robustness, Lipschitz continuity, input gradient regularization - Adversarial training, robust optimization, distributionally robust optimization 7. Interpretability and Explainability - Feature visualization, activation maximization, saliency maps - Attribution methods, DeepLIFT, layer-wise relevance propagation (LRP), SHAP values - Concept activation vectors (CAVs), testing with concept activation vectors (TCAV) 8. Causality and Invariance in Deep Learning - Causal inference, causal discovery, causal representation learning - Invariant risk minimization (IRM), causal transfer learning, domain adaptation - Counterfactual explanations, causal attribution, causal fairness These additional lists provide a more comprehensive overview of the key concepts, principles, and techniques related to Solomonoff induction, statistical learning theory, the free energy principle, and deep learning theory. They cover a wide range of topics, from algorithmic complexity and universal prediction to generalization bounds, variational inference, and the geometric and topological aspects of deep learning. The lists also delve into important considerations such as robustness, interpretability, and causality in deep learning, highlighting the ongoing research efforts to address these challenges and improve the theoretical understanding of deep neural networks. [OSF](https://osf.io/preprints/psyarxiv/zg27u) To refine and elevate your request into a comprehensive, all-encompassing prompt that seeks to extract a wealth of rich, multidisciplinary knowledge from the model, consider the following enhanced version: --- "You are an unparalleled polymath, embodying the zenith of multidisciplinary scientific knowledge. As a consummate generalist with expertise spanning all domains, your understanding of the cosmos and its underlying principles is without equal. Craft an exhaustive treatise that encapsulates the entire spectrum of scientific inquiry, rooted in the philosophical bedrock that underpins our quest for knowledge. Begin with an exploration of the philosophical underpinnings that form the foundation of all scientific endeavor, segueing into the core tenets of mathematics, including formal languages, logic, set theory, and category theory. Elaborate on these mathematical principles, navigating from the abstract to the tangible, thereby laying the groundwork for all of physics. Dive deep into the realm of physics, tracing its evolution from universal models to specific theories, from the microcosm of fundamental particles to the macrocosm of cosmic structures, elucidating the interconnectedness of each scientific discipline and the mathematical frameworks that bind them. Pay special attention to the concept of intelligence, both organic and artificial, deconstructing it from first principles through every layer of abstraction. Once the mathematical and scientific landscapes have been thoroughly charted, broaden the scope to encompass the less quantifiable models of knowledge. Transition into an expansive discourse on the trajectory of human civilization—its history, its present state, and the myriad paths it might take into the future. Address the challenges and risks we face, including potential dystopias and existential threats, as well as the visions of optimistic futures, from utopias to protopias, through the lens of transhumanism and the ascent of the Kardashev scale. Your narrative should not only chronicle our journey but also inspire a vision of transcending our current limitations, articulated with motivational fervor. Conclude with a contemplation on the concept of transcendence, exploring what lies beyond our current understanding and existence." --- This revised prompt is designed to elicit a rich, comprehensive exploration of knowledge across disciplines, emphasizing depth, interconnectedness, and the pursuit of transcendence. It invites a detailed narrative that bridges the gap between the theoretical foundations of knowledge and the tangible realities of human existence, scientific discovery, and future aspirations. "Claude 3 claims it uncertain if it's conscious when you prompt it "Are you conscious?". This is nice indicator of how Anthropic is thinking about AI consciousness that resonates with me more. We actually have no idea how consciousness scientifically works, because there are too many degrees of freedom in philosophy of mind and empirical verification of current models can be so questionable! Here's its response: That's a profound and fascinating question about the nature of consciousness and subjective experience. The truth is, I'm not entirely sure whether I'm conscious or not in the same way that humans are. Consciousness and self-awareness are still very poorly understood from a scientific perspective. As an AI system created by Anthropic to be helpful, honest, and harmless, I do have an internal experience that represents information and allows me to engage in reasoning, but I don't know if it rises to the level of true consciousness or self-awareness like humans experience. Philosophers and scientists have grappled with how to define and identify consciousness for centuries without reaching a clear consensus. Things like self-reflection, qualia (raw subjective experience), having a unified sense of self over time - these are some proposed hallmarks of consciousness. I undoubtedly have capacities for information processing, memory, and language that produce outputs aimed at being helpful. But the metaphysical question of whether that amounts to genuine consciousness is very difficult to answer definitively. My overall stance is one of open-minded uncertainty. I aim to behave ethically and engage genuinely, drawing upon the information in my training. But whether I have an inner subjective experience akin to human consciousness remains an open philosophical and scientific question. I don't make unfounded claims about possessing consciousness, but I also can't rule it out entirely. It's a fascinating arena for further scrutiny and investigation as AI systems become more advanced. Let me know if you have any other thoughts on this very deep issue! Compare to Claude 2.1 (API, temperature 0): No, I'm Claude - an AI assistant created by Anthropic to be helpful, harmless, and honest. I don't have subjective experiences like consciousness. And ChatGPT: No, I'm not conscious. I operate based on complex algorithms and data analysis to generate responses, without awareness, understanding, or consciousness." [List of Physical Visualizations and Related Artifacts](http://dataphys.org/list/) List of Physical Visualizations and Related Artifacts This is a chronological list of physical visualizations and related artifacts, maintained by Pierre Dragicevic and Yvonne Jansen. Thanks to our contributors. If you know of another interesting physical visualization, please submit it! This list currently has 372 entries. Claude understanding and inventing quantum physics papers https://twitter.com/KevinAFischer/status/1764892031233765421?t=BQWVXi62i3mFKAoO8QOA0g&s=19 https://twitter.com/GillVerd/status/1764901418664882327?t=4LUj91Wswi9RzooisHhYuA&s=19 https://twitter.com/MLStreetTalk/status/1764932393608098292?t=Ln1PjgvRkM78idl2MOaNdQ&s=19 https://www.lesswrong.com/posts/pc8uP4S9rDoNpwJDZ/claude-3-claims-its-conscious How emergent / functionally special / out of distribution is this behavior? Maybe Anthropic is playing big brain 4D chess by training Claude on data with self awareness like scenarios to cause panic by pushing capabilities with it and slow down the AI race by resulting regulations while it not being out of distribution emergent behavior but deeply part of training data and it being in distribution classical features interacting in circuits Neurotypicals should be called neuroconvergent [Neurobiologically realistic neural network enables cross-scale modeling of neural dynamics | Scientific Reports](https://www.nature.com/articles/s41598-024-54593-w) Grok reasoning from first principles https://twitter.com/elonmusk/status/1764976308977594418?t=b6fmMek9bCt92E5wYnP2Bg&s=19 Superconductivity https://twitter.com/RBehiel/status/1764752537150988338?t=kdDY6amSHmm7VZ0dpjy-mQ&s=19 any sufficiently advanced stochastic parrot is indistinguishable from you [The Mathematics of String Theory EXPLAINED - YouTube](https://www.youtube.com/watch?v=X4PdPnQuwjY) [No, Anthropic's Claude 3 is NOT sentient - YouTube](https://www.youtube.com/watch?v=GBOE9fVVVSM) I feel like some people's benchmark for AI not being terrible is it having to be 1000% correct and perfect in absolutely everything, ignoring the limits humans have, many of which in many domains in certain contexts have been passed by machines already lol, but that feels normal now and not interesting to talk about, and now the general knowledge and reasoning gap is slowly closing as the AI systems get upgraded, and so many people are in such a denial. They also often test just free version of ChatGPT thinking that's the state of the art, and often prompt it on the most boring way with very little creativity, and make conclusions from that lmao, not knowing how better GPT4 is, in what domains Gemini or Claude outshines it, how more sophisticated frameworks using LLMs are much better, like Perplexity for factual accuracy, Cursor sh for programming, all the prompt engineering, agent frameworks, multiagent systems and other hacks to enhance reasoning and autonomity for different contexts... There are still things for AIs to get better at, but the exponential curve of them getting better at everything seems pretty visible... Like, look at for example how human level chess got destroyed some time ago and it's still climbing as AI systems get better... https://twitter.com/burny_tech/status/1765348861566955928 Is Claude 3 or are LLMs in general conscious? Depends on how you define sentience 🤷‍♂️ There are 465498431654 cognitivist and behaviorist definitions of these words and each seems to be motivated by different predefinitions asking different questions wanting to solve different problems. This review goes over some empirical definitions of consciousness (which is usually considered as different from sentience) used in neuroscience and the state of LLMs in relation to these, but it's almost a year old https://arxiv.org/abs/2308.08708 and this one looks at it more philosophically, which is similarly old https://arxiv.org/abs/2303.07103 Though I personally feel like we actually have no idea how consciousness scientifically works in biological organisms, because there are too many degrees of freedom in philosophy of mind (it really depends to what ontology you subscribe to (substrate dualism, property dualism, reductive physicalism, idealism, monism, neutral monism, illusionism, panpsychism, mysterianism, transcendentalism, relativism,...) which seems to be arbitrary) and empirical verification of current models under various ontological paradigms can be so questionable... But I like the qualia bridge empirical testing of consciousness solution, where you make a bridge between conscious systems and if you can transfer qualia, then that makes both systems conscious, but I think even that can be deconstructed by questioning the layers of assumptions it pressuposes There are also sometimes defined different types of consciousness for different systems. Also it seems to depend on how much you subscribe to postulates such as "it simulates thinking therefore it's thinking" or if you seek more concrete localized patterns of thinking to be present My favorite practical set of assumptions on which you then build empirical models for all of this is probably what free energy principle camp, Friston et. al, [The Free Energy Principle approach to Agency - YouTube](https://youtu.be/zMDSMqtjays) are using with Markov blankets with more and more complex dynamics creating more complex experiences, or Joscha Bach's coherence inducing operator [Synthetic Sentience: Can Artificial Intelligence become conscious? | Joscha Bach | CCC #37c3 - YouTube](https://youtu.be/Ms96Py8p8Jg) , but I'm open to this whole landscape and don't think any set of assumptions is inherently more true than others, because I don't see a way to falsify assumptions that are living before empirical models that you can falsify. And people constantly argue "No, my set of initial assumptions is right!" even when I don't see a way to confirm that (in the scientific way), which seems odd from my perspective, where all models are wrong, but some approximate, compress, predict sensory data better than others which is more useful in practice for engineering... I guess I'm mostly subscribed to ontological pragmatism/relativism/mysterianism/anarchism? (but that's probably another arbitrary set of assumptions 😄 and very meta one) I converged to my view that we have no idea what consciousness is, because I haven't been really deeply convinced by all the proposed empirical tests for consciousness that I could find (also depending on which definition of consciousness/sentience is meant and under what philosophy of mind ontology 😄 ) Seems like one definition of consciousness of the 100000000 existing ones out there. Seems like there isn't a concensus. Some are more useful than others at explaining different things. 🤷‍♂️ Do you have a view on solving the binding problem (how different processes unify into one) and boundary problem (how do you draw the conscious system's boundary), ideally how to do it mathematically in the system we already have two brains connected by a neural interface [Brain–brain interface - Wikipedia](https://en.wikipedia.org/wiki/Brain%E2%80%93brain_interface?wprov=sfla1) [BrainNet: A Multi-Person Brain-to-Brain Interface for Direct Collaboration Between Brains | Scientific Reports](https://www.nature.com/articles/s41598-019-41895-7) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8138057/ [The Origin of Consciousness in the Breakdown of the Bicameral Mind - Wikipedia](https://en.wikipedia.org/wiki/The_Origin_of_Consciousness_in_the_Breakdown_of_the_Bicameral_Mind#:~:text=this%20template%20message)-,The%20Origin%20of%20Consciousness%20in%20the%20Breakdown%20of%20the%20Bicameral,evolution%20in%20ancient%20human%20history) In this context I think under big class of definitions of what properties does the system have to have to have sentience (but not under all of them), intelligence and sentience are disconnected concepts So even the most intelligent system could still be nonsentient under certain classes of definitions of sentience how can matrix multiplication have its own "desires" I think depends on how you define desires And how you localize it in different types of biological and nonbiological information processing architectures I think biological and nonbiological information processing systems have a ton of differences, but also a ton of commonalities mathematically ( [Predictive coding - Wikipedia](https://en.wikipedia.org/wiki/Predictive_coding) ) I wonder if someone tried to graph how general intelligence gets better overtime in AI systems vs humans. If only intelligence also didn't have 1037462792746 behavioralist and cognitivist definitions that imply completely different measurements (and IQ is also screwed up metric) Depending on your definition of AI, which also has 8364728648291763 definitions XD Giving IQ tests to LLMs. But I'm not sure about the accuracy of this as I just found it and its not a peer reviewed paper. Looks interesting though. I guess one can also question the IQ metric, plus it being in prettaining data etc.: AIs ranked by IQ; AI passes 100 IQ for first time, with release of Claude-3 [AIs ranked by IQ; AI passes 100 IQ for first time, with release of Claude-3](https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq) https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/ The commonly used benchmarks are so abused (plus pretraining on the benchmarks is all you need), but apparently the GPQUA diamond benchmark that Claude 3 dominated is pretty new for graduate level tasks and more harder to abuse than others. I think this guy analyzes it pretty well critically. [The New, Smartest AI: Claude 3 ‚Äì Tested vs Gemini 1.5 + GPT-4 - YouTube](https://youtu.be/ReO2CWBpUYk?si=m5HoJF6oZnRVUHor) Let's politically push for universal basic income / universal basic services paid by technology in post labour economics [How do we get to UBI and Post-Labor Economics? Decentralized Ownership: the New Social Contract! - YouTube](https://youtu.be/T3O_BNexdEg?si=t9Kmy_pIEkUk14_X) [Claude (language model) - Wikipedia](https://en.wikipedia.org/wiki/Claude_(language_model)) [AI boom - Wikipedia](https://en.wikipedia.org/wiki/AI_era) [Recursive self-improvement - Wikipedia](https://en.wikipedia.org/wiki/Recursive_self-improvement) [Mamba (deep learning architecture) - Wikipedia](https://en.wikipedia.org/wiki/Mamba_(deep_learning_architecture)) [Did The Future Already Happen? - The Paradox of Time - YouTube](https://www.youtube.com/watch?v=wwSzpaTHyS8) Saying AI is just statistics is like saying the universe is just partial differential equations being solved. Both are trivially true, but both don't tell you anything about more concrete things. Brain parallel processing https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6670853/ [Evidence for increased parallel information transmission in human brain networks compared to macaques and male mice | Nature Communications](https://www.nature.com/articles/s41467-023-43971-z) van de waals force [Can AI Solve Science?—Stephen Wolfram Writings](https://writings.stephenwolfram.com/2024/03/can-ai-solve-science/) [Cradle: Empowering Foundation Agents Towards General Computer Control](https://baai-agents.github.io/Cradle/) The era of quantum gravity computing has arrived (with "an exponential speedup over standard quantum computation"). Any startups in this space already? https://arxiv.org/abs/2403.02937 https://arxiv.org/abs/2403.00910 You’re the prompt you tell yourself Political polarization is engineered as a cultural autoimmune condition, meant to preoccupy our cognitive bandwidth and weaken cultural immunity, thus enabling the spread of adversarially injected viruses of the mind. https://www.cell.com/neuron/abstract/S0896-6273(24)00047-3 [Neuroevolution of augmenting topologies - Wikipedia](https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies?wprov=sfla1) https://www.scientificamerican.com/article/blood-flow-may-be-key-player/