Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs - Outperforms DALL-E 3 and SDXL, particularly in multi-category object composition and text-image semantic alignment with chain-of-thought [x.com](https://twitter.com/burny_tech/status/1751648334698123758) [[2401.11708v1] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://arxiv.org/abs/2401.11708v1) [GitHub - YangLing0818/RPG-DiffusionMaster: [ICML 2024] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (PRG)](https://github.com/YangLing0818/RPG-DiffusionMaster) https://www.scientificamerican.com/article/brains-are-not-required-when-it-comes-to-thinking-and-solving-problems-simple-cells-can-do-it/ LLMs are generalizing and creating new knowledge, but not sufficiently enough in sufficiently consistent way, we can do better with hybrid systems [Joscha Bach Λ Ben Goertzel: Conscious Ai, LLMs, AGI - YouTube](https://youtu.be/xw7omaQ8SgA?si=72Q9yfvdpM_Tcmx9&t=3383) LLMs are generalizing, but not human like, we are more hiearchical Maxxing out all organism's intellectual and physical hyperparamaters acceleration Incompatibility of quantum mechanics and general relativity [ChatGPT](https://chat.openai.com/share/e0188bb7-daae-4452-bbed-b7163e36a291) We are trying to find the best possible token, while LLMs are trying to find the next most likely token. You can think any perspective and just see how much it sticks with the rest of your perspective programs in an evolutionary battle for stability in terms of usefulness or other cost functions You can think any perspective and just see how much it sticks with the rest of your perspective programs in an evolutionary battle for stability in terms of usefulness or other cost functions """I""" """"identity"""" as informational topological pocket That sometimes lets itself go into ieffable void beyond formalizations and language https://nationalinterest.org/feature/new-lithium-discoveries-can-secure-america%E2%80%99s-clean-energy-future-208808 [Cyc - Wikipedia](https://en.wikipedia.org/wiki/Cyc) Cyc is a long-term artificial intelligence project that aims to assemble a comprehensive ontology and knowledge base that spans the basic concepts and rules about how the world works. Hoping to capture common sense knowledge, Cyc focuses on implicit knowledge that other AI platforms may take for granted. This is contrasted with facts one might find somewhere on the internet or retrieve via a search engine or Wikipedia. Cyc enables semantic reasoners to perform human-like reasoning and be less "brittle" when confronted with novel situations. [[1805.08592] Computable Variants of AIXI which are More Powerful than AIXItl](https://arxiv.org/abs/1805.08592) our understanding of higher intelligence was severely limited by the N=1 example of humans now we have LLMs and can start to make more progress, figuring out which aspects of intelligence can actually be independent, and what is/isn't required for different tasks Are LLMs enough for human level intelligence, or overkill, or are they completely alien intelligences? Given sufficient amount of time and resources, any bruteforce algorithm can in theory come up with any arbitrary creative (useful) pattern. Question about (embodied) (realtime) LLMs might not be "Can LLMs do everything humans can do, possibly including tons of things humans can't do?" But: "In order to be better than all humans at everything, do Transformers, Mamba, deep neural networks in general need 10x, 100x, 1000x, etc. more resources, time, engineering tricks, compatible hardware, etc. than they already use and have, or 420 quintilion more such things to converge to such a level?" Or: "In the statespace of all possible intelligences (statespace of all possible information processing systems), how close are LLM intelligences, given their training data, hardware and architectural details, to human intelligence running on this biological electrochemical hardware that was evolved for billions of years? As we scale and mutate their implementation details, do their paths converge or diverge in the limit in the statespace of all possible intelligences? How similar are evolutionary biologically learned inductive biases versus the ones we construct into machines?" I'm starting to like more and more free energy principle's point of view where our learned abstractions are in big part influenced by our curious, agentic and motivational architecture. What is your definition of intelligence? Mine is pretty general. I could have also said "Space of all possible information processing systems" and meaning would be very similar. [Fokker–Planck equation - Wikipedia](https://en.m.wikipedia.org/wiki/Fokker%E2%80%93Planck_equation) "A 4-year-old child has seen 50x more information than the biggest LLMs that we have." - @ylecun " - LLM: 1E13 tokens x 0.75 word/token x 2 bytes/token = 1E13 bytes. - 4 year old child: 16k wake hours x 3600 s/hour x 1E6 optical nerve fibers x 2 eyes x 10 bytes/s = 1E15 bytes. In 4 years, a child has seen 50 times more data than the biggest LLMs. 1E13 tokens is pretty much all the quality text publicly available on the Internet. It would take 170k years for a human to read (8 h/day, 250 word/minute). Text is simply too low bandwidth and too scarce a modality to learn how the world works. Video is more redundant, but redundancy is precisely what you need for Self-Supervised Learning to work well. Incidentally, 16k hours of video is about 30 minutes of YouTube uploads." https://twitter.com/ylecun/status/1750614681209983231?t=UximKCvpFmMfnQ9u6sc-8Q&s=19 [[2305.13172] Editing Large Language Models: Problems, Methods, and Opportunities](https://arxiv.org/abs/2305.13172) [Online machine learning - Wikipedia](https://en.m.wikipedia.org/wiki/Online_machine_learning) I feel like people give too much privilege, uniqueness to humanlike intelligence and possibly consciousness. They see them as too special. Protoconsciousness Trees might be conscious on much bigger timescales Deepfakes are accelerating Soon massively parralelized robot Bidens will walk around and ask eldery to vote https://twitter.com/AISafetyMemes/status/1751965558377939333?t=b9JSxzRJfLuRoVE1vO5_AA&s=19 maybe in order to have more intelligent AIs we should somehow encode these correlates in neural celluar automata-like systems [Genetic variation, brain, and intelligence differences | Molecular Psychiatry](https://www.nature.com/articles/s41380-021-01027-y) [The Future of AI is Self-Organizing and Self-Assembling (w/ Prof. Sebastian Risi) - YouTube](https://youtu.be/_7xpGve9QEE?si=Z8je-HPKTms-K7hs) Where is the Google galaxysized titan asteroid releasing SuperBard erasing OpenAI ant from existence https://slgero.medium.com/merge-large-language-models-29897aeb1d1a Deconstructive meditation, Relax/Acc [Stopping Doing, Embracing and Edge of Experience | guided meditation - YouTube](https://youtu.be/9H3H_VZda_Q?si=Wxp_4KkYJiolRo3g) Innovator GPT https://chat.openai.com/g/g-JaiQEuHRU-innovator/c/937fa488-cdb9-4a51-9924-9228f5fd7cec "In the beginning there was sensation Humans developed the ability to freeze sensations, then used these frozen pieces to construct a grammar for thought (vasocomputation) This led to words, magical spells that can conjure sensations in both caster and target This led to foom" https://twitter.com/johnsonmxe/status/1751994506478587977?t=bEHIzpD03uGbR7Ikukz4SA&s=19 Attention free Transformer works too well https://twitter.com/AlphaSignalAI/status/1752037592500142210?t=z6t9kZpTXPOxr4kltG2LPw&s=19 Maybe GPT5 will with each request automatically prompt engineer itself and create RAG multiagent system out of itself optimized perfectly for that task. This selforganization could be trained using a cost function too. [Circuits Updates - January 2024](https://transformer-circuits.pub/2024/jan-update/index.html) Chinese llm beating GPT4? https://twitter.com/_akhaliq/status/1752033872982806718 [Object personification in autism: This paper will be very sad if you don't read it - PubMed](https://pubmed.ncbi.nlm.nih.gov/30101594/) Alternatives to money (výměnný obchod, violence, regenerative money, crypto, status, kindness, arbitrary points in arbitrary (VR) games) wellbeing (eq in book, humberman), nootropics Mixture of experts https://twitter.com/sophiamyang/status/1733505991600148892?t=v8UsYMzTyQy1oBRoNVnsJA&s=19 Von Neumann loved noisy environments https://twitter.com/teortaxesTex/status/1751785262676386130?t=huvfbWPJB1WFAzyyJkyJwA&s=19 What can you do to make the world a better place: On foundations: philosophy, science, technology, infrastructure,... Directly: activism, sustainability, climate, AI integration and alignment, social help, love and kindness, diplomacy, democratic governance, donating anything, cleaning,... Indirectly: make money and donating it, investing, spreading the message,... My objective function most of the time is optimize for doing the most good effectively defined as optimizing for increasing collective wellbeing, progress and sustainabiliy (as much in synergy as possible) Explore Andres websites https://twitter.com/algekalipso/status/1675268948458340352?t=ksvy3mkT8Hteva85qJNxUQ&s=19 best open source llm https://twitter.com/bindureddy/status/1752092619373793614 best open source coding llm [META's new OPEN SOURCE Coding AI beats out GPT-4 | Code Llama 70B - YouTube](https://youtu.be/fAFxw16kWGs?si=Vk1TzhpseBrgGX7F) https://www.marktechpost.com/2024/01/27/researchers-from-stanford-and-openai-introduce-meta-prompting-an-effective-scaffolding-technique-designed-to-enhance-the-functionality-of-language-models-in-a-task-agnostic-manner/ [[2401.12954] Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://arxiv.org/abs/2401.12954) [How diffusion models work: the math from scratch | AI Summer](https://theaisummer.com/diffusion-models/) [Oja's rule - Wikipedia](https://en.wikipedia.org/wiki/Oja's_rule) [[2401.14887] The Power of Noise: Redefining Retrieval for RAG Systems](https://arxiv.org/abs/2401.14887) [The $2M Longevity Protocol: Bryan Johnson‚Äôs Biohacking Blueprint | Rich Roll Podcast - YouTube](https://youtu.be/roHeUk7ApUo?si=A5OJqOh5fTN6I0sm) https://pubs.acs.org/doi/full/10.1021/acsengineeringau.3c00058 Multiagent LLM RAG system to automate quantum calculations [Physics-informed deep learning for fringe pattern analysis](https://www.oejournal.org/article/doi/10.29026/oea.2024.230034) [Quantum non-equilibrium - Wikipedia](https://en.wikipedia.org/wiki/Quantum_non-equilibrium) https://twitter.com/Plinz/status/1752571039761186838?t=mufuVbvn69iBYFSmuy6NLg&s=19 save rational enlightenment [[2401.15347] A Comprehensive Survey of Compression Algorithms for Language Models](https://arxiv.org/abs/2401.15347) overview of compression algorithms for LLMs. Covers compression algorithms like pruning, quantization, knowledge distillation, low-rank approximation, parameter sharing, and efficient architecture design. https://twitter.com/skdh/status/1752753777147359512 The causal link has not been established. To the extent that (well powered) studies have found an influence of social media use on the well-being of adolescents it's been minor. [Mastering RAG: How To Architect An Enterprise RAG System - Galileo](https://www.rungalileo.io/blog/mastering-rag-how-to-architect-an-enterprise-rag-system) tree and graph network topology mutations of chain of thought: Kabbalistic Topology Networks are the final frontier in AI design, hypergraph of thought https://twitter.com/eshear/status/1752729486041518311?t=nvPF08xep5LlrNJML4fTaA&s=19 Notes [Mastering RAG: How To Architect An Enterprise RAG System - Galileo](https://www.rungalileo.io/blog/mastering-rag-how-to-architect-an-enterprise-rag-system) , for more details visit the link - metrics [Mastering RAG: 8 Scenarios To Evaluate Before Going To Production - Galileo](https://www.rungalileo.io/blog/mastering-rag-8-scenarios-to-test-before-going-to-production) - chain of thought, thread of thought, chain of note, chain of verification, emotion prompt, expert prompting - To facilitate service resets or reindexing onto an alternative vector database, it's advisable to store embeddings separately. - metadata to vc db, extract and store useful metadata everywhere possible in scraping and parsing - reranker, MMR achieves a balance between relevance and diversity - query rewriter. - store embedding results in sql database too if i wanna change vector database and not pay for embedding costs, and for backup - store chat history - fix issues by storing user feedback (thumbs up/thumbs down, rate x/10, or note) - track costs - subqueries, generate similar queries - set all RAG settings as parameters for metrics and easier customization - end-to-end training of components to make them more specialized or flexible - user authentication - user personalization, customization - each user gets his metadata in chat history database [Building Multi-Tenancy RAG System with LlamaIndex — LlamaIndex, Data Framework for LLM Applications](https://blog.llamaindex.ai/building-multi-tenancy-rag-system-with-llamaindex-0d6ab4e0c44b?gi=ede80bd6e806) - retrieval metrics like Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) - find optimal encoder - clustering of vecdb - query routing (with multiple vecdbs?) - smaller embeddings to vecdb for smaller cost and bigger speed? - if the chunks are too long, then the answers include generated noise. You can exploit summarisation techniques to reduce noise, text size, encoding cost and storage cost. [5 Levels Of LLM Summarizing: Novice to Expert - YouTube](https://www.youtube.com/watch?v=qaPMdcCqtWk) - vector db comparision [Vector DB Comparison](https://vdbs.superlinked.com/) Optimizing for recall (percentage of relevant results) versus latency (time to return results) is a trade-off in vector databases. Conduct benchmark studies on your data and queries to make an informed decision. Techniques like memory-mapped files allow scaling vector storage without compromising search speed - Hybrid Search: To address the limitations of vector search, hybrid search combines two methodologies: dense and sparse. [Getting Started with Hybrid Search | Pinecone](https://www.pinecone.io/learn/hybrid-search-intro/) - embedding/model etc. finetuned/optimized for my dataset? - Custom-filtered search, like Weaviate, combines pre-filtering with effective semantic search - Recent research has shown that LLMs can be easily distracted by irrelevant context and having a lot of context (topK retrieved docs) can lead to missing out of certain context due to the attention patterns of LLMs. Therefore it is crucial to improve retrieval with relevant and diverse documents. Let's look at some of the proven techniques for improving retrieval. - Hypothetical document embeddings - cashing of prompts and responces in a database to minimize costs [GitHub - zilliztech/GPTCache: Semantic cache for LLMs. Fully integrated with LangChain and llama_index.](https://github.com/zilliztech/GPTCache) offers valuable metrics such as cache hit ratio, latency, and recall, which provide insights into the cache's performance - autocut is designed to limit the number of search results returned by detecting groups of objects with close scores by analyzing the scores of the search results and identifying significant jumps in these values, which can indicate a transition from highly relevant to less relevant results - Recursive retrieval, aka the small-to-big retrieval technique, embeds smaller chunks for retrieval while returning larger parent context for the language model's synthesis. Smaller text chunks contribute to more accurate retrieval, while larger chunks provide richer contextual information for the language model. [Redirecting...](https://docs.llamaindex.ai/en/stable/examples/query_engine/pdf_tables/recursive_retriever.html) - Sentence window retrieval process fetches a single sentence and returns a window of text around that particular sentence. [Redirecting...](https://docs.llamaindex.ai/en/latest/examples/node_postprocessor/MetadataReplacementDemo.html) [5 Techniques for Detecting LLM Hallucinations - Galileo](https://www.rungalileo.io/blog/5-techniques-for-detecting-llm-hallucinations) - [Observe | Galileo](https://docs.rungalileo.io/galileo/llm-studio/llm-monitor) [Could Humans Survive the Dinosaur-Killing Asteroid? Featuring @LEMMiNO - YouTube](https://www.youtube.com/watch?v=FiO7Vyq8Rn8) [Tyler Cowen - Hayek, Keynes, & Smith on AI, Animal Spirits, Anarchy, & Growth - YouTube](https://www.youtube.com/watch?v=EY2nbAVZB-k) https://twitter.com/IntuitMachine/status/1752665006137643511 [[2401.14423] Prompt Design and Engineering: Introduction and Advanced Methods](https://arxiv.org/abs/2401.14423) prompt engineering OpenGPTs https://twitter.com/LangChainAI/status/1752737053262152162?t=rDgeBndvCKgDQDhIJfdqfg&s=19 Finetuning resources https://www.linkedin.com/posts/ghimiresunil_naturallanguageprocessing-transformer-genai-ugcPost-7158485069735571459-6Ttc?utm_source=share&utm_medium=member_android [GitHub - mlabonne/llm-course: Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.](https://github.com/mlabonne/llm-course) [[2401.17263] Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://arxiv.org/abs/2401.17263) [Plants Find Light Using Gaps Between Their Cells | Quanta Magazine](https://www.quantamagazine.org/plants-find-light-using-gaps-between-their-cells-20240131/) [Josephson effect - Wikipedia](https://en.wikipedia.org/wiki/Josephson_effect) [Macroscopic quantum phenomena - Wikipedia](https://en.wikipedia.org/wiki/Macroscopic_quantum_phenomena) Physics of measuring qubits in quantum computers [ChatGPT](https://chat.openai.com/share/c09b0ad3-f5f7-463c-a059-bd0dd2db99cf) Empirically verifying quantum field theory [ChatGPT](https://chat.openai.com/share/27770225-7136-4566-bf06-8877d2b0f8eb) Empirically verifying general relativity [ChatGPT](https://chat.openai.com/share/e9d517ec-9242-4b8f-8337-764ff666dbfc) Mathematics of resonance [ChatGPT](https://chat.openai.com/share/08bf58d6-891e-458f-8334-ed4e14da542a) Empirically verifying compitational neuroscience model [ChatGPT](https://chat.openai.com/share/9473c620-b78b-47e9-8950-6cdbba7ebbae) Brain networks [ChatGPT](https://chat.openai.com/share/635a6ebc-a7e7-44eb-83ea-5c834dbd2954) How psychedelics work in the brain [ChatGPT](https://chat.openai.com/share/0729dc62-13b0-4c19-a714-102299146da4) [[2401.14295] Demystifying Chains, Trees, and Graphs of Thoughts](https://arxiv.org/abs/2401.14295) The brain takes raw physical sensory data as a prompt and outputs aka predicts complex internal world simulation compressing actual outside physical dynamics using senses and connects it with internal world and a self model with actions on both internal and external worlds. Psychedelics and deconstructive meditation to certain extend increase entropy, dissolve, strenghten, restructure all these (useful) complex information processing circuits with their complex interconnected hetearchical networks of features, priors, representations, distinctions and boudaries. https://twitter.com/burny_tech/status/1752952466600026513?t=0_TcwGJ79rJaVK9Esxt1tw&s=19 [Nootropics · Gwern.net](https://gwern.net/nootropic/nootropics#caffeine) [Stimulant tolerance, or, the tears of things](https://www.gleech.org/stims) [The Big Misconception About Electricity - YouTube](https://www.youtube.com/watch?v=bHIhgxav9LY) [How Electricity Actually Works - YouTube](https://www.youtube.com/watch?v=oI_X2cMHNe0) [Better Call GPT: LLMs 300x cheaper than human lawyers (and almost as good!) - Knee of the Curve! - YouTube](https://www.youtube.com/watch?v=GXq0AYXAlzQ) [Ray Kurzweil Q&A - The Singularity, Human-Machine Integration & AI | EP #83 - YouTube](https://www.youtube.com/watch?v=Iu7zOOofcdg) Brain cells in a lab dish learn to play Pong — and offer a window onto intelligence This population of neurons was taught to play Pong by electrical stimulation reinforcement. The strategy was based on the Free Energy Principle, which states that brain cells want to be able to predict what's going on in their environment. So they would choose predictable stimulation over unpredictable stimulation. "If they hit the ball, we gave them something predictable, when they missed it, they got something that was totally unpredictable." Human brain cells seemed to achieve a slightly higher level of play than mouse brain cells. [Mouse and human brain cells in a lab dish learn to play video game Pong : Shots - Health News : NPR](https://www.npr.org/sections/health-shots/2022/10/14/1128875298/brain-cells-neurons-learn-video-game-pong) https://www.cell.com/neuron/fulltext/S0896-6273(22)00806-6?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627322008066%3Fshowall%3Dtrue "To plan a nanobot Map out the molecules to span the nanobot Add on the motors used to man the nanobot Signals to talk with a commander Software and logic's grammar Programmed in nanobot Let's see You've animated these lil' gadgets? Damn! Next let's see you handle informatics fam (Nano-informatics) 1-0 the structure of a cache or RAM Switches are the way to data save and plan (flip it on) Flip it chemically Photochemically Or electrically Add a flippable knee (then what?) Make a little more deep The decisional tree (data) If you need a calculator how be Logic gates, like "a, not b" Input's a cation, as key (that's key) And you get a photon as a reading So plan your nanobot To be in part an info scanner nanobot Reading and writing coded data that allot Freedom to do as you program it A Turing automatic Full gamut nanobot" 🎶 🤩 [Nanobot (Havana Parody) | A Capella Science ft. Dorothy Andrusiak - YouTube](https://www.youtube.com/watch?v=ObvxPSQNMGc) [Tiny robots made from human cells heal damaged tissue](https://www.nature.com/articles/d41586-023-03777-x) turing completeness from basic logic gates allowing to run arbitrary programs is everywhere, even magic the gathering is also turing complete 😆[It takes 8,400,000,000,000 years to use a Magic: The Gathering computer - YouTube](https://www.youtube.com/watch?v=uDCj-QOp5gE) i think “inject biological matter with digital code” was in some sense solved by all the attempts to create logic gates from these materials, and when you have logic gates, you have turing completeness, and therefore you can run in theory any function (code) on it [Scientists create modular logic gates from bacteria and DNA « the Kurzweil Library + collections](https://www.thekurzweillibrary.com/scientists-create-computing-building-blocks-from-bacteria-and-dna) [Engineering modular and orthogonal genetic logic gates for robust digital-like synthetic biology | Nature Communications](https://www.nature.com/articles/ncomms1516)