It would be great to have some mathematical theory that can determine which deep learning system or AI system in general is the best for given task Apparently geometric deep learning tries to go in that direction and already created some new architectures Or categorical deep learning, but I heard criticisms on the practical utility of that one, as it is with anything related to category theory, but I still love category theory. https://www.youtube.com/watch?v=bIZB1hIJ4u8 https://geometricdeeplearning.com/ GDL and Categorical DL can be best understood as a principled way of baking in a priori structure/symmetries from your problem/data into your model architecture. e.g. "I want a CNN that is invariant/equivariant under a certain group of transformations, I want an architecture adapted to data structured as binary trees, ...". You trade off some generality for stronger guarantees and possibly fewer parameters Math of inductive biases Yeah, and that's apparently how this architecture was born https://deepmind.google/discover/blog/tacticai-ai-assistant-for-football-tactics/ "TacticAI successfully predicts corner kick play by applying a geometric deep learning approach. First, we directly model the implicit relations between players by representing corner kick setups as graphs, in which nodes represent players (with features like position, velocity, height, etc.) and edges represent relations between them. Then, we exploit an approximate symmetry of the football pitch. Our geometric architecture is a variant of the Group Equivariant Convolutional Network that generates all four possible reflections of a given situation (original, H-flipped, V-flipped, HV-flipped) and forces our predictions for receivers and shot attempts to be identical across all four of them. This approach reduces the search space of possible functions our neural network can represent to ones that respect the reflection symmetry — and yields more generalizable models, with less training data."