initial theories of DL

motivation

2019 The unreasonable effectiveness of deep learning in artificial intelligence Terrence J. Sejnowski

RG and Deep Learning

The new work, completed by Pankaj Mehta of Boston University and David Schwab of Northwestern University, demonstrates that a statistical technique called “renormalization,” which allows physicists to accurately describe systems without knowing the exact state of all their component parts

• Learning hierarchical category structure in deep neural networks

• Stephane Mallat awarded Milner RS Prize 2023 for wavelet+RG view

Information Bottleneck METHOD

• 2015 DL and Information Bottleneck

Information Bottleneck sheds insight into AI as well as human brain

• Learnability can be undecidable

We describe simple scenarios where learnability cannot be proved nor refuted using the standard axioms of mathematics. The main idea is to prove an equivalence between learnability and compression.