Dense Associative Memories (also known as Modern Hopfield Networks) are recurrent neural networks with fixed point attractor states that are described by an energy function. In contrast to conventional Hopfield Networks, which were popular in the 1980s, their modern versions have a very large memory storage capacity, which makes them appealing tools for many problems in AI and neuroscience. In this talk, I will provide an intuitive understanding and a mathematical framework for this class of models, and will give examples of problems in AI that can be tackled using these new ideas. Specifically, I will explore the relationship between Dense Associative Memories and two prominent generative AI models: transformers and diffusion models. I will present a neural network, called the Energy Transformer, which unifies energy-based modeling, associative memories, and transformers in a single architecture. Furthermore, I will discuss an emerging perspective that views diffusion models as Dense Associative Memories operating above the critical memory storage capacity. This insight opens up interesting avenues for leveraging associative memory theory to analyze the memorization-generalization transition in diffusion models, revealing intriguing possibilities for future research.

Bio: Dmitry Krotov is a physicist working on neural networks and machine learning. He is a member of the research staff at the MIT-IBM Watson AI Lab and IBM Research in Cambridge, MA. Prior to this, he was a member of the Institute for Advanced Study in Princeton. He received a PhD in Physics from Princeton University in 2014. His research aims to integrate computational ideas from neuroscience and physics into modern AI systems. His recent work focuses on high-memory storage capacity networks known as Dense Associative Memories.