In the past decade, the revival of deep neural networks has led to dramatic success in numerous applications ranging from computer vision, to natural language processing, to scientific discovery and beyond. Nevertheless, the practice of deep networks has been shrouded with mystery as our theoretical understanding for the success of deep learning remains elusive. In this talk, we will focus on the representations learned by deep neural networks. For example, Neural collapse is an intriguing empirical phenomenon that persists across different neural network architectures and a variety of standard datasets. This phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. We will provide a geometric analysis for understanding why this happens on a simplified unconstrained feature model. We will also exploit these findings to improve training efficiency: we can set the feature dimension equal to the number of classes and fix the last-layer classifier to be a Simplex ETF for network training, reducing memory cost by over 20% on ResNet18 without sacrificing the generalization performance. Co-sponsored by: Christopher Reardon, Eric Ericson Speaker(s): Dr. Zhihui Zhu , 2155 East Wesley Avenue, Denver, Colorado, United States, 80208, Virtual: https://events.vtools.ieee.org/m/295539