We have a guest speaker Daniele Musso from Universidad de Santiago de Compostela and Daniele will give a talk at our Lab. You are all cordially invited to the AMLab Seminar on April 8th (Thursday) at 4:00 p.m. CEST on Zoom. And then Daniele will present a recent work titled “Partial local entropy and anisotropy in deep weight spaces”.
Title: Partial local entropy and anisotropy in deep weight spaces
Abstract: We refine a recently-proposed class of local entropic loss functions by restricting the smoothening regularization to only a subset of weights. The new loss functions are referred to as partial local entropies. They can adapt to the weight-space anisotropy, thus outperforming their isotropic counterparts. We support the theoretical analysis with experiments on image classification tasks performed with multi-layer, fully-connected neural networks. The present study suggests how to better exploit the anisotropic nature of deep landscapes and provides direct probes of the shape of the wide flat minima encountered by stochastic gradient descent algorithms. As a by-product, we observe an asymptotic dynamical regime at late training times where the temperature of all the layers obeys a common cooling behavior.
Paper Link: https://arxiv.org/pdf/2007.09091.pdf
To gain more insight into partial local entropy and anisotropy, feel free to join and discuss it!
See you there!