We have a guest speaker Laurence Aitchison from the University of Bristol and Laurence will present his research works at our Lab. You are all cordially invited to the AMLab Seminar on June 10th (Thursday) at 4:00 p.m. CEST on Zoom. And then Laurence will give a talk titled “A statistical theory of cold-posteriors, semi-supervised learning and out-of-distribution detection”.
Title: A statistical theory of cold-posteriors, semi-supervised learning and out-of-distribution detection
Abstract: Image classification datasets such as CIFAR-10 and ImageNet are carefully curated to exclude ambiguous or difficult to classify images. Remarkably, this curation process can be used to understand three very different areas in deep learning: semi-supervised learning, out-of-distribution detection and the cold posterior effect. We develop a generative model of dataset curation in which multiple annotators label every image, with the image being included in the dataset only if all the annotators agree. If any of the annotators disagree, the image is excluded. This directly explains the “cold posterior effect”, where artificially reducing uncertainty in the Bayesian posterior over neural network weights gives better test performance (ICLR 2021; arxiv.org/abs/2008.05912). In addition, if we marginalise over the class-label, we get a semi-supervised learning objective mirroring entropy minimization and pseudo-labelling, which allows us to use unlabelled points to improve the performance of a classifier (very early version: arxiv.org/abs/2008.05913). Finally, the curation process itself gives us insight into out-of-distribution detection, where we explicitly detect test points that are far from the training data, as our predictions might be inaccurate in those regions (arxiv.org/abs/2102.12959).
Paper Link: arxiv.org/abs/2008.05912 (ICLR 2021); arxiv.org/abs/2008.05913; arxiv.org/abs/2102.12959
To gain more insight into Bayesian deep learning and out-of-distribution detection, feel free to join and discuss it!