Author Archives: Philip Versteeg

Talk by Joan Bruna (NYU)

You are all cordially invited to the AMLab seminar talk this Tuesday October 11 at 16:00 in C3.163, where Joan Bruna from the Courant Institute at New York University  will give a talk titled “Addressing Computational and Statistical Gaps with Deep Neural Networks”. Afterwards there are the usual drinks and snacks!

Abstract: Many modern statistical questions are plagued with asymptotic regimes that separate our current theoretical understanding with what is possible given finite computational and sample resources. Important examples of such gaps appear in sparse inference, high-dimensional density estimation and non-convex optimization. In the former, proximal splitting algorithms efficiently solve the l1-relaxed sparse coding problem, but their performance is typically evaluated in terms of asymptotic convergence rates. In unsupervised high-dimensional learning, a major challenge is how to appropriately combine prior knowledge in order to beat the curse of dimensionality. Finally, the prevailing dichotomy between convex and non-convex optimization is not adapted to describe the diversity of optimization scenarios faced as soon as convexity fails.

In this talk we will illustrate how Deep architectures can be used in order to attack such gaps. We will first see how a neural network sparse coding model (LISTA, Gregor & LeCun’10) can be analyzed in terms of a particular matrix factorization of the dictionary, which leverages diagonalisation with invariance of the l1 ball, revealing a phase transition that is consistent with numerical experiments. We will then discuss image and texture generative modeling and super-resolution, a prime example of high-dimensional inverse problem. In that setting, we will explain how multi-scale convolutional neural networks are equipped to beat the curse of dimensionality and provide stable estimation of high frequency information. Finally, we will discuss recent research in which we explore to what extent the non-convexity of the loss surface arising in deep learning problems is hurting gradient descent algorithms, by efficiently estimating the number of basins of attractions.

Slides

Talk by Riaan Zoetmulder

You are all cordially invited to an AMLab seminar during the summer period at Tuesday August 23 at 16:00 in C3.163, where Riaan Zoetmulder will give a talk titled “Deep Causal Inference”. Afterwards there are the usual drinks and snacks!

AbstractDetermining causality is important for many fields of science. A variety of algorithms have been developed that are capable of discerning what the direction of causality is, given the data. Recent developments in deep learning however have shown that artificial deep neural networks have excellent performance on a variety of classification problems. This paper therefore seeks to ascertain whether causality can be determined using a deep learning approach. We have found that this is possible in two different ways; one can hand design features and train a deep neural network on them. Or one can design the deep neural network to detect features itself and learn how to classify accordingly. 

Talk by Christos Louizos

You are all cordially invited to the AMLab seminar at Tuesday July 12 at 16:00 in C3.163, where Christos Louizos will give a talk titled “Bayesian Deep Learning and Uncertainty”. Afterwards there are the usual drinks and snacks!

Abstract: In the first part of this talk we will show how we can extend upon recent advances in variational inference for Bayesian neural networks with a simple idea. Instead of the relative limited fully factorized Gaussian assumption in the posterior for the parameters of each layer we will instead assume that each weight matrix is distributed as a Matrix Gaussian. This parametrisation has several potential advantages; it introduces correlations among the weights, therefore increases the flexibility of the posterior, reduces the amount of variational parameters and furthermore allows for a (finite-rank) Gaussian Process interpretation for each layer and a Deep Gaussian Process interpretation of the entire network. We will show that this model is more effective than other Bayesian approaches in a regression and a classification task.

In the second part of this talk we will explore the predictive uncertainties that various Bayesian neural network approaches provide in classification tasks. Surprisingly we will see that none of the methods seem to perform well in inputs that are not from the data distribution, and as a result provide erroneously certain predictions. Interestingly this seems to be problem with the model class as even frequentist methods suffer from the same problem. We conclude with open questions and possible directions of research in order to tackle this intriguing problem.

Talk by Peter O’Connor

You are all cordially invited to the AMLab seminar at Tuesday July 5 at 16:00 in C3.163, where Peter O’Connor will give a talk titled “Deep Spiking Networks”. Afterwards there are the usual drinks and snacks!

AbstractWe introduce the Spiking Multi-Layer Perceptron (SMLP). The SMLP is a spiking version of a conventional Multi-Layer Perceptron with rectified-linear units. Our architecture is eventbased, meaning that neurons in the network communicate by sending “events” to downstream neurons, and that the state of each neuron is only updated when it receives an event. We show that the SMLP behaves identically, during both prediction and training, to a conventional deep network of rectified-linear units in the limiting case where we run the spiking network for a long time. We apply this architecture to a conventional classification problem (MNIST) and achieve performance very close to that of a conventional MLP with the same architecture. Our network is a natural architecture for learning based on streaming event-based data, and has potential applications in robotic systems systems, which require low power and low response latency.

Talk by Tameem Adel

You are all cordially invited to the AMLab seminar at Tuesday June 28 at 16:00 in C3.163, where Tameem Adel will give a talk titled “Collapsed Variational Inference for Sum-Product Networks”. Afterwards there are the usual drinks!

Abstract: Sum-Product Networks (SPNs) are probabilistic inference machines that admit exact inference in linear time in the size of the network. Existing parameter learning approaches for SPNs are largely based on the maximum likelihood principle and hence are subject to overfitting compared to more Bayesian approaches. Exact Bayesian posterior inference for SPNs is computationally intractable. We recently proposed a novel deterministic collapsed variational inference algorithm for SPNs that is computationally efficient, easy to implement and at the same time allows us to incorporate prior information into the optimization formulation. Experiments show a significant improvement in accuracy compared with a maximum likelihood based approach.

Talk by Matthias Reisser

You are all cordially invited to the AMLab seminar at Tuesday June 21 at 16:00 in C3.163, where Matthias Reisser will give a talk titled “Distributed Bayesian Deep Learning”. Afterwards there are the usual drinks and snacks!

Abstract: I would like to give you an overview on what my PhD topic going to be about, as well as present my first project along with initial results: Although deep learning becomes more and more data efficient, it is still true that with more data, more complex models with better generalization capabilities can be trained. More data and bigger models require more computation, resulting in longer training times and slow experiment cycles. One valid approach to speed up computations is by distributing them across machines. At the same time, in the truly huge data regime, as well as for privacy reasons, data may not be accessible from any machine, requiring distributed computations. In a first project, we look at variational inference and a principled approach to distributed training of one joint model. I am looking forward to your opinion and will be grateful for any feedback. Although I am a QUVA member, every UVA-employee is welcome to attend, independent on whether you have signed the QUVA NDA.

Talk by Thijs van Ommen

You are all cordially invited to the AMLab seminar at Tuesday June 7 at 16:00 in C3.163, where Thijs van Ommen will give a talk titled “Robust probability updating”. Afterwards there are the usual drinks and snacks!

AbstractIn the well-known Monty Hall problem, a car is hidden behind one of three doors, and the contestant wants to compute the probabilities of where the prize is hidden given partial information (a ‘message’) from the quizmaster. Most analyses of this problem assume that the quizmaster uses a fair coin flip to decide what message to give, whenever he has a choice. We don’t make this assumption, but instead use game theory to find a strategy for the contestant that works well against any strategy the quizmaster might use. With this approach, we can also deal with a large generalization of the problem: to any finite number of doors, with any initial distribution of the winning door, and with an arbitrary set of messages (subsets of doors) from which the quizmaster can choose. In Bayesian terms, this translates to computing a posterior distribution without knowing the full joint distribution. It turns out that in general, the optimal strategies for both players in this game depend on the loss function used to evaluate the contestant’s posterior distribution. However, for certain classes of message sets, there is a single optimal posterior that does not depend on the loss function, so that we obtain an objective and general answer to how one should update probabilities in the light of new information.

Slides

Talk by Matthijs Snel

You are all cordially invited to the AMLab seminar at Tuesday May 31 at 16:00 in C3.163, where Matthijs Snel from Optiver will give a talk titled “An introduction to market making and data science at Optiver”. Afterwards there are the usual drinks and snacks!

AbstractOptiver is an electronic market maker with significant presence on equity and derivatives exchanges around the world. Our automated trading strategies operate as semi-autonomous agents, processing information and making multiple decisions in the blink of an eye. In this talk, I will explain some basic market making concepts, supported by real-world examples of market microstructure. I will also provide an overview of what kind of data and challenges our strategies and machine learning applications deal with.

Talk by Ted Meeds

You are all cordially invited to the AMLab seminar at Tuesday May 24 at 16:00 in C4.174, where Ted Meeds will give a talk titled “Likelihood-free Inference by Controlling Simulator Noise”. Afterwards there are the usual drinks and snacks!

AbstractLikelihood-free inference, or approximate Bayesian computation (ABC), is a general framework for performing Bayesian inference in simulation-based science.  In this talk I will discuss two new approaches to likelihood-free inference that involve explicit control over a simulation’s randomness.  By re-writing simulation code with two sets of arguments, the simulation parameters and its random numbers, many algorithmic options open up.  The first approach, called Optimisation Monte Carlo, in an algorithm that efficiently and independently samples parameters from the posterior by first sampling a set of random numbers from a prior distribution, then running an optimisation algorithm—with fixed random numbers—to match simulation statistics with observed statistics.   The second approach is recent and ongoing research on a variational ABC algorithm that has been written in an auto-differentiation language allowing for the gradients of the variational parameters to be computed through the simulation code itself.  

Talk by Karen Ullrich

You are all cordially invited to the AMLab colloquium Tuesday May 17 at 16:00 in C3.163, where Karen Ullrich will give a talk titled “Combining generative models and deep learning”. Afterwards there are the usual drinks and snacks!

Abstract: Deep learners prove to perform well on very large datasets. For small datasets, however, one has to come up with new methods to model and train. My current project is in line with this thought. By combining a simple deep learner with a state space model we hope to perform well on visual odometry.