Category Archives: Uncategorized

DanieleMusso’s Talk

Hi everyone,

We have a guest speaker Daniele Musso from Universidad de Santiago de Compostela and Daniele will give a talk at our Lab. You are all cordially invited to the AMLab Seminar on April 8th (Thursday) at 4:00 p.m. CEST on Zoom. And then Daniele will present a recent work titled “Partial local entropy and anisotropy in deep weight spaces”.

Title: Partial local entropy and anisotropy in deep weight spaces

Abstract: We refine a recently-proposed class of local entropic loss functions by restricting the smoothening regularization to only a subset of weights. The new loss functions are referred to as partial local entropies. They can adapt to the weight-space anisotropy, thus outperforming their isotropic counterparts. We support the theoretical analysis with experiments on image classification tasks performed with multi-layer, fully-connected neural networks. The present study suggests how to better exploit the anisotropic nature of deep landscapes and provides direct probes of the shape of the wide flat minima encountered by stochastic gradient descent algorithms. As a by-product, we observe an asymptotic dynamical regime at late training times where the temperature of all the layers obeys a common cooling behavior.

Paper Link: https://arxiv.org/pdf/2007.09091.pdf 

To gain more insight into partial local entropy and anisotropy, feel free to join and discuss it!

See you there!

Sindy Löwe’s Talk

Hi everyone,

Our colleague Sindy Löwe will present her recent work at our AMLab Seminar and you are all cordially invited to this thrilling session on March 4th (Thursday) at 4:00 p.m. CET on Zoom. And then Sindy will give a talk titled “Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data”.

Title: Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data

Abstract: Standard causal discovery methods must fit a new model whenever they encounter samples from a new underlying causal graph. However, these samples often share relevant information – for instance, the dynamics describing the effects of causal relations – which is lost when following this approach. We propose Amortized Causal Discovery, a novel framework that leverages such shared dynamics to learn to infer causal relations from time-series data. This enables us to train a single, amortized model that infers causal relations across samples with different underlying causal graphs, and thus makes use of the information that is shared. We demonstrate experimentally that this approach, implemented as a variational model, leads to significant improvements in causal discovery performance, and show how it can be extended to perform well under hidden confounding.

Paper Link: https://arxiv.org/abs/2006.10833

To gain more insight into Causal Discovery, feel free to join and discuss it! See you there 😊. See you there!

Michaël Defferrard’s Talk

Hi everyone! We have a guest speaker Michaël Defferrard from École Polytechnique Fédérale de Lausanne (EPFL) and you are all cordially invited to the AMLab Seminar on February 25th (Thursday) at 4:00 p.m. CET on Zoom. And then Michaël will give a talk titled “Learning from graphs: a spectral perspective”. Michaël is an inspiring researcher, who has done a lot of interesting works on graph deep learning and you can find additional information from his website. The following is the information on this talk.

Title: Learning from graphs: a spectral perspective

Abstract: The architecture of a neural network constrains the space of functions it can implement. Equivariance is one such constraint—enabling weight sharing and guaranteeing generalization. But symmetries alone might not be enough: for example, social networks, finite grids, and sampled spheres have few automorphisms. I will discuss how spectral graph theory yields vertex representations and a generalized convolution that shares weights beyond symmetries.

To gain more insight into Graph Deep Learning, feel free to join and discuss it! See you there 🙂 !

Samuele Tosatto’s Talk

Hi everyone! We have a guest speaker Samuele Tosatto from TU Darmstadt and you are all cordially invited to the AMLab Seminar on February 18th (Thursday) at 4:00 p.m. CET on Zoom. And then Samuele will give a talk titled “Movement Representation and Off-Policy Reinforcement Learning for Robotic Manipulation“.

Title: Movement Representation and Off-Policy Reinforcement Learning for Robotic Manipulation

Abstract: Machine learning, and more particularly, reinforcement learning, holds the promise of making robots more adaptable to new tasks and situations.
However, the general sample inefficiency and lack of safety guarantees make reinforcement learning hard to apply directly to robotic systems.
To mitigate the aforementioned issues, we focus on two aspects of the learning scheme.
The first aspect regards robotic movements. Robotic movements are crucial in manipulation tasks. The usual parametrization of robotic movements allows high expressivity but is usually inefficient, as it covers movements not relevant to the task.  We analyze how to focus the representation of only those movements relevant to the considered task. This novel representation has the effect of ameliorating the sample efficiency and providing higher safety.
The low quality of a gradient estimator in reinforcement learning can cause another source of inefficiency.  On-policy gradient estimators are usually easy to obtain, but they are, due to their nature, sample inefficient. In contrast, state-of-the-art off-policy solutions are challenging to compute. These estimators are typically divided into importance-sampling and semi-gradient approaches. The first suffers from high variance, while the second suffers from high bias. In this talk, we show a third way to compute off-policy gradients that exhibit a fair bias/variance tradeoff using a closed-form solution of a proposed non-parametric Bellman equation. Using this estimator results in a particular high sample efficiency. Our algorithm can be applied offline on human-demonstrated data, providing a safe scheme that avoids dangerous interaction with the real robot.

To gain more insight into Reinforcement Learning and Robotics, feel free to join and discuss it! See you there 🙂 !

Yuge Shi’s Talk

Hi everyone! Happy New Year and our thrilling AMLab Seminar will come back this Thursday! We have an external speaker Yuge Shi from Oxford University and you are all cordially invited to the AMLab Seminar on January 14th at 4:00 p.m. CET on Zoom, where‪ Yuge will give a talk titled “Multimodal Learning with Deep Generative Models“.

Title : Multimodal Learning with Deep Generative Models

Abstract: In this talk, I will present my two works on multi-modal representation learning using deep generative models. In these works, we mainly focus on multi-modal scenarios that naturally occur in the real world that depict common concepts, such as image-caption, photo-sketch, video-audio etc. In the first work, we propose to use a mixture-of-expert posterior in VAE to achieve balanced representation learning of different modalities; by doing so, the model is able to leverage the commonality between modalities to learn more robust representations and achieve better generative performance. In addition, we also proposed 4 criteria (with evaluation metrics) that multi-modal deep generative models should satisfy; in the second work, we designed a contrastive-ELBO objective for multi-modal VAEs that greatly reduced the amount of paired data needed to train such models. We show that our objective is effective on multiple SOTA multi-modal VAEs and on different datasets, and showed that only 20% of data is needed to achieve similar performance to a model trained on the original objective.

To gain more deep insights into Multi-modal Learning, feel free to join and discuss it! See you there 🙂 !

Maximilian Ilse’s Talk

Hi everyone, you are all cordially invited to the AMLab Seminar on December 17th at 4:00 p.m. CET on Zoom, where‪ Maximilian Ilse will give a talk titled ” Selecting Data Augmentation for Simulating Interventions “.

Title : Selecting Data Augmentation for Simulating Interventions

Abstract: Machine learning models trained with purely observational data and the principle of empirical risk minimization \citep{vapnik_principles_1992} can fail to generalize to unseen domains. In this paper, we focus on the case where the problem arises through spurious correlation between the observed domains and the actual task labels. We find that many domain generalization methods do not explicitly take this spurious correlation into account. Instead, especially in more application-oriented research areas like medical imaging or robotics, data augmentation techniques that are based on heuristics are used to learn domain invariant features. To bridge the gap between theory and practice, we develop a causal perspective on the problem of domain generalization. We argue that causal concepts can be used to explain the success of data augmentation by describing how they can weaken the spurious correlation between the observed domains and the task labels. We demonstrate that data augmentation can serve as a tool for simulating interventional data. We use these theoretical insights to derive a simple algorithm that is able to select data augmentation techniques that will lead to better domain generalization.

Paper Link: https://arxiv.org/pdf/2005.01856.pdf

To gain more deep insights into Data Augmentation and Causal Inference, feel free to join and discuss it 🙂 !

Javier and James’ Talk

Hi everyone, we have guest speakers to present their works this Thursday. You are all cordially invited to the AMLab Seminar on December 10th at 4:00 p.m. CET on Zoom, where‪ Javier Antorán and James Allingham will give a talk titled ” Depth Uncertainty in Neural Networks “.

Title : Depth Uncertainty in Neural Networks

Abstract : Existing methods for estimating uncertainty in deep learning tend to require multiple forward passes, making them unsuitable for applications where computational resources are limited. To solve this, we perform probabilistic reasoning over the depth of neural networks. Different depths correspond to subnetworks which share weights and whose predictions are combined via marginalisation, yielding model uncertainty. By exploiting the sequential structure of feed-forward networks, we are able to both evaluate our training objective and make predictions with a single forward pass. We validate our approach on real-world regression and image classification tasks. Our approach provides uncertainty calibration, robustness to dataset shift, and accuracies competitive with more computationally expensive baselines.

Paper Link: https://arxiv.org/pdf/2006.08437.pdf

To gain more deep insights into Uncertainty in Deep Neural Networks, feel free to join and discuss it! See you there!

Miles Cranmer’s Talk

Hi, everyone! We have a guest speaker for our Seminar, and you are all cordially invited to the AMLab Seminar on Thursday 3rd December at 16:00 CET on Zoom, where‪ Miles Cranmer will give a talk titled “LAGRANGIAN NEURAL NETWORKS”.

Title : LAGRANGIAN NEURAL NETWORKS

Abstract : Accurate models of the world are built upon notions of its underlying symmetries. In physics, these symmetries correspond to conservation laws, such as for energy and momentum. Yet even though neural network models see increasing use in the physical sciences, they struggle to learn these symmetries. In this paper, we propose Lagrangian Neural Networks (LNNs), which can parameterize arbitrary lagrangian using neural networks. In contrast to models that learn Hamiltonians, LNNs do not require canonical coordinates and thus perform well in situations where canonical momenta are unknown or difficult to compute. Unlike previous approaches, our method does not restrict the functional form of learned energies and will produce energy-conserving models for a variety of tasks. We test our approach on a double pendulum and a relativistic particle, demonstrating energy conservation where a baseline approach incurs dissipation and modeling relativity without canonical coordinates where a Hamiltonianapproach fails. Finally, we show how this model can be applied to graphs and continuous systems using a Lagrangian Graph Network, and demonstrate it on the1D wave equation.

Paper Link: https://arxiv.org/pdf/2003.04630.pdf

To gain more deep insights into LAGRANGIAN NEURAL NETWORKS, feel free to join and discuss it! See you there!

David Duvenaud’s Talk

Hi, everyone! We have a guest speaker for our Seminar, and you are all cordially invited to the AMLab Seminar on Tuesday 24th November at 16:00 CET on Zoom, where‪ David Duvenaud will give a talk titled “Latent Stochastic Differential Equations for Irregularly-Sampled Time Series”.

Title: Latent Stochastic Differential Equations for Irregularly-Sampled Time Series

Abstract: Much real-world data is sampled at irregular intervals, but most time series models require regularly-sampled data. Continuous-time models address this problem, but until now only deterministic (ODE) models or linear-Gaussian models were efficiently trainable with millions of parameters. We construct a scalable algorithm for computing gradients of samples from stochastic differential equations (SDEs), and for gradient-based stochastic variational inference in function space, all with the use of adaptive black-box SDE solvers. This allows us to fit a new family of richly-parameterized distributions over time series. We apply latent SDEs to motion capture data, and to construct infinitely-deep Bayesian neural networks.

The technical details are in this paper: https://arxiv.org/abs/2001.01328 And the code is available at: https://github.com/google-research/torchsde

To gain more deep insights into neural stochastic differential equations, feel free to join and discuss it! See you there!

Bio: David Duvenaud is an assistant professor in computer science at the University of Toronto. His research focuses on continuous-time models, latent-variable models, and deep learning. His postdoc was done at Harvard University, and his Ph.D. at the University of Cambridge. David also co-founded Invenia, an energy forecasting and trading company.

Leon Lang’s Talk

Hi everyone, You are all cordially invited to the AMLab Seminar on Thursday 12nd Nov. at 16:00 CET on Zoom, where‪ Leon Lang will give a talk titled “A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels“.

Title: A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels

Abstract: Group equivariant convolutional networks (GCNNs) endow classical convolutional networks with additional symmetry priors, which can lead to a considerably improved performance. Recent advances in the theoretical description of GCNNs revealed that such models can generally be understood as performing convolutions with G-steerable kernels, that is, kernels that satisfy an equivariance constraint themselves. While the G-steerability constraint has been derived, it has to date only been solved for specific use cases – a general characterization of Gsteerable kernel spaces is still missing. This work provides such a characterization for the practically relevant case of G being any compact group. Our investigation is motivated by a striking analogy between the constraints underlying steerable kernels on the one hand and spherical tensor operators from quantum mechanics on the other hand. By generalizing the famous Wigner-Eckart theorem for spherical tensor operators, we prove that steerable kernel spaces are fully understood and parameterized in terms of 1) generalized reduced matrix elements, 2) ClebschGordan coefficients, and 3) harmonic basis functions on homogeneous spaces.

Link to paper: https://arxiv.org/pdf/2010.10952.pdf

To gain more deep insights into Group Equivariance, feel free to join and discuss it 🙂 !