Recent Publications

2022

1. Bio. Pysch.

   Interoception as Modeling, Allostasis as Control

   Sennesh, Eli, Theriault, Jordan, Brooks, Dana, Meent, Jan-Willem, Barrett, Lisa Feldman, and Quigley, Karen S.

   Biological Psychology 2022

   Abs HTML PDF

   The brain regulates the body by anticipating its needs and attempting to meet them before they arise – a process called allostasis. Allostasis requires a model of the changing sensory conditions within the body, a process called interoception. In this paper, we examine how interoception may provide performance feedback for allostasis. We suggest studying allostasis in terms of control theory, reviewing control theory’s applications to related issues in physiology, motor control, and decision making. We synthesize these by relating them to the important properties of allostatic regulation as a control problem. We then sketch a novel formalism for how the brain might perform allostatic control of the viscera by analogy to skeletomotor control, including a mathematical view on how interoception acts as performance feedback for allostasis. Finally, we suggest ways to test implications of our hypotheses.

2. Neuroinf.

   A Computational Neural Model for Mapping Degenerate Neural Architectures

   Khan, Zulqarnain, Wang, Yiyu, Sennesh, Eli, Dy, Jennifer, Ostadabbas, Sarah, van de Meent, Jan-Willem, Hutchinson, J. Benjamin, and Satpute, Ajay B.

   Neuroinformatics Mar 2022

   Abs HTML PDF

   Degeneracy in biological systems refers to a many-to-one mapping between physical structures and their functional (including psychological) outcomes. Despite the ubiquity of the phenomenon, traditional analytical tools for modeling degeneracy in neuroscience are extremely limited. In this study, we generated synthetic datasets to describe three situations of degeneracy in fMRI data to demonstrate the limitations of the current univariate approach. We describe a novel computational approach for the analysis referred to as neural topographic factor analysis (NTFA). NTFA is designed to capture variations in neural activity across task conditions and participants. The advantage of this discovery-oriented approach is to reveal whether and how experimental trials and participants cluster into task conditions and participant groups. We applied NTFA on simulated data, revealing the appropriate degeneracy assumption in all three situations and demonstrating NTFA’s utility in uncovering degeneracy. Lastly, we discussed the importance of testing degeneracy in fMRI data and the implications of applying NTFA to do so.

3. WACV

   Enhancing Few-Shot Image Classification With Unlabelled Examples

   Bateni, Peyman, Barber, Jarred, Meent, Jan-Willem, and Wood, Frank

   In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Jan 2022

   Abs HTML PDF Code

   We develop a transductive meta-learning method that uses unlabelled instances to improve few-shot image classification performance. Our approach combines a regularized Mahalanobis-distance-based soft k-means clustering procedure with a modified state of the art neural adaptive feature extractor to achieve improved test-time classification accuracy using unlabelled data. We evaluate our method on transductive few-shot learning tasks, in which the goal is to jointly predict labels for query (test) examples given a set of support (training) examples. We achieve state of the art performance on the Meta-Dataset, mini-ImageNet and tiered-ImageNet benchmarks. All trained models and code have been made publicly available

2021

1. AISTATS

   Rate-Regularization and Generalization in Variational Autoencoders

   Bozkurt, Alican, Esmaeili, Babak, Tristan, Jean-Baptiste, Brooks, Dana, Dy, Jennifer, and Meent, Jan-Willem

   In International Conference on Artificial Intelligence and Statistics Mar 2021

   Abs HTML PDF

   Variational autoencoders (VAEs) optimize an objective that comprises a reconstruction loss (the distortion) and a KL term (the rate). The rate is an upper bound on the mutual information, which is...

2. UAI

   Learning proposals for probabilistic programs with inference combinators

   Stites, Sam, Zimmermann, Heiko, Wu, Hao, Sennesh, Eli, and Meent, Jan-Willem

   In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence 27–30 jul 2021

   Abs HTML PDF Code

   We develop operators for construction of proposals in probabilistic programs, which we refer to as inference combinators. Inference combinators define a grammar over importance samplers that compose primitive operations such as application of a transition kernel and importance resampling. Proposals in these samplers can be parameterized using neural networks, which in turn can be trained by optimizing variational objectives. The result is a framework for user-programmable variational methods that are correct by construction and can be tailored to specific models. We demonstrate the flexibility of this framework by implementing advanced variational methods based on amortized Gibbs sampling and annealing.

3. EMNLP

   Disentangling Representations of Text by Masking Transformers

   Zhang, Xiongyi, van de Meent, Jan-Willem, and Wallace, Byron

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing Nov 2021

   Abs HTML PDF

   Representations from large pretrained models such as BERT encode a range of features into monolithic vectors, affording strong predictive accuracy across a range of downstream tasks. In this paper we explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects. Concretely, we learn binary masks over transformer weights or hidden units to uncover subsets of features that correlate with a specific factor of variation; this eliminates the need to train a disentangled model from scratch for a particular task. We evaluate this method with respect to its ability to disentangle representations of sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics. By combining masking with magnitude pruning we find that we can identify sparse subnetworks within BERT that strongly encode particular aspects (e.g., semantics) while only weakly encoding others (e.g., syntax). Moreover, despite only learning masks, disentanglement-via-masking performs as well as — and often better than —previously proposed methods based on variational autoencoders and adversarial training.

4. NAACL

   On the Impact of Random Seeds on the Fairness of Clinical Classifiers

   Amir, Silvio, van de Meent, Jan-Willem, and Wallace, Byron

   In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jun 2021

   Abs HTML PDF

   Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s). We explore the implications of this phenomenon for model fairness across demographic groups in clinical prediction tasks over electronic health records (EHR) in MIMIC-III —— the standard dataset in clinical NLP research. Apparent subgroup performance varies substantially for seeds that yield similar overall performance, although there is no evidence of a trade-off between overall and subgroup performance. However, we also find that the small sample sizes inherent to looking at intersections of minority groups and somewhat rare conditions limit our ability to accurately estimate disparities. Further, we find that jointly optimizing for high overall performance and low disparities does not yield statistically significant improvements. Our results suggest that fairness work using MIMIC-III should carefully account for variations in apparent differences that may arise from stochasticity and small sample sizes.

5. NeurIPS

   Nested Variational Inference

   Zimmermann, Heiko, Wu, Hao, Esmaeili, Babak, and Meent, Jan-Willem

   In Advances in Neural Information Processing Systems Jun 2021

   Abs HTML PDF

   We develop nested variational inference (NVI), a family of methods that learn proposals for nested importance samplers by minimizing an forward or reverse KL divergence at each level of nesting. NVI is applicable to many commonly-used importance sampling strategies and provides a mechanism for learning intermediate densities, which can serve as heuristics to guide the sampler. Our experiments apply NVI to (a) sample from a multimodal distribution using a learned annealing path (b) learn heuristics that approximate the likelihood of future observations in a hidden Markov model and (c) to perform amortized inference in hierarchical deep generative models. We observe that optimizing nested objectives leads to improved sample quality in terms of log average weight and effective sample size.

6. AAMAS

   Action Priors for Large Action Spaces in Robotics

   Biza, Ondrej, Wang, Dian, Platt, Robert, Meent, Jan-Willem, and Wong, Lawson L.S.

   In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems Jun 2021

   Abs HTML PDF Code

   In robotics, it is often not possible to learn useful policies using pure model-free reinforcement learning without significant reward shaping or curriculum learning. As a consequence, many researchers rely on expert demonstrations to guide learning. However, acquiring expert demonstrations can be expensive. This paper proposes an alternative approach where the solutions of previously solved tasks are used to produce an action prior that can facilitate exploration in future tasks. The action prior is a probability distribution over actions that summarizes the set of policies found solving previous tasks. Our results indicate that this approach can be used to solve robotic manipulation problems that would otherwise be infeasible without expert demonstrations. Source code is available at https://github.com/ondrejba/action_priors.

7. ICML

   Conjugate Energy-Based Models

   Wu*, Hao, Esmaeili*, Babak, Wick, Michael L, Tristan, Jean-Baptiste, and van de Meent, Jan-Willem

   In International Conference on Machine Learning Jun 2021

   Abs

   In this paper, we propose conjugate energy-based models (CEBMs), a new class of energy-based models that define a joint density over data and latent variables. The joint density of a CEBM decomposes into an intractable distribution over data and a tractable posterior over latent variables. CEBMs have similar use cases as variational autoencoders, in the sense that they learn an unsupervised mapping from data to latent variables. However, these models omit a generator network, which allows them to learn more flexible notions of similarity between data points. Our experiments demonstrate that conjugate EBMs achieve competitive results in terms of image modelling, predictive power of latent space, and out-of-domain detection on a variety of datasets.

2020

1. NeurIPS

   Neural Topographic Factor Analysis for fMRI Data

   Sennesh*, Eli, Khan*, Zulqarnain, Wang, Yiyu, Hutchinson, J. Benjamin, Satpute, Ajay, Dy, Jennifer, and van de Meent, Jan-Willem

   Advances in Neural Information Processing Systems 2020

   HTML PDF Code
2. ICML

   Amortized Population Gibbs Samplers with Neural Sufficient Statistics

   Wu, Hao, Zimmermann, Heiko, Sennesh, Eli, Le, Tuan Anh, and van de Meent, Jan-Willem

   Advances in Neural Information Processing Systems 2020

   Abs HTML PDF

   Amortized variational methods have proven difficult to scale to structured problems, such as inferring positions of multiple objects from video images. We develop amortized population Gibbs (APG) samplers, a class of scalable methods that frame structured variational inference as adaptive importance sampling. APG samplers construct high-dimensional proposals by iterating over updates to lower-dimensional blocks of variables. We train each conditional proposal by minimizing the inclusive KL divergence with respect to the conditional posterior. To appropriately account for the size of the input data, we develop a new parameterization in terms of neural sufficient statistics. Experiments show that APG samplers can be used to train highly-structured deep generative models in an unsupervised manner, and achieve substantial improvements in inference accuracy relative to standard autoencoding variational methods.

3. MLHC

   Query-Focused EHR Summarization to Aid Imaging Diagnosis

   McInerney, Denis Jered, Dabiri, Borna, Touret, Anne-Sophie, Young, Geoffrey, van de Meent, Jan-Willem, and Wallace, Byron C.

   Machine Learning for Healthcare Apr 2020

   Abs HTML PDF

   Electronic Health Records (EHRs) provide vital contextual information to radiologists and other physicians when making a diagnosis. Unfortunately, because a given patient’s record may contain hundreds of notes and reports, identifying relevant information within these in the short time typically allotted to a case is very difficult. We propose and evaluate models that extract relevant text snippets from patient records to provide a rough case summary intended to aid physicians considering one or more diagnoses. This is hard because direct supervision (i.e., physician annotations of snippets relevant to specific diagnoses in medical records) is prohibitively expensive to collect at scale. We propose a distantly supervised strategy in which we use groups of International Classification of Diseases (ICD) codes observed in ’future’ records as noisy proxies for ’downstream’ diagnoses. Using this we train a transformer-based neural model to perform extractive summarization conditioned on potential diagnoses. This model defines an attention mechanism that is conditioned on potential diagnoses (queries) provided by the diagnosing physician. We train (via distant supervision) and evaluate variants of this model on EHR data from a local hospital and MIMIC-III (the latter to facilitate reproducibility). Evaluations performed by radiologists demonstrate that these distantly supervised models yield better extractive summaries than do unsupervised approaches. Such models may aid diagnosis by identifying sentences in past patient reports that are clinically relevant to a potential diagnoses.