Selected Publications

1. ICLR

   Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence

   Tailor, Dharmesh, Correia, Alvaro, Nalisnick, Eric, and Louizos, Christos

   In 13th International Conference on Learning Representations (to appear) Apr 2025

   Abs HTML

   Uncertainty quantification is an important prerequisite for the deployment of deep learning models in safety-critical areas. Yet, this hinges on the uncertainty estimates being useful to the extent the predictive prediction intervals are well-calibrated and sharp. In the absence of inherent uncertainty estimates (e.g. pretrained models), popular approaches that operate post-hoc include Laplace’s method and split conformal prediction (split-CP). However, Laplace’s method can be miscalibrated when the model is misspecified and split-CP requires sample splitting, and thus comes at the expense of statistical efficiency. In this work, we construct prediction intervals for neural network regressors post-hoc without held-out data. This is achieved by approximating the full conformal prediction method (full-CP). Whilst full-CP nominally requires retraining the model for every test point and candidate label, we propose to train just once and locally perturb model parameters using Gauss-Newton influence to approximate the effect of retraining. Coupled with linearization of the network, we express the absolute residual nonconformity score as a piecewise linear function of the candidate label allowing for an efficient procedure that avoids the exhaustive search over the output space. On standard regression benchmarks and bounding box localization, we show the resulting prediction intervals are locally-adaptive and often tighter than those of split-CP.

2. AISTATS

   Learning to Defer to a Population: A Meta-Learning Approach

   Tailor, Dharmesh, Patra, Aditya, Verma, Rajeev, Manggala, Putra, and Nalisnick, Eric

   In 27th International Conference on Artificial Intelligence and Statistics May 2024

   Abs HTML PDF Code

   The learning to defer (L2D) framework allows autonomous systems to be safe and robust by allocating difficult decisions to a human expert. All existing work on L2D assumes that each expert is well-identified, and if any expert were to change, the system should be re-trained. In this work, we alleviate this constraint, formulating an L2D system that can cope with never-before-seen experts at test-time. We accomplish this by using meta-learning, considering both optimization- and model-based variants. Given a small context set to characterize the currently available expert, our framework can quickly adapt its deferral policy. For the model-based approach, we employ an attention mechanism that is able to look for points in the context set that are similar to a given test point, leading to an even more precise assessment of the expert’s abilities. In the experiments, we validate our methods on image recognition, traffic sign detection, and skin lesion diagnosis benchmarks.

3. NeurIPS

   The Memory-Perturbation Equation: Understanding Model’s Sensitivity to Data

   Nickl, Peter, Xu, Lu*, Tailor, Dharmesh*, Möllenhoff, Thomas, and Khan, Mohammad Emtiyaz

   In 37th Conference on Neural Information Processing Systems (NeurIPS) Dec 2023

   Abs HTML PDF Code

   Understanding model’s sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model’s sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.

4. UAI

   Exploiting Inferential Structure in Neural Processes

   Tailor, Dharmesh, Khan, Mohammad Emtiyaz, and Nalisnick, Eric

   In 39th Conference on Uncertainty in Artificial Intelligence (UAI) Aug 2023

   Abs HTML PDF Code

   Neural Processes (NPs) are appealing due to their ability to perform fast adaptation based on a context set. This set is encoded by a latent variable, which is often assumed to follow a simple distribution. However, in real-word settings, the context set may be drawn from richer distributions having multiple modes, heavy tails, etc. In this work, we provide a framework that allows NPs’ latent variable to be given a rich prior defined by a graphical model. These distributional assumptions directly translate into an appropriate aggregation strategy for the context set. Moreover, we describe a message-passing procedure that still allows for end-to-end optimization with stochastic gradients. We demonstrate the generality of our framework by using mixture and Student-t assumptions that yield improvements in function modelling and test-time robustness.

5. DP4ML

   Memory Maps to Understand Models

   Tailor, Dharmesh, Chang, Paul Edmund, Swaroop, Siddharth, Nalisnick, Eric, Solin, Arno, and Khan, Mohammad Emtiyaz

   In ICML 2023 Workshop on Principles of Duality for Modern Machine Learning Jul 2023

   HTML