Swagatam Haldar

Research

My current research interests broadly circle around understanding different aspects of LLM training: scaling laws, effect of different architectures such as mixture-of-experts, and optimizers, and automating them to some extent. In earlier years, my research spanned topics on simulation-based inference; explainability and interpretability. Find some of my research artefacts below!

1. Amortizing Scaling Law Construction Costs

   Jha, Abhash Kumar^*, Onuţu, Diana Alexandra^*, Mallik, Neeratyoy^*,  Haldar, Swagatam^*, Laing, Sam, Ajroldi, Niccolò, Liu, Shiwei, Vanschoren, Joaquin, and Klein, Aaron

   * denotes equal contribution

   Preprint 2026

   Abs arXiv Bib

   Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to 10-100x.

   @misc{jha2026amortizingscalinglawconstruction,
     title = {Amortizing Scaling Law Construction Costs},
     author = {Jha, Abhash Kumar and Onuţu, Diana Alexandra and Mallik, Neeratyoy and Haldar, Swagatam and Laing, Sam and Ajroldi, Niccolò and Liu, Shiwei and Vanschoren, Joaquin and Klein, Aaron},
     year = {2026},
     eprint = {2609.05016},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
   }

2. 

   AutoML for Simulation-based Inference

   Haldar, Swagatam

   Master's Thesis, University of Tübingen 2025

   Supervisors: Prof. Dr. Jakob H. Macke, Prof. Dr. Katharina Eggensperger

   Abs PDF

   Simulation-based inference (SBI) is a general tool for approximating Bayesian inference for complex simulators in science. Modern SBI methods—such as Neural Posterior Estimation (NPE)—rely on neural network based generative models. Building and training such SBI models can involve multiple design choices which can affect the quality of the inferred posteriors. Users typically manually tune them or fall back to default settings in available libraries, which can lead to suboptimal performance. Automating the selection of SBI variants and their hyperparameters with AutoML approaches could enhance the effectiveness and accessibility of SBI methods. However, the potential of AutoML for SBI as well as the choice of suitable optimization objectives, has not been systematically investigated yet. We here address this challenge and explore how AutoML can automate a crucial part of the SBI pipeline: selecting and training inference networks, with a focus on NPE. We empirically identify validation loss as a tractable and reliable proxy objective for hyperparameter optimization (HPO), as lower validation losses correspond to posterior estimates that are closer to a reference distribution, as verified by metrics such as the Classifier Two-Sample Test (C2ST). This finding makes it possible to apply standard HPO methods such as random search and Bayesian optimization. On 10 widely used benchmark tasks, we show that (a) HPO can significantly improve over default settings, (b) the optimal hyperparameter setting remains largely stable across simulation budgets ranging from 10^2 to 10^5 samples, and (c) transferring optimized hyperparameter values from an already tuned task to other tasks typically yields better posteriors than the default setting on the SBI benchmark. Overall, we provide a framework for tuning SBI hyperparameters, resulting in ≈6% C2ST improvement on average across all tasks and simulation budgets compared to available defaults. These results highlight AutoML’s potential to improve the robustness and effectiveness of SBI methods, and they open up further possibilities for leveraging budget-aware extensions for HPO and meta-learning across tasks.

3. Trust Regions for Explanations via Black-Box Probabilistic Certification

   Dhurandhar, Amit,  Haldar, Swagatam, Wei, Dennis, and Natesan Ramamurthy, Karthikeyan

   In International Conference on Machine Learning 2024

   Abs arXiv Bib Code

   Given the black box nature of machine learning models, a plethora of explainability methods have been developed to decipher the factors behind individual decisions. In this paper, we introduce a novel problem of black box (probabilistic) explanation certification. We ask the question: Given a black box model with only query access, an explanation for an example and a quality metric (viz. fidelity, stability), can we find the largest hypercube (i.e., \ell_∞ ball) centered at the example such that when the explanation is applied to all examples within the hypercube, (with high probability) a quality criterion is met (viz. fidelity greater than some value)? Being able to efficiently find such a trust region has multiple benefits: i) insight into model behavior in a region, with a guarantee; ii) ascertained stability of the explanation; iii) explanation reuse, which can save time, energy and money by not having to find explanations for every example; and iv) a possible meta-metric to compare explanation methods. Our contributions include formalizing this problem, proposing solutions, providing theoretical guarantees for these solutions that are computable, and experimentally showing their efficacy on synthetic and real data.

   @inproceedings{pmlr-v235-dhurandhar24a,
     title = {Trust Regions for Explanations via Black-Box Probabilistic Certification},
     author = {Dhurandhar, Amit and Haldar, Swagatam and Wei, Dennis and Natesan Ramamurthy, Karthikeyan},
     booktitle = {International Conference on Machine Learning},
     pages = {10736--10764},
     year = {2024},
     series = {Proceedings of Machine Learning Research},
     publisher = {PMLR},
   }

4. 

   Interpretable Differencing of Machine Learning Models

   Haldar, Swagatam, Saha, Diptikalyan, Wei, Dennis, Nair, Rahul, and Daly, Elizabeth M

   In Uncertainty in Artificial Intelligence 2023

   Abs arXiv Bib Code

   Understanding the differences between machine learning (ML) models is of interest in scenarios ranging from choosing amongst a set of competing models, to updating a deployed model with new training data. In these cases, we wish to go beyond differences in overall metrics such as accuracy to identify where in the feature space do the differences occur. We formalize this problem of model differencing as one of predicting a dissimilarity function of two ML models’ outputs, subject to the representation of the differences being human-interpretable. Our solution is to learn a Joint Surrogate Tree (JST), which is composed of two conjoined decision tree surrogates for the two models. A JST provides an intuitive representation of differences and places the changes in the context of the models’ decision logic. Context is important as it helps users to map differences to an underlying mental model of an AI system. We also propose a refinement procedure to increase the precision of a JST. We demonstrate, through an empirical evaluation, that such contextual differencing is concise and can be achieved with no loss in fidelity over naive approaches.

   @inproceedings{haldar2023interpretable,
     title = {Interpretable Differencing of Machine Learning Models},
     author = {Haldar, Swagatam and Saha, Diptikalyan and Wei, Dennis and Nair, Rahul and Daly, Elizabeth M},
     booktitle = {Uncertainty in Artificial Intelligence},
     pages = {788--797},
     year = {2023},
     organization = {PMLR},
   }