Abstract

This advanced course explores the intersection of active learning, uncertainty quantification, and information theory in deep learning. Drawing from recent advances in Bayesian neural networks and information-theoretic frameworks, we examine how these methods can improve both label and training efficiency. The curriculum spans foundational concepts in uncertainty quantification like aleatoric and epistemic uncertainty to cutting-edge techniques in batch active learning and prediction-oriented acquisition. We analyze methods such as BatchBALD and the Expected Predictive Information Gain (EPIG), which address key challenges in data subset selection and predictive uncertainty. Students will gain both theoretical depth and practical insight into applying these methods to scientific problems, with particular emphasis on uncertainty quantification in non-differentiable models and black-box settings.

Goals

1. Build strong information-theoretic foundations for active learning, including practical intuition for entropy, mutual information, and KL-divergence.

2. Master uncertainty quantification methods for deep learning, including Bayesian neural networks and ensemble techniques, with focus on distinguishing and estimating aleatoric and epistemic uncertainty in scientific applications. Build intuition for why certain data points are more valuable than others for model learning.

3. Understand modern batch active learning approaches like BatchBALD and transductive active learning approaches like EPIG.

4. Develop practical skills in scaling active learning to large datasets, including techniques for approximating acquisition functions and handling non-differentiable models through black-box approaches.

5. Learn to analyze and compare active learning methods through their theoretical properties and empirical behavior, enabling students to select and adapt approaches for specific scientific domains and computational constraints.

Citation

If you use this material in your work, please cite the following paper:

@misc{kirsch2024balitu,
    author = {Kirsch, Andreas},
    title = {(Bayesian) Active Learning, Information Theory, and Uncertainty},
    year = {2024},
    month = {Dec},
    url = {https://blackhc.github.io/balitu/}
}