Structured Models with Gaussian Processes

Abstract

Machine learning methods have seen great success recently in a wide range of digital domains such as speech recognition, computer vision or video games. However, bridging the gap to applications in the physical world has proved to be challenging as they introduce a new set of requirements. Machine learning systems must make efficient use of expert knowledge, handle low data regimes, and quantify uncertainties. This thesis explores how structured probabilistic models allow us to cope with these requirements. Structured models combine black-box and white-box modeling approaches to formalize expert knowledge while still being able to gain new insights from data.

In this work, we formulate Bayesian structured models using methods from Bayesian nonparametrics. We use information about the structure of a learning problem to formulate machine learning models that reproduce knowledge, are understandable for domain-experts, make physically plausible predictions in unseen situations and can quantify their own uncertainty. We explore how to embed general function approximators in Bayesian probabilistic models to enforce structure and discuss how to formulate inference schemes based on composite and hierarchical Gaussian process models. Using real-world industrial applications such as the detection of faulty sensors and the prediction of power generation in a wind-farm as examples, we show how to use structured models to factorize uncertainties, achieve interpretability, and generalize to unobserved inputs.

In settings where internal structure and generalization behavior come into focus, model selection using marginal likelihoods can be insufficient to identify desirable models. We consider how to formalize the subjectiveness in model selection through the task a model will be used to solve. We show that in a reinforcement learning problem, semantic models outperform other models with similar performance metrics and allow experts to influence agent behavior. We explore the properties of structured models in a broader context and discuss the limits of current inference schemes, why models with suboptimal marginal likelihoods can perform well in hierarchical systems, and how to formulate Bayesian inference problems that take downstream tasks into account.

Publication
PhD Thesis
Markus Kaiser
Markus Kaiser
Research Scientist

Research Associate at the University of Cambridge and Research Scientist at Siemens AG. I am interested in scalable Bayesian machine learning and Gaussian processes.