Title: Bayesian Optimization with Uncertainty Bounds for Neural Networks
Abstract: This thesis studies the application of neural networks (NNs) as estimators in Bayesian optimization and assesses if and how they can overcome this curse of dimensionality. Since quantifying the uncertainty of predictions is key in Bayesian optimization, several methods of representing predictive uncertainty for NNs are considered. Specifically, this thesis compares a very recent approach introduced by Heiss et al. (2021) called "neural optimization-based model uncertainty" (NOMU), against "Deep Ensembles" and "MC Dropout", two more established NN-based methods, and Gaussian processes. Experimental evaluations on synthetic data confirm that Gaussian processes outperform the NN-based methods in low dimensional settings (1D-2D) and show that NOMU performs as good or better than the other NN-based methods. However, the experiments in higher dimensions suggest that the NN-based methods improve and finally manage to outperform Gaussian processes with standard configurations. Furthermore, the results indicate that especially in higher dimensions, NOMU performs robustly as good or better than the other NN-based methods.
