Hands-on Bayesian Neural Networks – A Tutorial for Deep Learning Users

2. What is a Bayesian Neural Network?

Stochastic Neural Network:

• Stochastic Weights (Figure 3c): Instead of learning fixed weights, we learn a distribution over weights. That is, for a connection between neurons, instead of a single weight $w$, we model $w\sim p(w)$.
• Stochastic Activations (Figure 3b): Instead of using deterministic functions like ReLU or tanh, the activation itself is sampled from a distribution, e.g., $a_i=f(x)+\epsilon$, where $\epsilon$ is noise sampled from $N(0,{\sigma }^2)$ or some other distribution.
• Stochastic neural networks inherently simulate many models, due to their random components. Every forward pass is like evaluating a different model instance from a family governed by $p(\theta )$. This naturally resembles an ensemble of models.

Bayesian Neural Network:

• A Bayesian Neural Network models a distribution over weights, not fixed weights. Formally:

Instead of learning $\theta \ast$ (a set of all point estimate NN parameters), we learn a posterior distribution $p(\theta |D)$ given data $D$.

• To design a BNN, the first step is the choice of a deep neural network architecture, i.e., a functional model. Then, one has to choose a stochastic model, i.e., a prior distribution over the possible model parametrization $p(\theta )$ and a prior confidence in the predictive power of the model $p(y|x,\theta )$.

p(\boldsymbol{\theta}|D)=\frac{p(D_\boldsymbol{y}|D_\boldsymbol{x},\boldsymbol{\theta})p(\boldsymbol{\theta})}{\int_\boldsymbol{\theta}p(D_\boldsymbol{y}|D_\boldsymbol{x},\boldsymbol{\theta'})p(\boldsymbol{\theta'})d\boldsymbol{\theta'}}\propto p(D_\boldsymbol{y}|D_\boldsymbol{x},\boldsymbol{\theta})p(\boldsymbol{\theta})