Are latent variables normal distributed in VAE? Actually, there are other including uniformly distributed etc… The normal distribution is quite popular. Mathematically, a normal distribution is easy to manipulate and sample.
Removing information from your input forces you not to be overconfidence in your model which can be a sign of overfitting by training data. This is very similar to the idea of dropout in DL.