GAN — Why it is so hard to train Generative Adversarial Networks!


GAN Problems

  • Non-convergence: the model parameters oscillate, destabilize and never converge,
  • Mode collapse: the generator collapses which produces limited varieties of samples,
  • Diminished gradient: the discriminator gets too successful that the generator gradient vanishes and learns nothing,
  • Unbalance between the generator and discriminator causing overfitting, &
  • Highly sensitive to the hyperparameter selections.



Nash equilibrium

Generative model with KL-Divergence

Note: KL(p, q) is the integral of the red curve in the right.


Vanishing gradients in JS-Divergence

Unstable gradients


Why mode collapse in GAN?

Modified from source

Implicit Maximum Likelihood Estimation (IMLE)

Hyperparameters & training

Balance between the discriminator and generator

Cost v.s. image quality

Further reading




Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store