Image for post
Image for post
“A little boy holding a book with a surprised expression on his face” by Ben White

GAN — What is wrong with the GAN cost function?

We work hard to produce mathematical models for deep learning. But often, we are not successful and fall back to the empirical data to support our arguments. Arjovsky et al 2017 wrote a paper to illustrate the GAN problem mathematically. The paper develops a mathematical model in an effort to illustrate potential issues with the GAN cost functions. Here are some of the claims:

  • Theoretically, it happens even a slight misalignment between the ground truth and the model, and
  • During training, adding noise to generated images can stabilize the model.
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Original proposal
Image for post
Image for post
Alternative proposal
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

More thoughts

The mathematical model produces a good framework for discussion and study. But the debate on cost functions is likely to continue. The mathematical model paints a very negative picture on the original cost functions in GAN but not necessarily supported by experiments. There is a fundamental question: if the gradient behaves so badly, how can we explain the result using the original GAN. There is even a Google Brain report showing the “state-of-the-art” cost functions have no difference in performance if the GAN with the second objective functions is tuned more vigorously.

Reference

Towards principled methods for training Generative Adversarial Networks

Written by

Deep Learning

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store