GAN — What is Generative Adversarial Networks GAN?
To create something from nothing is one of the greatest feelings, ... It’s heaven.
By Prince
GANs are about creating, like drawing a portrait or composing a symphony, which is far more challenging than many other deep learning tasks. It is much easier to identify a Monet painting than to paint one, whether by a computer or a person. Yet this creative challenge brings us closer to understanding intelligence. The rise of GANs has fueled an enormous body of research, with tens of thousands of papers published over the past decade. In game development, for example, many production artists are hired to create animations frame by frame. By automating these with GANs, the creative process could eventually shift toward higher-level artistry rather than routine work. As part of the GAN series, this article introduces the concept of GANs and their core algorithm.
Note: While preparing my book, I reviewed the content as part of the material collection process. The material did not require many changes, but the explanations have been proof-checked with ChatGPT for clarity. If you would like to be notified when the book is released, please connect with me on LinkedIn.
What does GAN do?
The primary focus of GANs (Generative Adversarial Networks) is to generate data from scratch, most notably images, though applications in other domains such as music, text, and video have also been explored. For instance, GANs can perform image-to-image translation, such as transforming a horse into a zebra. In reinforcement learning, they have also been used to accelerate training by generating realistic environments or simulating experiences, enabling robots to learn much faster.
Generator and discriminator
GANs consist of two neural networks: a generator and a discriminator. This section first explains how the generator produces images, then later how it is trained.
The process begins by sampling a latent vector z from a simple prior distribution, for example, from a Gaussian (normal) distribution Ɲ(0, I) or from a uniform distribution U(−1, 1). The generator G maps z to an image x via x = G(z). Although this may seem almost magical, the process of training G to make x resemble samples from the data distribution will be explained step by step.
Conceptually, z represents the latent features of the generated images, such as color or shape. In deep learning classification tasks, we do not directly control the features that the model learns. Similarly, in GANs, the semantic meaning of each component of z is not explicitly controlled, and the training process learns these representations automatically. For example, we cannot specify which element of z corresponds to hair color. The most effective way to uncover such relationships is by generating images from different z values and examining the results. The following images were generated by a Progressive GAN from random noise z.
By gradually varying a single dimension of z, we can observe its semantic effect on the generated images, for example by changing attributes such as hair color.
So, what exactly is this “magic” generator G? One of the most popular designs is the DCGAN generator. It uses a series of transposed convolutions to upsample z and produce the image x. Conceptually, DCGAN can be thought of as a deep learning classifier operating in reverse.
A generator on its own would produce only random noise. The discriminator in a GAN provides the guidance needed for the generator to learn how to create realistic images.
For example, in CycleGAN, the generator is trained to transform real scenery into paintings in the style of Monet, while the discriminator learns to distinguish real from generated images. Its feedback guides the generator to produce paintings that increasingly resemble genuine Monet works.
So how is this done technically? The discriminator processes real training images and generated images separately, learning to distinguish between the two. Its output, D(x), represents the probability that the input image x is real, i.e., P(class of x = real image).
The discriminator is trained in the same way as a standard deep network classifier. For real images, we want D(x) = 1, and for generated images, we want D(x) = 0. Through this training, the discriminator gradually learns to identify the features that distinguish real images from generated ones.
On the other hand, the generator’s goal is to produce images that the discriminator classifies as real, meaning we want D(G(z)) = 1. To achieve this, we backpropagate the discriminator’s output error through the discriminator and into the generator. In this way, the generator is trained to create images that increasingly align with what the discriminator considers real.
We train both networks in alternating steps, locking them into a dynamic competition that drives mutual improvement. Over time, the discriminator becomes adept at spotting even the smallest differences between real and generated samples, while the generator learns to produce images so convincing that the discriminator can no longer distinguish them. When this balance is reached, the GAN is said to have converged, producing images that look natural.
The discriminator concept is not limited to GANs. Acting as a critic, it can be integrated into other deep learning applications to provide feedback that improves model performance.
Backpropagation
Let’s now look at the key equations. The discriminator outputs a value D(x), which represents the probability that x is a real image. Its goal is to correctly classify real images as real and generated images as fake, effectively maximizing the likelihood of the observed data. To measure this, we use cross-entropy loss, as is common in deep learning, expressed as p log(q). For real images, the true label p equals 1, while for generated images the label is 0 (i.e., one minus the real label). With this setup, the discriminator’s objective becomes:
The generator’s objective is to produce images that maximize D(x), pushing the discriminator to classify them as real.
GANs are often formalized as a minimax game, where the generator (G) seeks to minimize the value function V while the discriminator (D) aims to maximize it.
Once both objective functions are defined, training proceeds jointly using alternating gradient descent. First, the generator’s parameters are fixed while a gradient ascent step is performed on the discriminator using both real and generated images. Then, the discriminator is held fixed and the generator is updated with a gradient descent step. This alternating process continues until the generator produces high-quality images. The illustration below summarizes the data flow and the gradients used for backpropagation.
The pseudo-code below brings all the components together and illustrates the full training process of a GAN.
Generator diminished gradient
However, a common issue in GAN training is the problem of vanishing gradients for the generator. Early in training, the discriminator often outperforms the generator, easily distinguishing real images from generated ones. As a result, the value function V approaches 0, meaning −log(1 − D(G(z))) → 0. This causes the generator’s gradients to vanish, slowing down optimization. To address this, GAN training often uses an alternative objective function for the generator, which provides stronger gradients and improves learning.
More thoughts
In general, the concept of generating data brings both great potential and significant risks. GANs are just one example, and many other generative models exist.
This simple yet powerful idea has led to many research papers. As is often the case, simplicity proves effective. Here, we introduce the core concept, but the possibilities extend far beyond what is described. To understand why generative models have become so popular, let’s look at some of their potential applications:
