RL — Actor-Critic Methods: A3C, GAE, DDPG, Q-prop

Value-learning, including Q-learning, is believed to be unstable when a large and non-linear function approximator, like the deep network, is used. This is a curse for many reinforcement learning training (RL). To address these problems, DQN applies experience replay and the target network. However, DQN handles discrete and low-dimensional action spaces only. For high-dimensional or continuous action spaces, finding the action with maximum Q-value requires complex optimization that seems impractical.

Policy Gradient method is popular. However, it has low sample efficiency and high variance. To reduce it, we can subtract the expected rewards with a baseline b. We can have different choices of b as long as it is not dependent on the model parameter θ. The optimal solution will be the same.

Often, we use the value function V(s) as a baseline. Then, we compute a new metric called the advantage function A which equals Q(s, a) minus V(s). Intuitively, we increase the chance of an action if it performs better than the average action.

The policy gradient is rewritten as:


This new formularization introduces an actor, who determines the action based on a state. This can be modeled by a neural network NN parameterized by θ which supports actions for a continuous and high dimensional space also — something that DQN misses. In the next section, we introduce an NN called a critic to be used in computing A.


Here, we introduce another model V parameterized by 𝜙 below to estimate the value function of a state.

In this equation, we have two models: the actor which estimates the policy, and the critic which estimates the value function.

So how can we model V(s)? One good way is to use Monte-Carlo to find out the total rewards of a complete episode. Then, we use supervised learning to learn the critic model Φ.

Alternatively, we can apply the Temporal Difference TD concept to model V. Here is the formula with a 1-step lookahead.

In the equation below, we also add a regularization penalty to regulate the model 𝜙.

The Monte Carlo method is unbiased but with a high variance — a slight change in the sampled actions can change the result of a game completely. The TD method above is biased but low variance as it samples one action only. But the critic model is biased, at least in early training.

Generalized Advantage Estimation (GAE)

In the diagram below, we have a 1-step lookup, a 2-step lookup, and all the way to the Monte-Carlo method. Finally, we take a weighted sum to compute the expected rewards (underlined in red below).

Modified from source (λ is between 0 and 1)

Let’s write it down formally. The advantage function for an n-step lookahead is:

The Generalized advantage estimation (GAE) is a hybrid combination of 1-step lookahead to the Monte-Carlo method. The hyperparameter λ ranges from 0 to 1. When λ=0, it is TD, and when λ=1, it is the Monte-Carlo method.

Actor-critic networks


This should look similar to a Policy Gradient method. Policy Gradient utilizes gradient descent. It behaves nicely if the objective function is convex. However, Policy Gradient can suffer large variance. To overcome that, we can increase the batch size but it hurts the sample efficiency badly. By introducing a critic, it reduces variance and improves the sample efficiency at the cost of bias. To improve training stability, steps 2 and 4 above can be done with a batch of samples instead of one.

Asynchronous Advantage Actor-Critic (A3C)

Asynchronous actor-learners. A3C uses multiple asynchronous threads to learn the policy. Since threads are running asynchronous, each thread synchronizes its policy with the master at a different point in time. Hence, each thread explores a different part of the environment with slightly different policies. Actually, this helps the training with less correlation among samples. Deep learning is quite tolerable or welcomes errors (noises) in early training. In fact, lighting the accuracy requirements may help performance.


Previously, we use experience replay to improve RL performance. With this asynchronous calculation, experience replay is no longer needed. With multiple threads and the removal of experience replay, A3C produces better models in a shorter time.

k-step lookahead

A3C also applies TD with k-steps lookahead. Typically, it has k set to 5.

A3C algorithms

Below is the A3C algorithm. θ models the policy and θv models the value function V. Both θ and θv are implemented with the convolutional networks with parameters shared except the output layer. For θ, we use a softmax function to output the probabilities of actions. For θv, we use a linear layer to output a scalar value.


An entropy term on the policy is also added to the objective function. Entropy measures the randomness of the random variables. A deterministic policy has no randomness and therefore has zero entropy. The added term discourages the policy from collapsing to a deterministic policy too fast. This hurts exploration. So the extra term penalizes the model to be too sure of actions given specific states.

(β is a hyperparameter controlling the weight.)

Deep Deterministic Policy gradient DDPG

If we have a deterministic policy μ, we don’t need to have the second expectation function anymore. The equation can be simplified to:

In an on-policy RL method, like the vanilla Policy Gradient, the key focus is on improving the current policy. There is only one policy to determine what actions and what samples to use for the next training iteration. Whenever the policy is updated, we take new samples and they will be likely used once only. This hurt sample efficiency. For the equation above, the expectation function does not sample from μ. Indeed, Q can be fitted with all previous experience. In DDPG, previous samples are replayed to improve sample efficiency.

Applying chain rule, the policy gradient becomes:

DDPG is an off-policy method. There is more than one policy. Indeed, to fix the exploration limitation, a second policy is introduced. Gaussian noise is added to the current policy to determine the action to be sampled and executed.

The noise added helps exploration. Samples are stored in a buffer and randomly selected to train the critic and to refine the actor. This is the same as the experience replay in DQN. Also, DDPG uses the DQN’s target network concept. Here is the algorithm.


DDPG creates a deterministic policy for a high-dimensional continuous control space. We may argue that it is not exactly a deterministic policy. Nevertheless, empirical data show good performance with nice generalization to 20 different tasks without retuning hyperparameters.

Continuous & discrete space


Control variate

Let say we want to calculate I expressed as an integral of f(U) below. This can be done by sampling the value of U and take the average of f(U).

However, the estimation varies depending on the sampled values for U. Control variate is a technique to reduce the variance of this estimation. First, we establish a function g that is related to f. We choose g such that its expected value can be calculated analytically. By sampling U, we estimate I. But we also compute the corresponding g(U) and calculate how the values deviate from the expected g. We recalibrate our estimation of I by introducing a second term showing this deviation.

Modified from Wikipedia

Adaptive Q-Prop method

Now, we apply the same concept to solve the high variance problem in the policy gradient.

Let’s summarize something we already learned first. The following equation computes the Policy Gradient with Monte Carlo. It is unbiased.

We can use V as a baseline to reduce variance and the advantage function A is:

The idea of the Actor-critic is to fit the actor and critic model:

With the policy gradient calculated as:

So far, we have just repeat what we learn.

Let’s subtract and then add an arbitrary function f into the policy gradient.

where we estimate any arbitrary function f with the Taylor series.


So, let’s choose what is f.



In Policy Gradient, we use A instead of Q. So let’s make the same change here. The new formularization of the Policy Gradient becomes:

It composes of the residual REINFORCE gradient and the regular policy gradient. Q is the critic model that we learn.

Q-Prop is a Monte Carlo policy gradient estimator with a special form of a control variate. To reduce the variance, we introduce η(s) to scale some of the terms.

The optimal state-dependent factor η(s) can be computed per state using η* above. η* provides maximum reduction in the variance. The policy gradient computed in this form is called the adaptive Q-Prop method.

Conservative and Aggressive Q-Prop

η* has a high variance also. Therefore the Conservative Q-Prop is proposed as

Which turns off the control variate when  and Ā are negatively correlated, i.e. when the critic is very bad. In Aggressive Q-Prop, it is set to

Conservative Q-Prop will achieve more stable performance than the standard and aggressive variants. Here is the algorithm for the Adaptive Q-prop. It is a mixture of policy gradient and actor-critic.