This will give you more details on importance sampling.
But, in short:
The variance depends on how you sample data. That is why you cannot use the old policy for too long for the practical purpose.
I don’t know what do you mean by “shrink”.
See Dec 9, 2018 for a previous answer that may relate to your question.