You should feed all 3 classes of data to train the network. An imbalance of class data during training can cause overfitting easily even though it is already pre-trained.