Double check your work with images containing larger objects. If not working, go to YOLOv3 and give that a try. Otherwise, try something like FPN.