Just wake up so I am not sure I fully understand your question. Convolution layer is applied to features to create the position-sensitive score map. When we apply convolution, we still want to explore the spatial locality of these features. After the score map is created, we average the vote.