Different scale of information is an important component in computer vision systems. Recently, there are considerable researches on utilizing multi-scale information to solve the scale-invariant problems, such as GoogLeNet and FPN. In this paper, we introduce the notion of different scale information fusion (DSIF) and show that it has a significant effect on the performance of object recognition systems. We analyze the DSIF in several architecture designs, and the effect of nonlinear activations, dropout, sub-sampling and skip connections on it. This leads to clear suggestions for ways of the DSIF to choose.