Recently, there have been many studies to classify the image-based damage of bridge using the deep learning and to evaluate the condition. These attempts are one of the ways to overcome limitations of visual inspection through inspectors, and it is also aimed to reduce the cost of necessary maintenance budget by enabling accurate and rapid damage assessment of rapidly growing old facilities and difficult parts of visual inspection. However, it is possible to classify and quantitatively express simple damage (one damage classification such as cracks) with image information (big data) of bridges, but classification and quantification of complex damage can be done by using one deep learning is a limit. Therefore, this study presents considerations and a method to be used for damage detection on the image basis using deep learning.