An outrigger damper system has been proposed to reduce dynamic responses of tall buildings. In previous studies, an outrigger damper system was optimally designed to decrease a wind-induced or earthquake-induced dynamic response. When an outrigger damper system is optimally designed for wind excitation, its control performance for seismic excitation deteriorates. Therefore, a smart outrigger damper system is proposed in this study to make a control system that can simultaneously reduce both wind and seismic responses. A smart outrigger system is made up of MR (Magnetorheological) dampers. A fuzzy logic control algorithm (FLC) was used to generate command voltages sent for smart outrigger damper system and the FLC was optimized by genetic algorithm. This study shows that the smart outrigger system can provide good control performance for reduction of both wind and earthquake responses compared to the general outrigger system.