The effect of zirconium dioxide ($ZrO_2$) on the properties of color conversion glasses was examined in the $BaO-ZnO-B_2O_3-SiO_2$ system. The difference in refractive index between glass and phosphor affect the optical properties of the color conversion glass because of light scattering. Reducing the difference in refractive index is a method to improve the luminous efficacy of color conversion glasses. As a reference, a type of glass that contains 25 mol% of each component was used. To increase the refractive index of the glass samples, the BaO content was increased from 25 to 40 mol%, and $ZrO_2$ was added at levels of 1, 3, and 5 mol%. Color conversion glasses were prepared by sintering a mixture of glass and 5 wt% $YAG:Ce^{3+}$ phosphor. As a result, the refractive index of the glass was found to be dependent on the BaO and $ZrO_2$ contents in the BaO-ZnO-$B_2O_3-SiO_2$ system. As the BaO and $ZrO_2$ contents were increased, the luminous efficacy of the color conversion glass was improved because the refractive index difference between the glass and the $YAG:Ce^{3+}$ phosphor decreased.