This study used an in-depth interview survey to examine and analyze people’s perceptions of the decline in the use of the terms of address, seobangnim, doryeonmin, and agassi in the Korean language. We sought to determine the degree of and reasons for Korean people’s unfamiliarity and/or dislike of the terms, with an eye toward restoring them to common usage. We compared our results with those of Na Yoon Jung et al.’s study on standardized speech (2011) and survey results (2017). The combined results suggest that seobangnim, doryeonnim, and agassi no longer function properly as the terms of address and reference, as summarized in the following findings: (1) seobangnim, once commonly used between spouses to refer to parents, friends, or a spouse when calling or pointing, is no longer common; (2) the use of doryeonnim for addressing or referencing a husband’s brother(s) is disappearing and is now being viewed negatively; (3) agassi, once a common term used to address and reference a husband’s sister, has fallen from favor and is viewed negatively; (4) the most common reason for the declining use and negative associations is that these terms have connotations of gender and status inequalities. Our results support those found by Na Yoon Jeong et al. (2017).