The aim of this study was to investigate the environmental impact and human health risks associated with toxic trace element (TTE) exposure in the abandoned Yaro Mine, Korea. Carcinogenic and non-carcinogenic risks were assessed separately for adults and children. Among the various pathways, the rate of TTE intake from the ingestion of groundwater was highest, followed in descending order by crop consumption, soil ingestion, and soil contact. The carcinogenic risk from the ingestion of groundwater was highest, followed by crop consumption and ingestion of contaminated surface soil. The non-carcinogenic risk from the ingestion of groundwater was highest (53.57% of the total non-carcinogenic risk), followed by crop intake (38.53%) and surface soil ingestion (4.80%). The risk assessment revealed that contaminated soil around Yaro mine posed a high risk to the health of inhabitants, mainly via groundwater ingestion and crop consumption. Reclamation measures should include methods of disrupting the high-risk routes between the source and recipient. Stabilization and covering techniques are promising options for reducing the hazard (i.e., exposure to the bioavailable fraction of TTE) and creating a chemical or physicochemical barrier to the potential migration pathways.