This study assessed heavy metal contents and their mobility, bioaccessibility, environmental risk, and health effects in the farmland soils of Drinking Water Source Quality Protection (DWSQP) areas contaminated by livestock manure and a paddy field contaminated by co-use of irrigation and drainage canals in Taiwan. The risk assessment code (RAC) and synthesis toxicity index (STI) for the soils were obtained. The potential health effects caused from soil direct ingestion by hand-to-mouth activity and dermal contact frequently occurring to farmers were further evaluated. The Cu, Zn, and Cr levels in DWSQP areas and the Changhwa (CH) paddy field exceeded the standards promulgated by Taiwan Environmental Protection Administration (EPA). Nevertheless, RAC in DWSQP areas was in low risk levels. In contrast, RAC from Cu and Zn in CH paddy soils was in medium levels. Non-carcinogenic risks for farmers based on the total and bioaccessible metals in DWSQP areas and CH soils were all <1. However, carcinogenic risks based on bioaccessible Cr still exceeded 10−6 in several soils, indicating that the potential impacts on environmental and human health due to direct and indirect exposures to these contaminated soils should be concerned.