An understanding of the enrichment mechanisms of cadmium (Cd) and arsenic (As) in the process of rock weathering and soil formation is essential to develop agriculture according to local conditions. However, the enrichments of soil Cd and As under natural background conditions in karst areas are still uncertain. The enrichment factor, geo-accumulation index, redundancy analysis, and other methods were used to analyze the enrichment degree and the influencing factors of Cd and As on 5 rock–soil profiles and 15 topsoil samples, which were collected from a karst forest area in Libo County, Guizhou Province. The results showed that the enrichment process was divided into three stages. In the first stage, Cd and As were enriched in carbonate rocks, and their mean concentrations were 1.65 and 3.9 times those of the corresponding abundance of the crust. In the second stage, the enrichment of the parent rock into the soil, the enrichment factors of Cd and As in the parent material horizon relative to the bedrock horizon were 9.2 and 2.82, respectively. The third stage refers to the enrichments of Cd and As in the topsoil, where Cd enrichment was more obvious than that of As. Soil organic matter (SOM) and phosphorus (P) are important factors that influenced the enrichments of Cd and As in the topsoil. The functional groups of SOM were complexed with Cd and As; P easily formed precipitates with Cd, and the tree litter was fed back to the topsoil, which may be the reason for the surface enrichment of Cd and As. This study will help the scientific community understand the enrichment mechanisms of soil Cd and As in karst areas.