The molar ratio of methanol to rubber seed oil (RSO), catalyst loading, and the reaction time of RSO biodiesel production were optimized in this work. The response surface methodology, using the Box–Behnken design, was analyzed to determine the optimum fatty acid methyl ester (FAME) yield. The performance of various nanomagnetic CaO-based catalysts—KF/CaO-Fe3O4, KF/CaO-Fe3O4-Li (Li additives), and KF/CaO-Fe3O4-Al (Al additives)—were compared. Rubber seed biodiesel was produced via the transesterification process under subcritical methanol conditions with nanomagnetic catalysts. The experimental results indicated that the KF/CaO-Fe3O4-Al nanomagnetic catalyst produced the highest FAME yield of 86.79%. The optimum conditions were a 28:1 molar ratio of methanol to RSO, 1.5 wt % catalyst, and 49 min reaction time. Al additives of KF/CaO-Fe3O4 nanomagnetic catalyst enhanced FAME yield without Al up to 18.17% and shortened the reaction time by up to 11 min.