Grassland aboveground biomass (AGB) is a crucial indicator when studying the carbon sink of grassland ecosystems. The exploration of the grassland AGB inversion method with viable reproducibility is significant for promoting the practicability and efficiency of grassland quantitative monitoring. Therefore, this study provides a novel retrieval method for grassland AGB by coupling the PROSAIL (PROSPECT + SAIL) model and the random forest (RF) model on the basis of the lookup-table (LUT) method. These sensitive spectral characteristics were optimized to significantly correlate with AGB (ranging from 0.41 to 0.68, p < 0.001). Four methods were coupled with the PROSAIL model to estimate grassland AGB in the West Ujimqin grassland, including the LUT method, partial least square (PLSR), RF and support vector machine (SVM) models. The ill-posed inverse problem of the PROSAIL model was alleviated using the MODIS products-based algorithm. Inversion results using sensitive spectral characteristics showed that the PROSAIL + RF model offered the best performance (R2 = 0.70, RMSE = 21.65 g/m2 and RMESr = 27.62%), followed by the LUT-based method, which was higher than the PROSAIL + PLSR model. Relatively speaking, the PROSAIL + SVM model was more challenging in this study. The proposed method exhibited strong robustness and universality for AGB estimation in large-scale grassland without field measurements.