Suspended solids concentration (SSC) is an important indicator of the degree of water pollution. However, when using an empirical or semi-empirical model adapted to some of the inland waters to estimate SSC on unmanned aerial vehicle (UAV)-borne hyperspectral images, the accuracy is often not sufficient. Thus, in this study, we attempted to use the particle swarm optimization (PSO) algorithm to find the optimal parameters of the least-squares support vector machine (LSSVM) model for the quantitative inversion of SSC. A reservoir and a polluted riverway were selected as the study areas. The spectral data of the 36-point and 29-point 400–900 nm wavelength range on the UAV-borne images were extracted. Compared with the semi-empirical model, the random forest (RF) algorithm and the competitive adaptive reweighted sampling (CARS) algorithm combined with partial least squares (PLS), the accuracy of the PSO-LSSVM algorithm in predicting the SSC was significantly improved. The training samples had a coefficient of determination ( R 2 ) of 0.98, a root mean square error (RMSE) of 0.68 mg/L, and a mean absolute percentage error (MAPE) of 12.66% at the reservoir. For the polluted riverway, PSO-LSSVM also performed well. Finally, the established SSC inversion model was applied to UAV-borne hyperspectral remote sensing (HRS) images. The results confirmed that the distribution of the predicted SSC was consistent with the observed results in the field, which proves that PSO-LSSVM is a feasible approach for the SSC inversion of UAV-borne HRS images.