The accurate prediction of surface solar irradiance is of great significance for the generation of photovoltaic power. Surface solar irradiance is affected by many random mutation factors, which means that there are great challenges faced in short-term prediction. In Northwest China, there are abundant solar energy resources and large desert areas, which have broad prospects for the development of photovoltaic (PV) systems. For the desert areas in Northwest China, where meteorological stations are scarce, satellite remote sensing data are extremely precious exploration data. In this paper, we present a model using FY-4A satellite images to forecast (up to 15–180 min ahead) global horizontal solar irradiance (GHI), at a 15 min temporal resolution in desert areas under different sky conditions, and compare it with the persistence model (SP). The spatial resolution of the FY-4A satellite images we used was 1 km × 1 km. Particle image velocimetry (PIV) was used to derive the cloud motion vector (CMV) field from the satellite cloud images. The accuracy of the forecast model was evaluated by the ground observed GHI data. The results showed that the normalized root mean square error (nRMSE) ranged from 18.9% to 21.6% and the normalized mean bias error (nMBE) ranged from 3.2% to 4.9% for time horizons from 15 to 180 min under all sky conditions. Compared with the SP model, the nRMSE value was reduced by about 6%, 8%, and 14% with the time horizons of 60, 120, and 180 min, respectively.