Cloud covering is an important factor affecting solar radiation and causes fluctuations in solar energy production. Therefore, real-time recognition and the prediction of cloud covering and the adjustment of the angle of photovoltaic panels to improve power generation are important research areas in the field of photovoltaic power generation. In this study, several methods, namely, the principle of depth camera measurement distance, semantic segmentation algorithm, and long- and short-term memory (LSTM) network were combined for cloud observation. The semantic segmentation algorithm was applied to identify and extract the cloud contour lines, determine the feature points, and calculate the cloud heights and geographic locations of the cloud shadows. The LSTM algorithm was used to predict the trajectory and speed of the cloud movement, achieve accurate and real-time detection, and track the clouds and the sun. Based on the results of these methods, the shadow area of the cloud on the ground was calculated. The recursive neural LSTM network was also used to predict the track and moving speed of the clouds according to the cloud centroid data of the cloud images at different times. The findings of this study can provide insights to establish a low-cost intelligent monitoring predicting system for cloud covering and power generation.