Solar-induced chlorophyll fluorescence (SIF) has been proven to be an efficient indicator of vegetation photosynthesis. To investigate the relationship between SIF and Gross Primary Productivity (GPP), tower-based continuous spectral observations coordinated with eddy covariance (EC) measurements are needed. As the strong absorption effect at the O2-A absorption bands has an obvious influence on SIF retrieval based on the Fraunhofer Line Discrimination (FLD) principle, atmospheric correction is required even for tower-based SIF observations made with a sensor tens of meters above the canopy. In this study, an operational and simple solution for atmospheric correction of tower-based SIF observations at the O2-A band is proposed. The aerosol optical depth (AOD) and radiative transfer path length (RTPL) are found to be the dominant factors influencing the upward and downward transmittances at the oxygen absorption band. Look-up tables (LUTs) are established to estimate the atmosphere transmittance using AOD and RTPL based on the MODerate resolution atmospheric TRANsmission 5 (MODTRAN 5) model simulations, and the AOD is estimated using the ratio of the downwelling irradiance at 790 nm to that at 660 nm (E790/E660). The influences of the temperature and pressure on the atmospheric transmittance are also compensated for using a corrector factor of RTPL based on an empirical equation. A series of field measurements were carried out to evaluate the performance of the atmospheric correction method for tower-based SIF observations. The difference between the SIF retrieved from tower-based and from ground-based observations decreased obviously after the atmospheric correction. The results indicate that the atmospheric correction method based on a LUT is efficient and also necessary for more accurate tower-based SIF retrieval, especially at the O2-A band.