The purpose of this study was to examine the impact of summer breaks on the body composition and cardiovascular fitness of elementary school children who participated in a multi-year school-based physical activity intervention. Participants were 404 children who had their height and weight measured and completed the Progressive Aerobic Cardiovascular Endurance Run (PACER) during physical education classes at the beginning and end of the school year for three consecutive years. To examine the effects of time on health-related fitness data, general linear mixed effects models were employed. The results indicate that there was a trend toward an increase in body mass index (BMI) after the summer of 2015 (p = 0.958), and a significant increase in BMI after the summer of 2016 compared to time point 1 (p < 0.001). For PACER laps, there were trends toward decreases in PACER laps after the summers of 2015 (p = 0.515) and 2016 (p = 0.073). Summer breaks tended to attenuate the BMI and PACER lap improvements that were observed during the intervention. While school-based physical activity programming has had some successes in improving health-related fitness markers, the loss of these improvements over the summer is of concern to both practitioners and researchers. It is clear that additional efforts are needed to limit obesogenic behaviors during the summer months.