Car use has been identified as sedentary behavior, although it may enhance mobility, particularly in the older population. This cross-sectional study aimed to compare the time spent in objectively determined sedentary behavior (SB) and physical activity (PA) between older drivers and non-drivers. Four hundred and fifty Japanese older adults (74.3 ± 2.9 years) who had valid accelerometer data were included. They were asked to respond to a questionnaire and wear an accelerometer (HJA-350IT, Omron Healthcare) on their waist for 7 consecutive days in 2015. To compare activity time between drivers and non-drivers, we calculated estimated means using analysis of covariance, adjusting for sociodemographic, physical, and psychological factors and accelerometer wear time. Compared to non-drivers, drivers engaged in more light-intensity PA (LPA) (drivers: 325.0 vs. non-drivers: 289.0 min/day) and moderate-to-vigorous PA (drivers: 37.5 vs. non-drivers: 30.0 min/day) and less SB (drivers: 493.4 vs. non-drivers: 535.9 min/day) (all p < 0.05). After stratification by age, sex, and residential area, larger effect of driving on PA time was found in older-older adults, in men, and in rural residents. Older drivers were found to be more physically active than non-drivers, suggesting more access to outdoor activities or expanding social network.