This paper presents a nanofluid-based cooling method for a brushless synchronous generator (BLSG) by using Al2O3 lubricating oil. In order to demonstrate the superiority of the nanofluid-based cooling method, analysis of the thermal performance and efficiency of the nanofluid-based cooling system (NBCS) for the BLSG is conducted along with the modeling and simulation cases arranged for NBCS. Compared with the results obtained under the base fluid cooling condition, results show that the nanofluid-based cooling method can reduce the steady-state temperature and power losses in BLSG and decrease the temperature settling time and changing ratio, which demonstrate that both steady-state and transient thermal performance of NBCS are improved as nanoparticle volume fraction (NVF) in nanofluid increases. Besides, although the input power of cycling pumps in NBCS has ~30% increase when the NVF is 10%, the efficiency of the NBCS has a slight increase because the 4.1% reduction in power loss of BLSG is bigger than the total incensement of input power of the cycling pumps. The results illustrate the superiority of the nanofluid-based cooling method, and it indicates that the proposed method has a broad application prospect in the field of thermal control of onboard synchronous generators with high power density.