In this paper, the effect of the swirler scheme on combustion instability is studied. Through the proper orthogonal decomposition (POD) of flame images, Abel inverse transform and other methods, the influence of swirl intensity on the characteristic frequency of combustion instability was emphatically studied. Based on the low order thermoacoustic network (LOTAN) of the combustor, the flame transfer function (FTF) under different swirl schemes was obtained by the optimization method. The experimental results show that the stable combustion equivalence ratio boundary of the system decreases monotonously with the decrease in swirl intensity, while the characteristic frequency of unstable combustion is not monotonous with the swirl intensity (the oscillating frequency of swirler A with the largest swirl intensity is approximately 319 Hz, swirler B is approximately 280 Hz, swirler C with the smallest swirl intensity is approximately 290 Hz). The optimization results of FTF can easily introduce this non monotonic phenomenon. The swirl intensity mainly affects the hysteresis time of the system (the lag time of swirlers A, B and C are 5.98 ms, 6.82 ms and 6.20 ms, respectively), which is mainly caused by affecting the flame structure and convection velocity. At the same time, the FTF obtained by optimization reflects the same trend with the experimental results, and there is no significant difference in value, which proves the rationality of the optimization method. This work emphasizes the importance of FTF for combustion instability analysis.