The demand for the high-performance and low-cost position control actuators in many applications promotes the development of three-phase hybrid stepper motors. The torque ripple, loss, and pullout torque of the motor are the key factors to be considered in the motor application. In order to solve the problems of the open-loop control mode, this paper proposes a new “sensorless closed-loop” control mode to significantly improve the performance of three-phase hybrid stepper motors. This control mode is developed by estimating the rotor position with the Extended Kalman filter observer, thereby realizing the closed-loop control of the motor with sensorless technology. This paper illustrates the effects of this control mode by analyzing motor noise, losses, and pullout-torque. The results show that the use of the “sensorless closed-loop” control mode presented can effectively improve the performance of the stepper motor while maintaining the advantages of the motor in terms of cost and size. These results have positive significance for the development, application, and promotion of high-performance three-phase hybrid stepper motor systems.