As a classical method to deal with nonlinear and nonstationary signals, the Hilbert–Huang transform (HHT) is widely used in various fields. In order to overcome the drawbacks of the Hilbert–Huang transform (such as end effects and mode mixing) during the process of empirical mode decomposition (EMD), a revised Hilbert–Huang transform is proposed in this article. A method called local linear extrapolation is introduced to suppress end effects, and the combination of adding a high-frequency sinusoidal signal to, and embedding a decorrelation operator in, the process of EMD is introduced to eliminate mode mixing. In addition, the correlation coefficients between the analyzed signal and the intrinsic mode functions (IMFs) are introduced to eliminate the undesired IMFs. Simulation results show that the improved HHT can effectively suppress end effects and mode mixing. To verify the effectiveness of the new HHT method with respect to fault diagnosis, the revised HHT is applied to analyze the vibration displacement signals in a rotor system collected under normal, rubbing, and misalignment conditions. The simulation and experimental results indicate that the revised HHT method is more reliable than the original with respect to fault diagnosis in a rotor system.