The vibration signals of faulty rotating machinery are typically nonstationary, nonlinear, and mixed with abundant compounded background noise. To extract the potential excitations from the observed rotating machinery, signal demodulation and time-frequency analysis are indispensable. This work proposes a novel particle swarm optimization-based variational mode decomposition method, which adopts the minimummean envelope entropy to optimize the parameters (alpha and K) in the existing variational mode decomposition. The proposed fault-detection framework separated the observed vibration signals into a series of intrinsic modes. A certain number of the intrinsic modes are then selected by means of the Hilbert transform-based square envelope spectral kurtosis. Subsequently, in this study, the feature representations were reconstructed via the selected intrinsic modes; then, the envelope spectra of the real faulty conditions were generated in the rotating machinery. To verify the performance of the proposed method, a testbed platform of a gearbox with a combination of different faults was implemented. The experimental results demonstrated that the proposed method represented the patterns of the fault frequency more explicitly than the available empirical mode decomposition, the local mean decomposition, and the wavelet package transform method.