The achievement of emotional functions relies on the interactions of various functional systems of the human brain. Numerous studies tried to explore the mechanism of the emotions based on the functional connectivity. However, the contribution of the transient and stable patterns of brain communication in brain emotions was still unclear. The recently proposed activation network framework assumed the activity of functional connectivity (AFC) and background of functional connectivity (BFC) to respectively represent transient and stable patterns of functional connectivity. In this paper, we employed the activation network framework to SEED-IV dataset to achieve the emotion recognition and evaluate the performance of the transient and stable patterns in emotional activities. The top 100 critical connections of each subject were extracted by a data-driven feature selection strategy. The critical connections across all subjects of both AFC and BFC suggested the importance of Gamma band in emotion recognition. Especially, the AFC and BFC showed the different communication modes during the emotions. Finally, the subject-independent classification was employed on each subject’s critical connections to achieve the emotion recognition. The BFC showed the best classification accuracy of 78.71% ± 1.73% (mean ± std). The findings demonstrated that human emotions were mostly influenced by the consistent brain communication patterns. The findings of this investigation offer a new insight on the studies of human emotion.