Electroencephalogram (EEG) data is commonly applied in the emotion recognition research area. It can accurately reflect the emotional changes of the human body by applying graphical-based algorithms or models. EEG signals are non-linear signals. Biological tissues’ adjustment and adaptive ability will inevitably affect electrophysiological signals, making EEG have the typical non-linear characteristics. Graph Convolutional Broad Network (GCB-net) extracted features from non-linear signals and abstract features via stacked CNN. It adopted the broad concept and enhanced the feature by the broad learning system (BLS), obtaining sound results. However, it performed poorly with increasing network depth, and the accuracy of some features decreased with BLS. This paper proposed a Residual Graph Convolutional Broad Network (Residual GCB-net), which promotes the performance on a deeper-layer network and extracts higher-level information. It substitutes the original convolutional layer with residual learning blocks, which solves the deep learning network degradation and extracts more features in deeper networks. In SEED data set, GCB-Res net could obtain the best accuracy (94.56%) on the all-frequency band of differential entropy (DE) and promote much on another feature. In Dreamer, it obtained the best accuracy (91.55%) on the dimension of Arousal. The result demonstrated the excellent classification performance of Residual GCB-net in EEG emotion recognition.