Recently, deep learning has made great achievements in facial expression recognition. However, occlusion and large skew will greatly affect the accuracy of facial expression recognition in practice. Therefore, we propose a novel framework based on symmetric SURF and heterogeneous soft partition network to quickly recognize facial recognition under partial occlusion. In this framework, an occlusion detection module based on symmetric SURF is presented to detect the occlusion part, which helps to locate the horizontal symmetric area of the occlusion area. After that, a face inpainting module based on mirror transition is presented to rapidly accomplish the face inpainting under the unsupervised circumstance. Moreover, a recognition network based on heterogeneous soft partitioning is proposed for the facial expression recognition. After heterogeneous soft partitioning, the weights of each part are input and to into the recognition network as more prior information for training. Finally, we feed the weighted image into the trained neural network for expression recognition. Experimental results show that the accuracy of the proposed method is respectively 7% and 8% higher than the average accuracies from the state-of-the-art methods on Cohn-Kanade (CK +) and fer2013 datasets. Besides, the run time of our method is 2.38 s faster than the most advanced.