Although the existing Fine-Grained Visual Classification (FGVC) researches has made some progress, there are still some deficiencies need to be refined. Specifically, 1. The feature maps are used directly by most methods after they are extracted from the original images, which lacks further processing of feature maps and may lead irrelevant features to negatively affect network performance; 2. In many methods, the utilize of feature maps is relatively simple, and the relationship between feature maps that helpful for accurate classification is ignored. 3. Due to the high similarity between subcategories as well as the randomness and instability of training, the network prediction results may sometimes not accurate enough. To this end, we propose an efficient Self-supervised Attention Filtering and Multi-scale Features Network (SA-MFN) to improve the accuracy of FGVC, which consists of three modules. The first one is the Self-supervised Attention Map Filter, which is proposed to extract the initial attention maps of subcategories and filter out the most distinguishable and representative local attention maps. The second module is the Multi-scale Attention Map Generator, which extracts a global spatial feature map from the filtered attention maps and then concatenates it with the filtered attention maps. The third module is the Reiterative Prediction, in which the first prediction result of the network is re-utilized by this module to improve the accuracy and stability. Experimental results show that our SA-MFN outperforms the state-of-the-art methods on multiple fine-grained classification datasets, especially on the dataset of Stanford Cars, the proposed network achieves the accuracy of 94.7%.