Aiming at the problems of the expensive computational cost of Self-attention and cascaded Self-attention weakening local feature information in the current ViT model, the ESA (Efficient Self-attention) module for optimizing computational complexity and the LE (Locally Enhanced) module for enhancing local information are proposed. The ESA module sorts the attention intensity of the class token and patch tokens of each Transformer encoder in the ViT model, only retains the weight value of patch token strongly associated with the class token in the attention matrix, and reuses the attention matrix of adjacent layers, so as to reduce the calculation of the model and accelerate the reasoning of the model; the LE module parallels a Depth-wise convolution in each Transformer encoder, it enables Transformer to capture global feature information and strengthen local feature information at the same time, which effectively improves the image recognition rate. A large number of experiments are performed on common image recognition datasets such as Tiny ImageNet, CIFAR-10 and CIFAR-100, experimental results show that the proposed method performs better in recognition accuracy under the premise of less computation.