Image classification is an important research field of computer vision. Traditional image classification requires manual design of feature extraction methods, and the accuracy of classification is closely related to the selected feature extraction methods. With the rapid development of network multimedia technology, the number of images that need to be classified becomes larger and it is more complex to classify the images. The manual design of feature extraction methods not only consumes time, but also lowers the accuracy. The accuracy of image classification using deep learning methods can reach or even exceed the level of manual classification. In this paper, we first propose an average weight selective kernel networks (AWSKnet) model. The idea of ensemble learning is introduced into selective kernel networks (SKnet) to construct AWSKnet, integrating the features of convolution layer learning. It makes the features learned in the convolution layer more discriminative and confluent, which enhances the feature training effect of network. Second, we use the basic solution of a generalized differential operator to generate a base kernel function in the H space and use the multi-layer strategy of deep learning to construct the multi-layer convolution kernel in the H and H space by using the base kernel functions in the H space.Finally, we use the AWSKnet network model to learn the characteristics of the image data, and then use the Gaussian process classifier based on the multi-layer convolution kernel function (MKGPC) to perform image classification experiments on the CIFAR-10, SVHN and MNIST datasets. An experimental analysis on three public image datasets shows that our methods outperform all state-of-the-art image classification models we use for comparison.