Electroluminescence (EL) images, which have the high spatial resolution, provide the opportunity to detect tiny defects on the surface of photovoltaic (PV) modules. However, manual analysis of EL images is usually an expensive and time-consuming project and requires extensive expertise. Therefore, automatic defect detection is becoming more and more important in the photovoltaic field. This paper proposes an intelligent algorithm for defect detection of photovoltaic modules based the high-resolution network (HRNet). First, aiming at the problem of insufficient data, a data augmentation method is designed to expand the dataset of EL images. Next, an identification algorithm adapted to the image model, called the self-fusion network (SeFNet), is improved. Here, we use the SeFNet to replace the classification layer in the HRNet. SeFNet allows better feature fusion of multi-resolution information in image models. At the same time, it utilizes the improved asymmetric convolution module to enhance the convolution kernel performance through parallel triple operations, so it improves the classification accuracy. Multiple evaluation metrics in the experiment show that the proposed method has better defect recognition performance.