Alzheimer's disease (AD) has become a severe chronic disease that affects the health of the elderly all over the world. And the number of patients currently suffering continues to rise each year. With the rapid development of medical imaging technology, although researchers have done extensive works on the diagnosis of AD through new computer vision technology, it is still a challenge to realize the diagnosis of AD and Mild Cognitive Impairment (MCI) as precise as possible end-to-end by relying on Magnetic Resonance Imaging (MRI) image resources. In this paper, a new variant model of the Broad Learning System (BLS) for accurate diagnosis of AD and MCI is presented for MRI images. The proposed model is composed of two modules named feature mapping module and feature enhancement module. To adapt to the characteristics of medical images, a new feature mapping module that contains multi groups of feature down-sampling is designed to get the multi-scale features of the images without any additional feature selection. As a result, the proposed model can integrate multi-scale convolution features of the feature mapping module and abstract features of the feature enhancement module end-to-end when learning the AD diagnostic task. At the same time, the proposed model is a lightweight model whose complexity has been significantly simplified. To verify the validity of the proposed model, the ANDI-1 dataset was used in the relevant experiments. After 5-fold cross-validation, the proposed model has achieved the accuracy of 91.83% and 75.52% for the AD diagnostic task and MCI diagnostic task, respectively. The experimental results demonstrate that the proposed model could achieve better performance compared to other methods under the AD and MCI diagnostic tasks.