Gastric intestinal metaplasia (GIM) is a pre-malignant lesion of gastric cancer, which is the fourth leading cause of cancer-related mortalities. The accurate diagnosis and effective treatment of GIM can decrease the incidence of gastric cancer. Traditionally, GIM diagnosis is conducted through upper endoscopy imaging, which is highly dependent on endoscopists’ experience, and the diagnostic results may fluctuate with their discrepant skills or potential fatigue. Thus, computer-aided diagnosis (CAD) of GIM with high accuracy is urgently needed, while currently there is no such computer system in commercial market. In this paper, a novel broad learning system stacking framework with multi-scale attention (BLS2-MSA) is proposed, which contains Level-0 for preliminary diagnosis and Level-1 for fnal decision. In Level-0 of the BLS2-MSA, there are fve classifers, four of which are constructed using multi-scale features from the backbone neural network with the proposed parallel attention module, and the other classifer adopts a standard TL method only. In Level-1 of the BLS2-MSA, a broad learning system-based incremental updating approach is frst proposed to boost the performance of classifers in Level-0. Experimental results show that the True Positive Rate (TPR), the True Negative Rate (TNR), the Positive Predictive Value (PPV), the Accuracy (ACC), the F1and the Area Under ROC Curve (AUC) of the BLS2-MSA are 93.6%, 91.2%, 93.6%, 93.2%, 93.6 and 0.931 respectively, and the diagnostic results demonstrate that the BLS2- MSA could perform competitively compared with skilled endoscopists. All of these indicate that the proposed method enables an accurate and reliable GIM diagnosis.