Gastrointestinal metaplasia (GIM) is a disease that is closely related to early gastric cancer. The early diagnosis of GIM can effectively avoid gastric cancer. Traditionally, GIM diagnosis is done through human analysis of endoscopy imaging, which is time-consuming and exhausting. Computer aided diagnosis of GIM is urgently needed but currently there is no such computer system in commercial market. Considering the complex features of gastroscopic images, and different pixels contain different weight information of color and texture features, a novel multi feature fusion method composed of new feature module (FM) and attention feature module (AFM) is proposed. First, a residual deep network is used as the base framework to build FM combined with high and low-level features, which can make up for the deficiency of single high-level features. Then, the RGB image, HSV (Hue Saturation Value) image, and LBP (Local Binary Pattern) features are considered as 3-way inputs of the proposed model. In other words, the deep features of the endoscopy image are extracted respectively from image pixels, colors, and texture. Finally, these deep features are sent into a novel AFM to generate the final features for GIM recognition. AFM first adaptively learns feature weights through attention mechanism, and then fuses the above three types of features. Experimental results show that the proposed method achieves a high recognition accuracy of 90.28% under a dataset of 1050 images collected from a local hospital. In addition, the proposed method is superior to single-featured networks and existing method in term of recognition accuracy.