Many people suffer from the Fatty Liver disease due to the changes in diet and lifestyle, and the convenient diagnosis of it has attracted many attentions in recent years. The computerized tongue or facial diagnosis as an important diagnostic tool provides a possible way to detect the disease in the daily life. Most of existing approaches only takes a single modality (e.g., tongue or face) into account, although various modalities would contribute complementary information which is beneficial for the improvement of the diagnosis accuracy. To circumvent this issue, a novel multi-modal fusion method is presented in this paper. Particularly, a noninvasive capture device is first used to captured the tongue and facial images, followed by the feature extraction. Our so-called joint discriminative and collaborative representation approach is then proposed to not only reveal the correlation between the tongue and facial images, but also keep the discriminative representation of each class simultaneously. To optimize the proposed method, an efficient algorithm is proposed, obtaining a closed-form solution and greatly reducing the computation. In identification of the Fatty Liver Disease for healthy controls, the proposed multi-modal fusion approach achieves 85.10% in average accuracy and 0.9363 in the area under ROC curve, which obviously outperform the case of using a single modality and some state-of-the-art methods.