Numerous applications have explored medical image classification using deep learning models. With the emergence of Explainable AI (XAI), researchers have begun to recognize its potential in validating the authenticity and correctness of results produced by black-box deep learning models. On the other hand, current diagnostic approaches for osteonecrosis face significant challenges, including difficulty in early detection, subjectivity in image interpretation, and reliance on surgical interventions without a comprehensive diagnostic foundation. This paper presents a novel Medical Computer-Aid-Diagnosis System—the Shadow Learning System framework—which integrates a convolutional neural network (CNN) with an Explainable AI method. This system not only performs conventional computer-aiding-diagnosis functions but also uniquely exploits misclassified data samples to provide additional medically relevant information from the machine learning model’s perspective, assisting doctors in their diagnostic process. The implementation of XAI techniques in our proposed system goes beyond merely validating CNN model results; it also enables the extraction of valuable information from medical images through an unconventional machine learning perspective. Our paper aims to enhance and extend the general structure and detailed design of the Shadow Learner System, making it more advantageous not only for human users but also for the deep learning model itself. A case study on femoral head osteonecrosis was conducted using our proposed system, which demonstrated improved accuracy and reliability in its prediction results. Experimental results interpreted using XAI methods are visualized to prove the confidence of our proposed model that generates reasonable results, confirming the effectiveness of the proposed model.