Knowledge distillation aims to improve the performance of a smaller student model by transferring knowledge from a larger teacher model. Traditional methods focus on training the student to mimic the teacher’s output activations. In this paper, we propose a multi-model training framework, MPTF (multi-model pyramid training framework), and a novel loss function, C-LOSS, to enhance the student model’s ability to distinguish between similar classes. By appropriately sacrificing storage and computation time, our framework merges categories with similar features into super categories during the initial training phase, followed by refining the classification of these super categories. The C-LOSS function is introduced to optimize intra-class and inter-class distances to improve discriminative ability. Experiments on CIFAR-10, CIFAR-100, and EuroSAT datasets show that our method significantly improves overall accuracy and confusing class accuracy. Despite these advancements, further research is needed to refine the segmentation of super categories and enhance the scalability of our framework. This study demonstrates the potential of staged training strategies and optimized loss functions in improving the performance of small models in fine-grained classification tasks.