Drug combination therapy is a highly effective approach for enhancing the therapeutic efficacy of anti-cancer drugs and overcoming drug resistance. However, the innumerable possible drug combinations make it impractical to screen all synergistic drug pairs. Moreover, biological insights into synergistic drug pairs are still lacking. To address this challenge, we systematically analyzed drug combination datasets curated from multiple databases to identify drug pairs more likely to show synergy. We classified drug pairs based on their MoA and discovered that 110 MoA pairs were significantly enriched in synergy in at least one type of cancer. To improve the accuracy of predicting synergistic effects of drug pairs, we developed a suite of machine learning models that achieve better predictive performance. Unlike most previous methods that were rarely validated by wet-lab experiments, our models were validated using two-dimensional cell lines and three-dimensional tumor slice culture (3D-TSC) models, implying their practical utility. Our prediction and validation results indicated that the combination of the RTK inhibitors Lapatinib and Pazopanib exhibited a strong therapeutic effect in breast cancer by blocking the downstream PI3K/AKT/mTOR signaling pathway. Furthermore, we incorporated molecular features to identify potential biomarkers for synergistic drug pairs, and almost all potential biomarkers found connections between drug targets and corresponding molecular features using protein-protein interaction network. Overall, this study provides valuable insights to complement and guide rational efforts to develop drug combination treatments.