To identify the chemical components responsible for the anti-hyperglycemic effect of Cyclocarya paliurus (Batal.) Iljinsk (Juglandaceae) leaves, an ethanol extract (CPE) and a water extract (CPW) of C. paliurus leaves, as well as their total flavonoids (CPF), triterpenoids (CPT) and crude polysaccharides (CPP), were prepared and assessed on streptozotocin (STZ)-induced diabetic mice. After being orally administrated once a day for 24 days, CPF (300 mg/kg), CPP (180 mg/kg), or CPF+CPP (300 mg/kg CPF + 180 mg/kg CPP) treatment reversed STZ-induced body weight and muscle mass losses. The glucose tolerance tests and insulin tolerance tests suggested that CPF, CPP, and CPF+CPP showed anti-hyperglycemic effect in STZ-induced diabetic mice. Furthermore, CPF enhances glucose-stimulated insulin secretion in MIN6 cells and insulin-stimulated glucose uptake in C2C12 myotubes. CPF and CPP suppressed inflammatory cytokine levels in STZ-induced diabetic mice. Additionally, CPF and CPP improved STZ-induced diabetic nephropathy assessed by H&E staining, blood urea nitrogen content, and urine creatinine level. The molecular networking and Emperor analysis results indicated that CPF showed potential anti-hyperglycemic effects, and HPLC–MS/MS analysis indicated that CPF contains 3 phenolic acids and 9 flavonoids. In contrast, CPT (650 mg/kg) and CPC (300 mg/kg CPF + 180 mg/kg CPP + 650 mg/kg CPT) did not show anti-hyperglycemic effect. Taken together, polysaccharides and flavonoids are responsible for the anti-hyperglycemic effect of C. paliurus leaves, and the clinical application of C. paliurus need to be refined.