Researches have shown that silica nanoparticles (SiNPs) could reduce both the quantity and quality of sperm. However, the mechanism of toxicity induced by SiNPs in the male reproductive system is still unclear. In this study, male mice were randomly divided into a control group, and SiNPs treated group (20 mg/kg dose; n = 30 per group). Half of the mice per group were sacrificed on 35 days and the remaining on 50 days of the SiNPs exposure. SiNPs were found to decrease sperm count and mobility, increase the sperm abnormality rate, and damage the testes' structure. Furthermore, SiNPs decreased the protein levels of Protamine 1(PRM1) and elevated the histones' levels and suppressed the chromatin condensation of sperm. There was a significant reduction of the ubiquitinated H2A (ubH2A)/H2B (ubH2B) and RING finger protein 8 (RNF8) levels in the spermatid nucleus, while the RNF8 level in the spermatid cytoplasm increased evidently. The protein expression levels of PIWI-like protein 1(MIWI) in the late spermatids significantly increased on day 35 of SiNPs exposure. After 15 days of the withdrawal, the sperm parameters and protamine levels, and histones in the epididymal sperm were unrecovered; however, the changes in testis induced by SiNPs were recovered. Our results suggested that SiNPs could decrease the RNF8 level in the nucleus of spermatid either by upregulating of the expression of MIWI or by inhibiting its degradation. This resulted in the detention of RNF8 in the cytoplasm that maybe inhibited the RNF8-mediated ubiquitination of ubH2A and ubH2B. These events culminated in creating obstacles during the H2A and H2B removal and chromatin condensation, thereby suppressing the differentiation of round spermatids and chromatin remodeling, which compromised the sperm quality and quantity.