The intrinsic brittleness of cement hydrates can bring about the potential deterioration of concrete performance, affecting the service life of cement hydrates. In this study, we introduced polyvinylpyrrolidone (PVP) coupled with in situ polymerized acrylic acid (AA) to toughen conventional cement by constructing a polymer network in the cement matrix. The flexible polymer network, intermingled with cement hydrates, offered softness to the stiff cement matrix, significantly improving its flexural strength and toughness. Compared with the control cement paste, the AA-PVP-modified cement paste displayed an increment in flexural strength up to 86% with the comparable compressive strength to the control one by adjusting the AA-PVP fraction. Cement hydration promoted the polymerization reaction of AA by releasing heat, which ensured a synergistic effect between cement hydration and polymerization. Chemical interactions between carboxyl groups and metal ions (Ca2+ and Al3+) contributed to the improvement in mechanical strength. This method, the combination of polymer and in situ polymerization of monomers, exhibited extensive application prospects in toughening cement-based materials.