In Saccharomyces cerevisiae, cryptic transcription at the coding region is prevented by the activity of Sin3 histone deacetylase (HDAC) complex Rpd3S, which is carried by the transcribing RNA polymerase II (RNAPII) to deacetylate and stabilize chromatin. Despite its fundamental importance, the mechanisms by which Rpd3S deacetylates nucleosomes and regulates chromatin dynamics remain elusive. Here, we determined several cryo-EM structures of Rpd3S in complex with nucleosome core particles (NCP), including the H3/H4 deacetylation states, the alternative deacetylation state, the linker tightening state, and a state in which Rpd3S co-exists with the Hho1 linker histone on NCP. These structures suggest that Rpd3S utilizes a conserved Sin3 basic surface to navigate through the nucleosomal DNA, guided by its interactions with H3K36 methylation and the extra-nucleosomal DNA linkers, to target acetylated H3K9 and sample other histone tails. Furthermore, our structures illustrate that Rpd3S reconfigures the DNA linkers and acts in concert with Hho1 to engage the NCP, potentially unraveling how Rpd3S and Hho1 work in tandem for gene silencing.