Anharmonicity of phonons correlates with less dispersive potential surfaces and usually governs the thermal transport of low-dimensional materials. Here, we demonstrate the significant role of the so-called “rattling” action in affecting lattice anharmonicity, originating from the ease of freedom of confined but loose atoms in two-dimensional space. Based on calculations of X₂Si₂Te₆ (X = Sb and Bi) within the Peierls–Boltzmann framework, the degree of high-order four-phonon scattering differs strikingly despite their isostructural feature. Upon switching on four-phonon scattering, a significant drop of thermal conductivity (κ_ph) occurs in Bi₂Si₂Te₆ up to 43.15% (71.62%) at 300 K (1000 K), while a moderate reduction occurs for Sb₂Si₂Te₆. This arises from a stronger quartic anharmonicity of Bi₂Si₂Te₆ than Sb₂Si₂Te₆, dominated by the redistribution four-phonon process (λ + λ′ → λ″ + λ‴). We show that the strong quartic anharmonicity is more likely to occur in systems with flat phonon bands, large atoms, and rattling atomic units. These new insights provide perspectives in the design of materials with low κ_ph through introducing rattling units in layered materials or interfaces.