Chinese medicinal material is the foundation of traditional Chinese medicine (TCM) industry. Its quality is not only closely related to the health of residents but also the key to the development of the TCM industry. Pesticide residues, heavy metals and mycotoxins are the major pollutants of Chinese medicinal materials. In recent years, quite a number of rapid detection methods for pollutants have been constructed. Among them, surface-enhanced Raman scattering (SERS), which has been widely used in food chemistry, environmental analysis, and other fields because of its speediness and non-destructiveness, shows its great potential in the pollutant detection in Chinese medicinal material. This paper firstly reviews the application of SERS for the detection of common pollutants in Chinese medicinal material. We then discussed the characteristics and advantages of SERS technique for pesticide detection, including the principle, SERS substrate design, specific recognition, etc. Finally, simultaneous detection of multiple pesticide residues in Chinese medicinal material was explored.