Over the past 20 years (1999–2019), we have witnessed a rapid increase in publications involving chemical macroautophagy/autophagy modulators. However, an overview of the methodologies used in these studies is still lacking, and methodology flaws are frequently observed in some reports. To provide an objective and quantitative analysis of studies involving autophagy modulators, we present an Autophagy Modulator Scoring System (AMSS), which is designed to evaluate methodological integrity. AMSS-A includes the autophagy characterization by 4 aspects, namely, autophagosome quantification, autophagy-related biochemical changes, autophagy substrate degradation, and autophagic flux. AMSS-B contains the pharmacological and functional characteristics of chemical autophagy modulators, including lysosomal function, drug targets, autophagy-dependent pharmacological effects, and validation in multiple cell lines and in vivo models. Our analysis shows that of the 385 studies reporting chemical autophagy modulators, only 142 single studies had examined all 4 aspects of autophagy characterization in AMSS-A, and only 10 out of 142 studies had fulfilled all the AMSS criteria in a single study. A comprehensive analysis of the methodologies used in all the studies was made, along with a summary of studies that demonstrated the highest methodological integrity based on AMSS ranking. To test the reliability of the AMSS, a co-efficiency analysis of scores and co-citation values in the co-citation network was performed, and a significant co-efficiency was obtained. Collectively, AMSS provides insight into the methodological integrity of autophagy modulators studies and also offers a user-friendly toolkit to help choose appropriate assays to characterize autophagy modulators.