International comparative studies suggest that students from countries with high-stakes examinations often perform better than students from other countries. Nowadays more and more countries, including the United States, are implementing high-stakes examinations at the national or state levels. In this paper, we use the National Higher Education Entrance Examination (NHEEE) in China as an example to illustrate the premises and challenges of high-stakes examinations. NHEEE, commonly known as Gaokao, is the only measure used in China to determine if and which college a high school graduate is admitted to. This study examines the reliability and validity of scores obtained from the 2014 mathematics test of this critical examination that determines the future of thousands of students in China. Results of the Rasch analysis indicated that the unidimensionality assumption was tenable. The results also showed that the item reliability and separation were satisfactory, but the person reliability and separation were low. The low person separation reliability indicates that the exam is not sensitive enough to distinguish between low- and high-performing students. Examination of the person-item map suggested a need for more items at the intermediate and difficult levels to improve the reliability and validity of the test scores and to match the students’ ability levels. Results showed that the majority of items displayed little or no DIF between male and female students. Predictive aspects of validity are also reported.