Rapidly developing science and technology has led to a growing body of research on automated essay scoring (AES), simply defined as the assessment and scoring of written prose via a computer (Page, 2003; Shermis & Burstein, 2003), over the last few decades in the field of language assessment. The pioneering instructional application of AES, Project Essay Grade, was developed in the mid-1960s (Page, 1994).
Later on, new applications were created including IntelliMetric by Vintage Learning, Intelligent Essay Assessor (IEA) by Pearson Education Inc., and E-rater by Educational Testing Service (ETS), together with other instructionally-oriented systems such as WriteToLearn, MY Access, and Pigai. In addition to AES, automated diagnostic feedback is also offered by these abovementioned systems. With the combination of scoring and evaluating, these systems are also referred to as automated writing evaluation (AWE) systems (Liu & Kunnan, 2016; McNamara et al., 2015).
The last decade has witnessed an upsurge in applying AWE systems to assorted tests (Hockly, 2019). They have been predominantly utilized in the evaluation of large-scale standardized tests for native English speakers (NES). However, the research on non-native English speakers (NNES), especially Chinese test takers, which hosts probably the largest ESL student population, is still relatively scarce.
Furthermore, previous studies have emphasized the importance of classroom-based assessments but may ignore that large-scale assessments are also of great significance. Besides, empirical studies have mainly
touched upon the application of systems such as WriteToLearn (e.g., Liu & Kunnan, 2016) or MY Access (e.g., Hoang & Kunnan, 2016), very few have concentrated on the Pigai (Lu, 2016), developed especially for Chinese ESL learners (Bai & Hu, 2017). So far, it has been used in over 1,000 schools or universities in China. The Foreign Language Teaching and Research Press (FLTRP) even uses it for the annual writing competition (Guo, 2020). Due to its increasing influence on Chinese ESL learners and English learning, the reliability and validity of Pigai do merit further examination. This study then investigated the application of Pigai by examining the correlation between writing scores given by both Pigai and two human raters, and the accuracy of its error feedback at the linguistic level compared with the human raters.The results would probably provide preliminary feedback in classroom instruction.