Post-secondary data literacy education is critical to students’ academic and career success. However, the literature has not adequately addressed the conceptualization and assessment of data literacy for post-secondary students. In this study, we introduced a novel digital performance-based assessment for teaching and evaluating post-secondary students’ data literacy skills. The purpose of this study is to validate the assessment and identify problematic items for later modifications using the argument-based approach to validation. We analyzed students’ item responses and eye movements using psychometric and eye-tracking analyses to collect two types of validity evidence: internal structure and response processes. Descriptive and psychometric results showed that the nine example items measuring basic data analysis were of acceptable psychometric quality. The eye-tracking analysis of two representative items indicated that most students first attended to and processed expected item areas when the items were available. In addition, the critical item regions for task success were associated with students’ highest cognitive effort. These results rejected our alternative score interpretation that the developed assessment questions evaluate students’ abilities that are weakly connected to the skill of basic data analysis. Possible explanations of the findings and theoretical and pedagogical implications of our study were discussed.