Identifying data conditions to enhance subscale score accuracy based on various psychometric models

Abstract

As a result of the requirements in the NCLB Act of 2001, subscale score reporting has drawn much attention from educational researchers and practitioners. Subscale score reporting has an important diagnostic value because it can give information about respondents’ cognitive strengths and weaknesses in specific content domains. Although several testing programs have reported their results in subscales, there have been many concerns about the reported subscale scores due to their lack of appropriate psychometric quality, especially in reliability. Various subscale scoring methods have been proposed to overcome the lack of reliability (Monaghan, 2006; Haberman, 2008). However, their efficiency in subscale scoring seems to fluctuate under different data conditions. The current study seeks the optimal data conditions for maximizing reliability or accuracy of subscale scores using CTT- and IRT-based methods. Both real-world data and simulation data are used to compute subscale scores, and their accuracies of these estimations (i.e., reliability) are compared. For a real-world data study, response data of a math achievement test from 5,000 eighth grade students in a Midwestern state are used. For the simulation study, response data are generated varying the subscale length, between-subscales correlations, within-subscale correlations, and level of item difficulty. Each data condition has 100 replications.