Applied Psychological Measurement

Sinharay, Sandip; Johnson, Matthew S.

doi: 10.1177/01466216251316559pmid: 39881856

This article suggests a new approach based on Bayesian decision theory (e.g., Cronbach & Gleser, 1965; Ferguson, 1967) for detection of test fraud. The approach leads to a simple decision rule that involves the computation of the posterior probability that an examinee committed test fraud given the data. The suggested approach was applied to a real data set that involved actual test fraud.

Trout, Nicholas; Gorney, Kylie

doi: 10.1177/01466216251322353pmid: 40041094

Romero et al. (2015; see also Wollack, 1997) developed the ω statistic as a method for detecting unusually similar answers between pairs of examinees. For each pair, the ω statistic considers whether the observed number of similar answers is significantly larger than the expected number of similar answers. However, one limitation of ω is that it does not account for the particular items on which similar answers are observed. Therefore, in this study, we propose a weighted version of the ω statistic that takes this information into account. We compare the performance of the new and existing statistics using detailed simulations in which several factors are manipulated. Results show that while both the new and existing statistics are able to control the Type I error rate, the new statistic is more powerful, on average.

Smith, Nate R.; Keller, Lisa A.; Feinberg, Richard A.; Liu, Chunyan

doi: 10.1177/01466216251320403pmid: 39989924

Item preknowledge refers to the case where examinees have advanced knowledge of test material prior to taking the examination. When examinees have item preknowledge, the scores that result from those item responses are not true reflections of the examinee’s proficiency. Further, this contamination in the data also has an impact on the item parameter estimates and therefore has an impact on scores for all examinees, regardless of whether they had prior knowledge. To ensure the validity of test scores, it is essential to identify both issues: compromised items (CIs) and examinees with preknowledge (EWPs). In some cases, the CIs are known, and the task is reduced to determining the EWPs. However, due to the potential threat to validity, it is critical for high-stakes testing programs to have a process for routinely monitoring for evidence of EWPs, often when CIs are unknown. Further, even knowing that specific items may have been compromised does not guarantee that any examinees had prior access to those items, or that those examinees that did have prior access know how to effectively use the preknowledge. Therefore, this paper attempts to use response behavior to identify item preknowledge without knowledge of which items may or may not have been compromised. While most research in this area has relied on traditional psychometric models, we investigate the utility of an unsupervised machine learning algorithm, extended isolation forest (EIF), to detect EWPs. Similar to previous research, the response behavior being analyzed is response time (RT) and response accuracy (RA).