GENERALIZABILITY ANALYSIS OF THE READING SUBTEST IN AN ENGLISH DIAGNOSTIC TEST

Authors : Arida Susyetina, Raden Rosnawati

Pages : 57-79

Doi:10.35452/caless.1725051

View : 64 | Download : 110

Publication Date : 2025-12-31

Article Type : Research Paper

Abstract :Ensuring measurement precision in diagnostic language assessment remains a critical challenge for educational institutions worldwide. This study employs Generalizability Theory (G-Theory) with a fully crossed person × item (p × i) design to evaluate the reliability of a reading comprehension subtest within a diagnostic English test used for university admission placement. Using R statistical software with the gtheory package, responses from 63 prospective university students across 15 multiple-choice reading items were analyzed through complementary G-study and D-study frameworks. The G-study decomposed total score variance into three components: persons (11.2%), items (7.0%), and residual error combining person-by-item interaction with random error (81.8%). This variance structure reveals significant psychometric limitations—the low person variance falls substantially short of the 40% benchmark for high-stakes decisions, while the dominant residual component indicates considerable measurement inconsistency. Reliability analysis yielded a generalizability coefficient (Eρ²) of 0.67 and dependability coefficient (Φ) of 0.65, both failing to meet conventional thresholds of 0.80 and 0.75 for consequential educational decisions. However, D-study simulations demonstrate a clear pathway for improvement: extending the test to 25–30 items would elevate both coefficients above acceptable standards while remaining feasible within institutional constraints. These findings indicate that the reading subtest, in its current 15-item form, does not meet minimal quality standards for high-stakes placement use and requires targeted revision. This study demonstrates that G-Theory provides an accessible yet rigorous framework for diagnostic test evaluation, offering evidence-based guidance for enhancing assessment quality even within resource-constrained institutional settings.
Keywords : Genellenebilirlik kuramı, Tanılayıcı değerlendirme, Okuma testi, Güvenirlik, Eğitimde ölçme, Karar çalışması

ORIGINAL ARTICLE URL