A Comparison of Different Designs in Scoring of PISA 2009 Reading Open Ended Items According to Generalizability Theory

Authors : Meral ALKAN, Nuri DOĞAN

Pages : 106-117

Doi:10.21031/epod.1210917

View : 93 | Download : 81

Publication Date : 2023-06-30

Article Type : Research Paper

Abstract :This study compares the different designs obtained through four raters’ scoring the open-ended items used in PISA 2009 reading literacy altogether or alternately according to the Generalizability Theory. The sample of the research was composed of 362 students insert ignore into journalissuearticles values(out of 4996 students participating in PISA 2009); who responded to the items of reading skills and who were scored by more than one rater. Two designs were created so as to be used in generalizability theory in the study. One of them was the crossed design symbolized as “s x i x r” insert ignore into journalissuearticles values(student x item x rater);, in which students are scored by each rater in terms of the same skills. The second was the nested design symbolized as “insert ignore into journalissuearticles values(r:s); x i”, where each rater scored only a group of students and raters are nested in students and the items were crossed with these variables. On comparing the s x i x r design with insert ignore into journalissuearticles values(r:s); x i design, it was found that the relative and absolute error variances estimated for insert ignore into journalissuearticles values(r:s); x i design were smaller than those for s x i x r design and that therefore the G and Phi coefficients took on bigger values. On increasing the number of raters in both designs, the G and Phi coefficients also increased in the D study. While acceptable values of G and Phi coefficients were reached on reducing the number of raters by half in Booklet 2, raising the number of raters seemed more appropriate in Booklet 8.
Keywords : Generalizability theory, reliability, G study, D study, PISA 2009

ORIGINAL ARTICLE URL