Comparison of Inter-Rater Reliability Techniques in Performance-Based Assessment

Authors : Sinem ARSLAN MANCAR, Hamide Deniz GÜLLEROĞLU

Pages : 515-533

Doi:10.21449/ijate.993805

View : 29 | Download : 20

Publication Date : 2022-06-26

Article Type : Research Paper

Abstract :The aim of this study is to analyse the importance of the number of raters and compare the results obtained by techniques based on Classical Test Theory insert ignore into journalissuearticles values(CTT); and Generalizability insert ignore into journalissuearticles values(G); Theory. The Kappa and Krippendorff alpha techniques based on CTT were used to determine the inter-rater reliability. In this descriptive research data consists of twenty individual investigation performance reports prepared by the learners of the International Baccalaureate Diploma Programme insert ignore into journalissuearticles values(IBDP); and also five raters who rated these reports. Raters used an analytical rubric developed by the International Baccalaureate Organization insert ignore into journalissuearticles values(IBO); as a scoring tool. The results of the CTT study show that Kappa and Krippendorff alpha statistical techniques failed to provide information about the sources of the errors causing incompatibility in the criteria. The studies based on G Theory provided comprehensive data about the sources of the errors and increasing the number of raters would also increase the reliability of the values. However, the raters raised the idea that it is important to develop descriptors in the criteria in the rubric.
Keywords : Inter rater reliability, Performance based assessment, Generalizability theory, International baccalaureate diploma programme, Scientific literacy