Investigating a new method for standardising essay marking using levels-based mark schemes

Authors : Jackie GREATOREX, Tom SUTCH, Magda WERNO, Jess BOWYER, Karen DUNN

Pages : 218-234

Doi:10.21449/ijate.564824

View : 16 | Download : 11

Publication Date : 2019-07-15

Article Type : Research Paper

Abstract :Standardisation is a procedure used by Awarding Organisations to maximise marking reliability, by teaching examiners to consistently judge scripts using a mark scheme. However, research shows that people are better at comparing two objects than judging each object individually. Consequently, Oxford, Cambridge and RSA insert ignore into journalissuearticles values(OCR, a UK awarding organisation); proposed investigating a new procedure, involving ranking essays, where essay quality is judged in comparison to other essays. This study investigated the marking reliability yielded by traditional standardisation and ranking standardisation. The study entailed a marking experiment followed by examiners completing a questionnaire. In the control condition live procedures were emulated as authentically as possible within the confines of a study. The experimental condition involved ranking the quality of essays from the best to the worst and then assigning marks. After each standardisation procedure the examiners marked 50 essays from an AS History unit. All participants experienced both procedures, and marking reliability was measured. Additionally, the participants’ questionnaire responses were analysed to gain an insight into examiners’ experience. It is concluded that the Ranking Procedure is unsuitable for use in public examinations in its current form. The Traditional Procedure produced statistically significantly more reliable marking, whilst the Ranking Procedure involved a complex decision-making process. However, the Ranking Procedure produced slightly more reliable marking at the extremities of the mark range, where previous research has shown that marking tends to be less reliable.
Keywords : Comparative judgement, Marking, Standardisation, Reliability, Essay