Abstract. Research suggests “write-to-learn” tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy with large class sizes. This talk seeks to articulate the benefit of free-response tasks and timely formative assessment feedback, a roadmap for developing natural language processing (NLP) assisted feedback, and results from a pilot study establishing proof of principle. In the pilot study, several short-answer tasks completed by nearly 2000 introductory statistics students were evaluated by human raters and an NLP algorithm. Results indicate substantial inter-rater agreement using quadratic weighted kappa for rater pairs (each QWK > 0.74) and group consensus (Fleiss’ Kappa = 0.68). With compelling rater agreement, the study then introduces cluster analysis of response text as a mechanism for scalable formative assessment. The talk will conclude with implications for teaching and research building upon this work.
Matthew Beckman
Assoc Research Professor
Department of Statistics
office: 421C Thomas Building
email: mdb268 [at] psu [dot] edu
webpage: https://mdbeckman.github.io/