Abstract. This seminar seeks to articulate the benefit of free-response tasks and timely formative assessment feedback, a roadmap for developing human-in-the-loop natural language processing (NLP) assisted feedback, and results from a pilot study establishing proof of principle. If we are to pursue Statistics and Data Science Education across disciplines, we will surely encounter both opportunity and necessity to develop scalable solutions for pedagogical best practices. Research suggests “write-to-learn” tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy when class sizes grow large. In the pilot study, several short-answer tasks completed by nearly 2000 introductory tertiary statistics students were evaluated by human raters and an NLP algorithm. After briefly describing the tasks, the student contexts, the algorithm and the raters, this talk discusses the results which indicate substantial inter-rater agreement and group consensus. With compelling rater agreement, the study then examines a preliminary cluster analysis of response text as a mechanism for scalable formative assessment. The talk will conclude with recent developments building upon this pilot, as well as implications for teaching and future research.
Matthew Beckman
Associate Research Professor
Department of Statistics
office: 421C Thomas Building
email: mdb268 [at] psu [dot] edu
webpage: https://mdbeckman.github.io/
This work was supported by the National Science Foundation under Grant No. DUE-2236150. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.