(Virtual) Statistics Seminar

Texas State University

Progress toward NLP-assisted formative assessment feedback in large classes

Abstract. This talk seeks to articulate the benefit of free-response tasks and timely formative assessment feedback, a roadmap for developing human-in-the-loop natural language processing (NLP) assisted feedback, and results from a pilot study establishing proof of principle. If we are to pursue Statistics and Data Science Education across disciplines, we will surely encounter both opportunity and necessity to develop scalable solutions for pedagogical best practices. Research suggests “write-to-learn” tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy when class sizes grow large. In the pilot study, several short-answer tasks completed by nearly 2000 introductory tertiary statistics students were evaluated by human raters and an NLP algorithm. After briefly describing the tasks, the student contexts, the algorithm and the raters, this talk discusses the results which indicate substantial inter-rater agreement and group consensus. The talk will conclude with recent developments building upon this pilot, as well as implications for teaching and future research.

Resources

• Slides (PDF)
• ArXiv Preprint (link): Beckman, Burke, Fiochetta, Fry, Lloyd, Patterson, Tang, (2024). Developing Consistency Among Undergraduate Graders Scoring Open-Ended Statistics Tasks. Preprint URL: https://arxiv.org/abs/2410.18062
• EMNLP Paper: Li, Z., Lloyd, S., Beckman, M. D., & Passonneau, R. J. (2023). Answer-state Recurrent Relational Network (AsRRN) for Constructed Response Assessment and Feedback Grouping. Findings of the Association for Computational Linguistics: EMNLP 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.254
• ICOTS Paper (PDF): Lloyd, S. E., Beckman, M., Pearl, D., Passonneau, R., Li, Z., & Wang, Z. (2022). Foundations for AI-Assisted Formative Assessment Feedback for Short-Answer Tasks in Large-Enrollment Classes. In Proceedings of the eleventh international conference on teaching statistics. Rosario, Argentina.
• Project CLASSIFIES Website (link)

Acknowledgment

• We’re grateful to the US National Science Foundation for funding this research (NSF DUE-2236150: Project CLASSIFIES)

Contact & Short Bio

• email: mdb268 [at] psu [dot] edu
  
• personal webpage: https://mdbeckman.github.io/
  
• CAUSE webpage: https://www.causeweb.org

Matt Beckman is an Associate Research Professor of Statistics at Penn State University and Director of the Consortium for the Advancement of Undergraduate Statistics Education (CAUSE). He is co-founder of a Statistics & Data Science Education Research Lab at Penn State, and his primary research interests tend to focus on assessment. Matt is currently PI for the NSF-funded “Project CLASSIFIES” which investigates the use of NLP tools to assist instructors of large-enrollment classes with providing formative assessment feedback on short-answer, free response tasks.