Joint Mathematics Meetings

Washington DC

Evaluating NLP tools designed to assist instructors with formative assessment for large-enrollment STEM classes

Abstract. This talk seeks to articulate the benefit of free-response tasks and timely formative assessment feedback, a roadmap for developing human-in-the-loop natural language processing (NLP) assisted feedback, and results from a pilot study establishing proof of principle. If we are to pursue Statistics and Data Science Education across disciplines, we will surely encounter both opportunity and necessity to develop scalable solutions for pedagogical best practices. Research suggests that “write-to-learn” tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy when class sizes grow large. Several short-answer tasks completed by post-secondary introductory statistics students; the responses were then evaluated by human raters and an NLP algorithm. After briefly describing the tasks, the student contexts, the algorithm and the raters, this talk discusses the results which indicate substantial inter-rater agreement and group consensus. The talk will conclude with recent developments, as well as implications for teaching and future research.

Resources

• Slides (PDF)
• ArXiv Preprint (link): Beckman, Burke, Fiochetta, Fry, Lloyd, Patterson, Tang, (in review). Developing Consistency Among Undergraduate Graders Scoring Open-Ended Statistics Tasks. Preprint URL: https://arxiv.org/abs/2410.18062
• EMNLP Paper (PDF): Li, Z., Lloyd, S., Beckman, M. D., & Passonneau, R. J. (2023). Answer-state Recurrent Relational Network (AsRRN) for Constructed Response Assessment and Feedback Grouping. Findings of the Association for Computational Linguistics: EMNLP 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.254
• Pilot Study (PDF): Lloyd, S. E., Beckman, M., Pearl, D., Passonneau, R., Li, Z., & Wang, Z. (2022). Foundations for AI-Assisted Formative Assessment Feedback for Short-Answer Tasks in Large-Enrollment Classes. In Proceedings of the eleventh international conference on teaching statistics. Rosario, Argentina.
• ArXiv Preprint (link): Wei, Y., Pearl, D., Beckman, M., Passonneau, R. (2025). Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis. Preprint URL: https://arxiv.org/pdf/2504.03877
• JMM Session Description (link)–NSF Special Session on Outcomes and Innovations from NSF Undergraduate Education Programs in the Mathematical Sciences

Acknowledgments

• US National Science Foundation (NSF DUE-2236150: Project CLASSIFIES)
• Penn State Center for Socially Responsible Artificial Intelligence
• Strategic partnership between University of Auckland and Penn State University

Contact

Matthew Beckman
Assoc Research Professor | Penn State University
Director | CAUSE

email: mdb268 [at] psu [dot] edu
personal webpage: https://mdbeckman.github.io/
CAUSE webpage: https://www.causeweb.org