Development and quasi-experimental evaluation of a large language model-based automated feedback system for nursing innovation pitches

Aim: To develop and evaluate an LLM-based system for grading nursing students' pitch scripts and providing actionable feedback to enhance innovation training. Background: AI is transforming healthcare, requiring nurses to collaborate in multidisciplinary teams and contribute to innovation. However, nursing curricula often lack communication training, hindering nurses’ ability to advocate for change. Elevator pitches can help, but faculty shortages and subjective grading limit student progress. Large Language Models offer a solution by automating grading and delivering personalized feedback. Design: A Quasi-experimental study. Methods: A two-phase study was conducted. In Phase 1, four LLMs were fine-tuned (n = 178) and tested (n = 134) on grading pitch scripts using few-shot learning. Performance was measured by Root Mean Square Error (RMSE), Mean Absolute Error (MAE) and the coefficient of determination (R²). The generated suggestions were compared with instructor feedback. Phase 2 involved a quasi-experimental evaluation with an intervention group (n = 215) and a control group (n = 317). Results: GPT-4o-mini achieved the lowest RMSE (2.81), MAE (2.24) and the highest R² (0.3913), excelling in problem identification (RMSE=0.84) and solution clarity (MSE= 0.66). LLMs demonstrated significant advantages over manual evaluation in providing feedback. In Phase 2, the intervention group (Mean=19.68, SD 2.38) achieved significantly higher pitch scores than the control group (Mean=17.30, SD 3.92) (P < 0.001). Additionally, students reported positive experiences using the system. Conclusions: The LLM-based pitch grading system accurately evaluates nursing students' pitch scripts and provides valuable, objective feedback. This AI-driven approach has significant potential to enhance communication skills training in nursing education, thereby fostering nurse-led innovation in healthcare.