Feedback Assisted Training for Safety Planning (FAST-SP)

Safety Planning Interventions (SPIs) are a cornerstone of clinical care for suicide prevention. SPIs are evidence-based, broadly recommended for youth at risk of suicide, and delivered across diverse healthcare settings (e.g., emergency departments, pediatrics, behavioral health, inpatient units). Implementing high-quality SPIs with youth requires appropriate provider training and supervision, which involves significant resources including cost, staffing, and time that are difficult to scale and infeasible in many healthcare settings. The objective of this proposal is to address this resource deficit through advances in Large Language Models (an Artificial Intelligence application) to develop a cost-effective, automated tool to support provider training in the delivery of SPIs for youth suicide prevention. This provider-facing tool can rapidly analyze audio recordings of providers SPI sessions and generate 1) an assessment of the quality of the intervention, and 2) feedback on clinical skill and delivery to support training and promote improvement in the implementation of SPIs. The prototype version of our LLM tool rated the quality of dozens of SPI sessions with youth, showing 95% agreement with expert clinician ratings. We now propose the next critical steps to move this work toward clinical utility. We aim to refine our SPI quality framework with consultation from national domain experts (n=3-5) and adaptation of existing SPI quality rating systems used with adults. We aim to capture critical and unique features of safety planning for youth (e.g., caregiver involvement in crisis supports and lethal means restriction, caregiver collaboration on the development of the safety plan) (Aim 1a) and refine the LLM algorithm using the revised framework (Aim 1b). We will validate the quality framework and LLM algorithm (Aim 2) by recruiting expert clinicians (n=6-8) to rate recorded SPI sessions conducted with youth through clinical trials in pediatric medicine and inpatient settings, and routine care in an outpatient clinic at Western Psychiatric Hospital. The same SPI sessions will be analyzed by the LLM, and we predict greater than 80% concordance between expert clinician and LLM-generated quality ratings. We will also assess preliminary acceptability, feasibility, and efficacy of the LLM tool in an outpatient behavioral health clinic (Aim 3). We will recruit clinician trainees (n=4-6), and licensed staff clinicians supervisors (n=6-8) to participate in qualitative interviews and rate the acceptability of an AI-based training tool, and the quality and relevance of the clinical feedback. We will also assess feasibility of using the LLM tool with trainees and staff in the clinic workflow and incorporating the feedback into clinical supervision. Finally, we will test the efficacy of the tool for training. Clinical trainees will use the LLM tool on their own SPI sessions conducted during routine clinical care. Trainees will review the quality ratings and feedback with their supervisors, and we will descriptively evaluate whether quality scores improve. The results of this study will critically inform the utility of an automated provider-facing training support tool for SPI implementation. This tool has the potential to be deployed across diverse healthcare settings to equip providers with supports to deliver high-quality interventions.