Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta
Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt
BACKGROUND Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. METHODS This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard. RESULTS Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed. CONCLUSION Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.
Originalsprog | Engelsk |
---|---|
Tidsskrift | Journal of Trauma and Acute Care Surgery |
Vol/bind | 91 |
Udgave nummer | 4 |
Sider (fra-til) | 663-671 |
ISSN | 2163-0755 |
DOI | |
Status | Udgivet - 2021 |
Bibliografisk note
Funding Information:
Copenhagen Academy for Medical Education and Simulation has received a PhD grant for author M.E. from Neurescue, a Danish REBOA company, which partly funded this study. Neureuscue did not participate in the design, data collection, analysis, or writing of the article. M.T. is a clinical advisor for Neurescue. L.L. is medical director for Mentice. The remaining authors declared conflicts of interest.
Publisher Copyright:
© 2021 Wolters Kluwer Health, Inc. All rights reserved.
ID: 284198679