TY - JOUR
T1 - Construct validity of the endoscopic sinus surgery simulator II. Assessment of discriminant validity and expert benchmarking
AU - Fried, Marvin P.
AU - Sadoughi, Babak
AU - Weghorst, Suzanne J.
AU - Zeltsan, Michael
AU - Cuellar, Hernando
AU - Uribe, José I.
AU - Sasaki, Clarence T.
AU - Ross, Douglas A.
AU - Jacobs, Joseph B.
AU - Lebowitz, Richard A.
AU - Satava, Richard M.
N1 - Copyright:
Copyright 2008 Elsevier B.V., All rights reserved.
PY - 2007/4
Y1 - 2007/4
N2 - Objectives: To establish discriminant validity of the endoscopic sinus surgery simulator (ES3) (Lockheed Martin, Akron, Ohio) between various health care provider experience levels and to define benchmarking criteria for skills assessment. Design: Prospective multi-institutional comparison study. Setting: University-based tertiary care institution. Participants: Ten expert otolaryngologists, 14 otolaryngology residents, and 10 medical students. Interventions: Subjects completed the ES3's virtual reality curriculum (10 novice mode, 10 intermediate mode, and 3 advanced mode trials). Performance scores were recorded on each trial. Performance differences were analyzed using analysis of variance for repeated measures (experience level as between-subjects factor). Main Outcome Measures: Simulator performance scores, accuracy, time to completion, and hazard disruption. Results: The novice mode accurately distinguished the 3 groups, particularly at the onset of training (mean scores: senior otolaryngologists, 66.0; residents, 42.7; students, 18.3; for the paired comparisons between groups 1 and 2 and groups 1 and 3, P=.04 and .03, respectively). Subjects were not distinguished beyond trial 5. The intermediate mode only discriminated students from other subjects (P = .008). The advanced mode did not show performance differences between groups. Scores on the novice mode predicted those on the intermediate mode, which predicted advanced mode scores (r=0.687), but no relationship was found between novice and advanced scores. All groups performed equally well and with comparable consistency at the outset of training. Expert scores were used to define benchmark criteria of optimal performance. Conclusions: This study completes the construct validity assessment of the ES3 by demonstrating its discriminant capabilities. It establishes expert surgeon benchmark performance criteria and shows that the ES3 can train novice subjects to attain those. The refined analysis of trial performance scores could serve educational and skills assessment purposes. Current studies are evaluating the transfer of surgical skills acquired on the ES3 to the operating room (predictive validity).
AB - Objectives: To establish discriminant validity of the endoscopic sinus surgery simulator (ES3) (Lockheed Martin, Akron, Ohio) between various health care provider experience levels and to define benchmarking criteria for skills assessment. Design: Prospective multi-institutional comparison study. Setting: University-based tertiary care institution. Participants: Ten expert otolaryngologists, 14 otolaryngology residents, and 10 medical students. Interventions: Subjects completed the ES3's virtual reality curriculum (10 novice mode, 10 intermediate mode, and 3 advanced mode trials). Performance scores were recorded on each trial. Performance differences were analyzed using analysis of variance for repeated measures (experience level as between-subjects factor). Main Outcome Measures: Simulator performance scores, accuracy, time to completion, and hazard disruption. Results: The novice mode accurately distinguished the 3 groups, particularly at the onset of training (mean scores: senior otolaryngologists, 66.0; residents, 42.7; students, 18.3; for the paired comparisons between groups 1 and 2 and groups 1 and 3, P=.04 and .03, respectively). Subjects were not distinguished beyond trial 5. The intermediate mode only discriminated students from other subjects (P = .008). The advanced mode did not show performance differences between groups. Scores on the novice mode predicted those on the intermediate mode, which predicted advanced mode scores (r=0.687), but no relationship was found between novice and advanced scores. All groups performed equally well and with comparable consistency at the outset of training. Expert scores were used to define benchmark criteria of optimal performance. Conclusions: This study completes the construct validity assessment of the ES3 by demonstrating its discriminant capabilities. It establishes expert surgeon benchmark performance criteria and shows that the ES3 can train novice subjects to attain those. The refined analysis of trial performance scores could serve educational and skills assessment purposes. Current studies are evaluating the transfer of surgical skills acquired on the ES3 to the operating room (predictive validity).
UR - http://www.scopus.com/inward/record.url?scp=34247338433&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=34247338433&partnerID=8YFLogxK
U2 - 10.1001/archotol.133.4.350
DO - 10.1001/archotol.133.4.350
M3 - Article
C2 - 17438249
AN - SCOPUS:34247338433
SN - 0886-4470
VL - 133
SP - 350
EP - 357
JO - Archives of Otolaryngology - Head and Neck Surgery
JF - Archives of Otolaryngology - Head and Neck Surgery
IS - 4
ER -