Interexaminer agreement on the American Board of Emergency Medicine oral certification examination

Leonard Bianchi, E. John Gallagher, Robert Korte, Hazen P. Ham

Research output: Contribution to journalArticle

16 Scopus citations

Abstract

Study objective: One of the principal criticisms of performance assessments, particularly those using an oral format to evaluate physician performance, is the lack of interrater reliability. The objective of this study was to assess interexaminer agreement in the scoring of candidates taking the American Board of Emergency Medicine (ABEM) oral certification examination. Methods: Examiner-observer pairs independently scored individual candidates on each of the 6 simulated cases selected for the October 1999 examination. Simple proportionate agreement for "acceptable" and "unacceptable" scores on critical actions and performance ratings were used to measure interexaminer (examiner-observer) agreement. Further analyses were performed to assess interexaminer agreement at the level of the raw scores of all performance ratings. The candidate's actual score was determined entirely by the examiner administering the case. The study examiner was present only to observe and independently score the candidate. Examiners were blinded to each other's scoring. All examiner-observer, examiner-candidate, and observer-candidate pairings were uniquely randomized and occurred only once during the examination. Results: Of 564 candidates, 186 were randomly selected for scoring by examiner-observer pairs drawn from a pool of 107 trained oral examiners. Interexaminer agreement for 47 critical actions was 97% (95% confidence interval [CI] 96% to 98%), and interexaminer agreement for 68 performance ratings was 95% (95% CI 94% to 96%). Further analyses of raw scores of performance ratings indicated that 94% (95% CI 93% to 95%) of 2,648 pairs of ratings differed by 1 point or less. Conclusion: High interexaminer agreement on the scoring of the ABEM oral certification examination supports the reliability of this final step in the certification process. On the basis of these findings, ABEM has incorporated this methodology into the administration of each oral examination as an ongoing quality control measure.

Original languageEnglish (US)
Pages (from-to)859-864
Number of pages6
JournalAnnals of emergency medicine
Volume41
Issue number6
DOIs
StatePublished - Jun 1 2003

ASJC Scopus subject areas

  • Emergency Medicine

Fingerprint Dive into the research topics of 'Interexaminer agreement on the American Board of Emergency Medicine oral certification examination'. Together they form a unique fingerprint.

  • Cite this