Interexaminer agreement on the American Board of Emergency Medicine oral certification examination

Study objective: One of the principal criticisms of performance assessments, particularly those using an oral format to evaluate physician performance, is the lack of interrater reliability. The objective of this study was to assess interexaminer agreement in the scoring of candidates taking the American Board of Emergency Medicine (ABEM) oral certification examination. Methods: Examiner-observer pairs independently scored individual candidates on each of the 6 simulated cases selected for the October 1999 examination. Simple proportionate agreement for "acceptable" and "unacceptable" scores on critical actions and performance ratings were used to measure interexaminer (examiner-observer) agreement. Further analyses were performed to assess interexaminer agreement at the level of the raw scores of all performance ratings. The candidate's actual score was determined entirely by the examiner administering the case. The study examiner was present only to observe and independently score the candidate. Examiners were blinded to each other's scoring. All examiner-observer, examiner-candidate, and observer-candidate pairings were uniquely randomized and occurred only once during the examination. Results: Of 564 candidates, 186 were randomly selected for scoring by examiner-observer pairs drawn from a pool of 107 trained oral examiners. Interexaminer agreement for 47 critical actions was 97% (95% confidence interval [CI] 96% to 98%), and interexaminer agreement for 68 performance ratings was 95% (95% CI 94% to 96%). Further analyses of raw scores of performance ratings indicated that 94% (95% CI 93% to 95%) of 2,648 pairs of ratings differed by 1 point or less. Conclusion: High interexaminer agreement on the scoring of the ABEM oral certification examination supports the reliability of this final step in the certification process. On the basis of these findings, ABEM has incorporated this methodology into the administration of each oral examination as an ongoing quality control measure.