S. Classen, A.K.A. Yarney, M. Monahan, S.M. Winter, K. Platek, A.L. Lutz

Pages: 99-108

Abstract
For studies relying on rater judgments of a vulnerable population, such as returning combat veterans, high consistency between primary and secondary raters is necessary. This study examined the inter-rater reliability of a primary and two secondary raters on scoring simulated driving assessments. Seven volunteer drivers were rated in two rounds (Round 1 n=4, Round 2 n=3). Driver performance was rated for two driving scenarios, a) ten-minute city/highway scenario, and b) six-minute suburban/residential scenario to measure driving errors. Based on a set 90% cut point for intra-class correlations (ICC), we examined each rated variable for each rater pairing. After round 1, secondary raters did not achieve the 90% cut point and participated in further training. Analysis of inter-rater reliability following round 2 showed improved inter-rater reliability, i.e., 99.3% for city/highway and 98% for the suburban/residential conditions. Our results suggest that by pinpointing incongruencies in raters’ assessment of driving errors, strategies including problem identification, verifying driving errors via the simulator playback function, and establishing parameters for rating each error, can resolve such inconsistencies to significantly improve their inter-rater reliability.

Keywords: simulator; raters; reliability