Development and Testing of Screen-Based and Psychometric Instruments for Assessing Resident Performance in an Operating Room Simulator
Table 3
Interrater agreement for the Global Rating Scale.
Item
Circuit disconnect
Bradycardia
Endobronchial intubation
Hypovolemia
Pulmonary embolism
Light anesthesia
All scenarios
Overall performance
0.756
0.900
0.772
0.789
0.908
0.844
0.804
State change detection
0.796
0.907
0.883
0.888
0.828
0.537
0.819
Situational awareness
0.889
0.848
0.872
0.673
0.964
0.835
0.866
Therapy/resource utilization
0.756
0.798
0.739
0.825
0.805
0.867
0.787
Subject perception of crisis resolution
0.683
0.633
0.740
0.529
0.414
0.538
0.624
Total
0.798/0.852
0.899/0.906
0.812/0.860
0.849/0.838
0.892/0.929
0.760/0.826
0.825/0.856
Interrater agreement assessed by calculating the intraclass correlation coefficient (ICC) using a two-way mixed effects for consistency between two expert rater responses. Not significant. The second ICC does not include the “subject perception of crisis resolution” item.