Research Article

Development and Testing of Screen-Based and Psychometric Instruments for Assessing Resident Performance in an Operating Room Simulator

Table 3

Interrater agreement for the Global Rating Scale.

ItemCircuit disconnectBradycardiaEndobronchial intubationHypovolemiaPulmonary embolismLight anesthesiaAll scenarios

Overall performance0.7560.9000.7720.7890.9080.8440.804
State change detection0.7960.9070.8830.8880.8280.5370.819
Situational awareness0.8890.8480.8720.6730.9640.8350.866
Therapy/resource utilization0.7560.7980.7390.8250.8050.8670.787
Subject perception of crisis resolution0.6830.6330.7400.5290.4140.5380.624
Total0.798/0.8520.899/0.9060.812/0.8600.849/0.8380.892/0.9290.760/0.8260.825/0.856

Interrater agreement assessed by calculating the intraclass correlation coefficient (ICC) using a two-way mixed effects for consistency between two expert rater responses.
Not significant.
The second ICC does not include the “subject perception of crisis resolution” item.