Nontechnical skills are critical for good anesthetic practice but are seldom addressed explicitly in clinical training. The purposes of this study were (1) to evaluate the reliability and validity of the observation-based assessment tool Nurse Anaesthetists' Non-Technical Skills system (N-ANTS) and (2) to evaluate the effect of training nurse anesthetist supervisors in the use of N-ANTS. This system comprises a global rating score, 4 categories, and 15 elements to rate nurse anesthetists' nontechnical skills. A 1-day workshop was conducted for 22 nurse anesthetist supervisors to rate nurse anesthetists' nontechnical skills in 9 scripted video scenarios. Data were gathered from 2 rating sessions separated by a 2-hour training session. The interrater reliability was high before and after the training. It remained stable for the global rating score in N-ANTS and its category ratings, but improved at the elements level. When the raters' ratings were compared with ratings by an expert reference group, there was no statistically significant effect on training. We conclude that Danish nurse anesthetist supervisors without further rater training besides their work experience can use N-ANTS to assess nontechnical skills of nurse anesthetists.