Improvement in Validity and Reliability of Airway Assessment Using Ultrasonography by Novice Personnel: A Learning Cohort Study

Introduction

Complications in airway management are often unanticipated and can lead to disability or death. The most common complication is an unexpected encounter of a difficult airway, posing challenges in achieving adequate intubation and ventilation by anesthesiologists. Thus, accurate airway assessment is necessary for predicting difficulties in intubation and ventilation, as well as preparing the assistants and the equipment needed to secure the airway. The use of a correct technique for airway management can reduce the risk of severe complications.^1–3

Airway can be evaluated by physical examination, including determining the Mallampati classification score (sensitivity, 43–49%), interincisor gap (sensitivity, 46%), thyromental distance (sensitivity, 20–25%), and sternothyroid distance (sensitivity, 62%), as well as performing the upper lip bite test (sensitivity, 7–19%) and flexion and extension of the neck (sensitivity, 10–16%).^4,5 However, these modalities have a low sensitivity, which can lead to inaccurate prediction of difficulties in intubation and ventilation.

Ultrasonography is used in patients with trauma or critical illnesses for vascular assessment, diagnosis, regional anesthesia, and airway management. Ultrasonography for airway management can be performed in the pre-intubation phase to evaluate the pathology and measure the trachea width to estimate the tube size,^6–8 intubation phase which detect endotracheal in situ faster than lung auscultation or end tidal CO₂ monitoring,^9,10 and extubation phase to measure diaphragm excusion and predict the extubation success.¹¹

However, the technique of using ultrasonography for airway assessment is new, and only a few experts use ultrasonography for airway assessment in Thailand. Therefore, skill development in novice personnel pertaining to the use of ultrasonography for airway assessment needs to be investigated. We aimed to assess improvements in the validity and reliability of novices’ skills in performing ultrasonography for airway assessment.

Materials and Methods

This learning cohort study was conducted after receiving approval from the Human Research Ethics Committee, Faculty of Medicine, Prince of Songkla University (REC 61-420-8-4). Twenty anesthesiology residents with no experience in using ultrasonography for airway assessment were included in this study. Four ultrasonography parameters of airway assessment were analyzed in 10 volunteers, random number generation from all nurse assistants, by the 20 anesthesiology residents from one institute. This analysis was first performed by one of two anesthesiologists with experience in airway ultrasonography. The exclusion criteria for the volunteers were presence of a neck mass, history of radiation therapy or surgery, wound in the head and neck region, and/or infection in the neck region.

After obtaining informed consent from the residents and volunteers, their demographic data were recorded. A Philips Affiniti 50 (Philips, Amsterdam, the Netherlands) ultrasound machine with a linear probe (L12-5) was used in this study (Figure 1).

Figure 1 Philips Affiniti 50 ultrasound machine with linear probe (L12-5).

The two anesthesiologists who had the experience of airway ultrasound for 3 years performed ultrasonography three times for airway assessment in each volunteer. These values were averaged to obtain the “gold standard” measurement values for each parameter and standardized by interpersonal validity.

Before the residents performed any measurements, the method of performing ultrasonography for airway assessment was explained, including the assessment of the following four parameters: 1) soft tissue thickness at the level of hyoid bone (STT-HY, distance between the skin and the upper border of hyoid bone; Figure 2); 2) soft tissue thickness at the level of true vocal cords (STT-VC, distance between the skin and the anterior commissure of true vocal cords; Figure 3); 3) soft tissue thickness at the level of thyroid isthmus (STT-TI, distance between the skin and the anterior surface of tracheal cartilage at the level of thyroid isthmus; Figure 4); and 4) soft tissue thickness at the level of the suprasternal notch (STT-SN, distance between the skin and the anterior surface of tracheal cartilage at the level of the suprasternal notch; Figure 5). The method of performing ultrasonography for airway assessment was explained via Supplementary Video before the time of first measurement by each resident. Measurements for each parameter were recorded thrice in millimeters, and the mean was calculated and recorded.

Figure 2 STT-HY, distance between skin and upper border of hyoid bone.

Figure 3 STT-VC, distance between skin and anterior commissure of true vocal cord.

Figure 4 STT-TI, distance between skin and anterior surface of tracheal cartilage at the level of thyroid isthmus.

Figure 5 STT-SN, distance between skin and anterior surface of tracheal cartilage at the level of suprasternal notch.

All 20 residents performed ultrasonography for airway assessment in all 10 volunteers, but in a different order. After the first measurements were performed by each resident, one of two experienced anesthesiologists provided feedback to the residents regarding any errors in the technique, though the discrepancy from the “gold standard” value was not disclosed. All subsequent measurements by the residents on the same and subsequent volunteers were performed on their own, without any feedback by any experienced anesthesiologist. Three measurements were performed for each parameter by each resident on the same day in a particular volunteer. The average values were used to calculate the discrepancy from the “gold standard” value for that parameter in that volunteer.

Statistical Analysis

Statistical analyses were performed using RStudio (version 2.13.0; R Foundation for Statistical Computing, Vienna, Austria) and Stata (version 14.2; StataCorp, College Station, Texas, USA). The magnitude of discrepancies between the measurements performed by residents and anesthesiologists was calculated over the sequence of measurements in the 10 volunteers. Change in discrepancies over the sequence of measurements was evaluated graphically and using mixed-effects random-intercept linear regression models, in which the residents were considered as random elements and the interaction between discrepancy and sequence was considered a fixed effect.

Sample Size

The required sample size was estimated to achieve a power of 80% in detecting a significant difference in the size of the standard deviation (SD) of discrepancies between the first and last measurements in the sequence of measurements at an alpha of 0.05. The sample size for the test of the difference between the two variances was used.

The study hypothesis was that the SD of performed measurements by residents at the tenth time in the sequence would be less than or equal to 0.7 times the SD at the first time in the sequence.

n=17

Consequently, a study population of 20 residents was estimated to provide an adequate sample size.

Results

The mean age of the residents was 27.4 ± 2.1 (mean ± SD) years. Of the 20 residents, 4 were male, and 8 were first-year, 6 were second-year, and 6 were third-year residents (Table 1).

Table 1 Demographic Characteristics of 20 Residents

The mean age of the volunteers was 30.9 ± 3.9 years, and the mean height, body weight, and body mass index were 160.3 ± 4.8 cm, 62 ± 15.1 kg, and 24 ± 4.8 kg/m², respectively (Table 2). The mean ultrasonic measurements of STT-HY by the experienced anesthesiologists and residents were significantly different (11.09 ± 3.14 mm vs 8.53 ± 3.02 mm, respectively; P = 0.008), whereas those of STT-VC, STT-TI, and STT-SN were not (7.18 ± 1.70 vs 7.14 ± 1.93 mm, P = 0.32; 7.81 ± 2.14 vs 7.73 ± 2.19 mm, P = 0.62; and 11.32 ± 3.33 vs 10.30 ± 3.02 mm, P = 0.35, respectively; Table 3).

Table 2 Demographic Characteristic of 10 Volunteers

Table 3 Parameters Performing by Experienced Anesthesiologists and Residents (mean±SD)

The mean discrepancy between measurements performed by residents and experienced anesthesiologists was close to zero throughout the sequence of measurements of STT-TI and STT-VC. However, the measurement values of STT-HY and STT-SN were considerably lower when performed by residents than when performed by experienced anesthesiologists throughout the sequence of measurements in the 10 volunteers. The range of discrepancies between measurements performed by residents and experienced anesthesiologists for each sequential measurement was wide for all measurements, particularly for the measurement of STT-HY, and the SD of discrepancies did not decrease over the sequence of measurements (Figure 6). Regression models including sequences revealed no significant decrease in the discrepancy in any parameter over the sequence of measurements.

Figure 6 Mean discrepancy and standard deviation (SD) value of each parameter of performing airway ultrasonography by twenty residents compared with two experienced anesthesiologists in ten volunteers over the sequences.

Discussion

Accurate airway assessment is essential to predict difficulties in intubation and ventilation. We evaluated the sensitivity of parameters used in this study and compared it with that of parameters such as Mallampati classification (sensitivity, 43–49%), interincisor gap (sensitivity, 46%), thyromental distance (sensitivity, 20–25%), sternomental distance (sensitivity, 62%), upper lip bite test (sensitivity, 7–19%), and flexion and extension of neck (sensitivity, 10–16%).^4,5 The sensitivity of these parameters is low. Therefore, we need tools with a higher sensitivity for promptly securing the airway when a difficult airway is predicted.

Ultrasonography has played a key role in many investigations and procedures. Airway ultrasonography is non-invasive, easy to perform, and useful in preparing for the management of difficult airways. Ultrasonography for airway management can be performed in the pre-intubation phase (airway assessment and prediction of tube size),^6–8 intubation phase (ultrasound-guided intubation and confirmation of tube placement),^9,10 and extubation phase (estimation of diaphragmatic function and examination of lung pathology).¹¹ In this study, we focused on performing ultrasonography for airway assessment in the pre-intubation phase.

We hypothesized that performing airway ultrasonography in at least 10 cases would improve the physician’s skill because the technique is not difficult. We designed a study to compare the SD of measurement values for each parameter determined by each resident with those determined by experienced anesthesiologists. Computed tomography (CT) is the gold standard for airway imaging.¹² Sustic et al observed a strong correlation between tracheal width and left main bronchus width with ultrasonography and not CT.¹³ However, Prasad et al¹⁴ found that the parameters of hyomental distance, depth of the epiglottis below the hyoid, thyrohyoid distance, and depth of the arytenoid cartilage did not differ significantly between sonographic and CT measurements.

Healthy volunteers exposed to radiation while undergoing CT of the head and neck region are at an increased risk of cataract and malignant tumor development in radiosensitive organs such as the salivary gland, thyroid gland, and retina.¹² It would have been unethical to expose volunteers to unnecessary radiation for this study. Thus, we designed this study to compare the learning curve of novices with that of experienced anesthesiologists to avoid the possible adverse effects of radiation exposure by CT.

Although CT is the first-choice imaging modality for the larynx and hypopharynx, the values of some parameters measured by CT are significantly different from those measured by sonography, such as the distance to the posterior surface of the tongue, thickness of the submental region, depth of the epiglottis from the skin, and fat pad thickness at the level of the thyroid cartilage.¹⁴

Over the sequence of measurements for airway assessment using ultrasonography in 10 volunteers by 20 residents in this learning trial, we did not find any evidence of increasing validity or reliability compared to that of experienced anesthesiologists. A possible explanation could be that the residents were not given adequate feedback after each measurement. No feedback was provided regarding the magnitude of the discrepancy in measurements. Experienced anesthesiologists provided some feedback about the technique for identifying the structure of the airway for the first measurement on the first day, but not for subsequent measurements. Consequently, the residents never understood whether their measured values were more or less than those of experienced anesthesiologists. Another explanation could be that 10 cases were not adequate to observe the improvement in validity or reliability.

STT-HY measurements can accurately indicate the possible difficulties in laryngoscopy (1.69 cm, 95% confidence interval [CI] = 1.19–2.19 in the difficult laryngoscopy group vs 1.37 cm, 95% CI = 1.27–1.46 in the easy laryngoscopy group).¹⁵ STT-HY measurements can also indicate the possible difficulties in mask ventilation (0.88 ± 0.3 cm for easy mask ventilation vs 1.4 ± 0.019 cm for difficult mask ventilation).¹⁶ STT-VC > 23 mm has been associated with difficulties in intubation (sensitivity, 85.7%; specificity, 57%; and area under the curve, 0.73).⁷ STT-TI ranged from 0.78 ± 0.35 cm to 1.08 ± 0.33 cm but could not accurately predict difficulties in mask ventilation and laryngoscopy.¹⁶ STT-SN has been used to verify the placement of the tube at the correct depth after endotracheal intubation.¹⁷ STT-VC, STT-TI, and STT-SN were found to be simple measurements for novices. The measurement values recorded by residents were similar to those recorded by experienced anesthesiologists.

The mean and SD of measurement values for each parameter determined by the residents over the sequence of measurements, as compared with those determined by experienced anesthesiologists, showed that the range of discrepancies was wide and did not change over the sequence. The residents’ measurements of STT-HY were significantly lower than those of experienced anesthesiologists. However, the values of STT-VC, STT-TI, and STT-SN were not significantly different over the sequence between the measurements recorded by residents and experienced anesthesiologists. This result can be explained by the compression due to the ultrasound probe. In volunteers with a greater thickness of soft tissue, the ultrasound probe could have led to greater compression. As the residents had no experience in performing ultrasonography for airway assessment, they may have used the ultrasound probe more aggressively than experienced anesthesiologists.

This study has some limitations. First, the residents were not given adequate feedback. If they had received qualitative and adequate feedback for each measurement, the discrepancy in the measurement values between residents and experienced anesthesiologists may have improved in each sequence. Second, the small number of volunteers did not permit any analysis of volunteer characteristics that may have affected the accuracy of the residents’ measurements. Third, the mean of measurements performed by experienced anesthesiologists using ultrasonography was considered the gold standard. Ultrasonography is an operator-dependent procedure; thus, values measured by anesthesiologists may have been inaccurate. However, both anesthesiologists were validated in the use of ultrasonography for airway assessment using the parameters used in this study. In future studies, values obtained by other imaging modalities, such as CT and magnetic resonance imaging, may serve as a more suitable gold standard.

Conclusion

Over the sequence of airway assessment in 10 volunteers by 20 residents in this learning trial, we did not find any improvement in measurement accuracy. Discrepancies between the residents’ and anesthesiologists’ measurements and the variability in discrepancy across residents were greatest in the measurements of STT-HY.