Exploring Endoscopic Competence in Gastroenterology Training: A Simulation-Based Comparative Analysis of GAGES, DOPS, and ACE Assessment Tools

Introduction

Gastrointestinal endoscopy has been abundantly used as a minimally invasive diagnostic and therapeutic procedure for a plethora of illnesses since its advent by Adolf Kussmaul in 1868.¹ Due to widespread use, growing demand, and increasing requirements of trainees, the role of assessment tools and simulation has become important in assessing competency of endoscopic skills.^2–5Previously, a set number of procedures were used as a benchmark to establish accreditation and granting privileges to conduct such procedures.^6,7 This number also varied significantly between different organizations and countries. Recently, the American Society for Gastrointestinal Endoscopy (ASGE) and the Joint Advisory Group for Gastrointestinal Endoscopy (JAG) have both made recommendations for establishment and maintenance of quality and standards of endoscopic training and intervention in the United States (US) and the United Kingdom (UK), respectively.^8–10

Various clinical assessment tools have been developed and validated to ensure competency in gastroenterology trainees. Two such prominent and validated tools include Direct Observation of Procedural Skills (DOPS) used by JAG for Gastroenterology in the UK, and the Global Assessment of Gastrointestinal Endoscopic Skills (GAGES) used by the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) in the US.^11,12 However, these tools focus primarily on motor skills associated with endoscopy with little emphasis on cognitive skills.

The ASGE developed another tool namely Assessment of Competency in Endoscopy (ACE) that assesses both motor and cognitive skills in a comprehensive manner.¹³ ACE has been conceived using the Mayo Colonoscopy Skills Assessment Tool (MCSAT) as a platform whilst bridging its limitations. Because it is a common tool that is used extensively to assess competence in GI in parts of the world,^14–16 we considered it an important one to be included in this study.

There is a dearth of literature comparing the performance of the aforementioned assessment tools, despite a wealth of well-documented research on their individual validation. Consequently, there is a lack of standardization of endoscopy competence assessment across training programs, leading to heterogeneity between endoscopists and training programs as to which assessment tool is superior.^14,17,18

Thus, it is imperative to implement a standardized and structured process for evaluating endoscopic skill competence to facilitate accurate trainee comparisons.^13,19,20 With multiple assessment tools available, it is essential to select the most reliable, discriminative, and user-friendly tool, for it to be used as a standard tool of assessment.

Simulation as an educational instrument allows trainees to perform in a controlled environment with different levels of difficulty while maintaining patient safety. Various studies have shown it to boost trainee confidence, improve patient outcomes and reduce procedure related discomfort.^3,21,22 It has been a widely and extensively used modality in endoscopy training.^23,24 The standard cases and the identical nature of difficulty lend simulation very well to assess competence in a cohort of trainees with similar baseline skills.²⁵

This study’s framework was based on the Accreditation Council for Graduate Medical Education (ACGME) Advisory Committee on Educational Outcome Assessment’s framework for assessing the quality of an assessment measure. The ACGME defines six criteria for evaluating standards, including reliability, validity, ease of use, required resources, ease of interpretation, and educational impact.²⁶ Since each of the three tools are independently validated and reliable, we chose to employ ease of use, required resources, and ease of interpretation in our study.

The objective of our study is to compare three different assessment tools to identify the tool that most accurately reflects competence in endoscopic skills.

Methodology

A mixed methods quantitative-qualitative approach was employed, specifically utilizing a deductive-sequential design where the primary component was quantitative, and the secondary component was qualitative.

Materials

Endoscopic Assessment Tools:

1. Direct observation of procedural skills (DOPS): Attached as Supplementary Materials 1 and 2 for EGD and colonoscopy, respectively. It uses a numerical scale as follows: Score 1 (requiring maximal supervision), Score 2 (requiring significant supervision), Score 3 (requiring minimal supervision), Score 4 (competent for unsupervised practice).
2. Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): Attached as Supplementary Material 3 and 4 for EGD and colonoscopy, respectively. It uses a similar scale ranging from 1 (unsatisfactory) to 5 (competent).
3. Assessment of Competency in Endoscopy (ACE) tool: It uses a 4-point grading scale defining the continuum from novice (1) to competent (4).¹³

Quantitative Methodology

Participants were trainees in the Gastroenterology training program at the Aga Khan University Hospital (AKUH), Karachi Pakistan. The AKUH is a 750-bed private tertiary care teaching hospital serving the needs of the city of Karachi, and the country at large. The Gastroenterology training program at the AKUH is a 3-year program. As part of their training, the trainees have defined competency goals at each level of training, for which they are required to achieve competence in particular skills by the end of that training year.

The gastroenterology training curriculum utilizes simulation to develop competence in trainees. As per this curriculum, they practice endoscopic skills on the Simbionix GI-BRONCH-mentor high fidelity simulator²⁷ until they achieve competence as per the simulator generated report for that procedure. They can then start supervised procedures on patients with faculty.

The sample included all 9 gastroenterology trainees in the GI training program during the year 2020–21, which constituted 100% of the population under study.

For the study, the trainee had to be deemed competent as per the simulator report in upper GI endoscopy (EGD) and colonoscopy. Once the required cohort was ready, they were taken through pre-simulation briefing as per the Standards of Best Practice for simulation briefing,²⁸ introducing all aspects of the simulation and explaining the learning objectives in detail as well as the tasks expected from them to perform. PGY1 trainees performed 2 upper GI endoscopies, while trainees in PGY2 and PGY3 performed 2 upper GI endoscopies and 2 colonoscopies. PGY1 trainees were not made to perform colonoscopies as they had not learnt how to perform them at the time of the study.

The cases were pre-decided by faculty, were not known to the trainee, and were diagnostic in nature. The cases for all the trainees were different, but of the same difficulty level as per the in-built modules of Simbionix GI-Bronch-Mentor high-fidelity simulator. These modules in software version SI-2825 comprise cases with similar findings and difficulty for learners to practice and were utilized for the study. For EGD, case 1 from module 1 was used, which was about a 30-year-old male with epigastric pain referred for endoscopy. For colonoscopy, case 1 from module 1 was used, which was about a 30-year-old female with irregular bowels referred for colonoscopy.

A simulated patient was used to evaluate the consent process, and a nurse assisted the participant in the conduct of the simulated procedure by informing the participant about the vital signs of the patient and assisting in taking a biopsy sample. Two faculty raters directly observed the trainee’s performing the procedures. Once a trainee performed a procedure, 2 faculty independently, and without consultation, completed all three of the assessment tools. This structure was followed for each procedure. Prior to starting, structured rater training was done. Rater training was held in a half-day workshop for all the raters. A didactic lecture on rater error training was conducted to familiarize trainers about common rating errors, followed by a Frame-Of-Reference (FOR) training²⁹ using videos where the group discussed discrepancies between individual raters with a facilitator.

Qualitative Methodology

Though psychometrics is important for providing validity and reliability evidence for assessment tools, the factors which often effective implementation of the assessment rely on its feasibility (cost, time and resource effectiveness) and acceptability (by faculty and students). It is for this reason that

educators need information regarding not only the trustworthiness of scores, but also the logistics and practical issues such as cost, acceptability, and feasibility that arise during test implementation and administration.³⁰

Therefore, a qualitative component was added to the study to provide evidence for ease of use, resources required, and ease of interpretation. For this, a focus group discussion (FGD) was considered suitable^31,32 and, hence, conducted for one hour to collect data about the utilization and implementation of the different assessment tools. Gastroenterology faculty (n=6) who assessed trainee performances, took part in this FGD. The purpose of the focused group was to gather information from the faculty about assessing trainee performance using the different tools. The prompt questions (attached as Supplementary Material 5) used for the FGD were designed to probe raters about their experiences using the multiple assessment tools. These questions explored rater experience with the tool and ease of interpretation which could help the authors qualitatively understand if the rater training that was conducted was helpful in minimizing biases. It also helps authors understand raters’ opinion about feasibility and acceptability, which are important qualitative characteristics of simulation assessment tools.³³ The FGD was recorded after all participants provided informed consent. The audio recordings were transcribed to a written document by an author who was not part of the simulation sessions or initial study design but was only provided the audio recording. The first author validated the transcribed document, while all authors generated the themes after reviewing the transcription.

Data Collection & Analysis

Participant scores were collected individually. Component and total scores of the DOPS, GAGES, and ACE skills were calculated individually for each procedure. An analysis was conducted where the mean and standard deviation were computed for each scale. The total score of each tool was stratified by level of training (ie, PGY1, PGY2, and PGY3) using one-way analysis of variance.

To assess internal consistency, Cronbach’s alpha was computed for each scale. Additionally, inter-rater reliability (IRR) was evaluated using the Intraclass Correlation. Alpha and IRR calculations were done with all trainees included in the analysis. All these analyses were performed using SPSS version 25.

Psychometric Analysis: Reliability analysis (internal consistency and inter-rater reliability) was calculated for each tool.

Reliability Analysis

1. Internal consistency describes the extent to which all the items in a test measure the same concept or construct and, hence, it is connected to the inter-relatedness of the items within the test.³⁴ Cronbach’s Alpha is the measure of internal consistency for ordinal and interval data.^35,36 The choice of cut-off values varies based on the type of examination: a threshold of 0.90 is recommended for high-stakes exams, while the range of 0.80–0.89 is considered acceptable for moderate tests, and 0.70–0.79 is deemed acceptable for lower-stakes tests.
2. Inter-rater reliability (IRR) refers to the level of agreement achieved by independent assessors who rate the same performance. Intraclass correlation is used for ordinal and interval data³⁷ and is interpreted as the proportion of variance in the ratings of the phenomenon being rated.³⁵ Therefore, intraclass correlation of the scores given by different raters to the same trainee can be analyzed to provide evidence for inter-rater reliability.³⁵

Qualitative Analysis

The FGD was recorded and transcribed for data collection. After member checking, content analysis was done to determine each tool’s ease of use, ease of interpretation and cognitive load.

This study was in compliance with the World Medical Association Declaration of Helsinki³⁸ and was approved by the Aga Khan University ethical review committee. We strictly monitored and maintained data confidentiality. Each trainee was given a unique identity and trainee identifiers were kept anonymous.; only the research team had access to this material. The data was stored as per good clinical practice guidelines. Written informed consent was taken from all participants prior to study commencement, and the informed consent included publication of anonymized responses.

Results

Demographic Data

The faculty raters were 6 in number. The trainees were 9 in number (2 PGY1, 3 PGY2 and 4 PGY3 trainees), and all of them met the inclusion criteria.

Participant scores for each tool

Table 1 shows a summary of average scores for each tool by level of training.

Table 1 Summary of Scores for Each Tool According to the Level of Training

Quantitative analysis for upper GI endoscopy – esophagogastroduodenoscopy (EGD)

Table 2 shows the quantitative analysis of each tool for upper GI endoscopy, with Cronbach’s Alpha for internal consistency of the tools and Inter-Rater Reliability.

Table 2 Quantitative Analysis for Upper GI Endoscopy – Esophagogastroduodenoscopy (EGD)

Quantitative analysis for colonoscopy

Table 3 shows the quantitative analysis of each tool for colonoscopy, with Cronbach’s Alpha for internal consistency of the tools and Inter-Rater Reliability.

Table 3 Quantitative Analysis for Colonoscopy

Qualitative Analysis

A series of open-ended questions were asked regarding the ease of use of the tools, the strengths and areas of improvement of each tool, the comprehensiveness of each tool, and the resources that were required to complete the tool. Analysis of the transcript resulted in three major themes emerging regarding the tools, namely that the best tool needs to be: scientifically sound, comprehensive and user-friendly. The next section will describe these themes in further detail and provide quotations that most accurately depict the themes and dimensions emerging from the FGD.

Scientifically Sound Tool

We asked the trainers which tool in their opinion was the most complete, in terms of the robustness of assessment and utility of use. All universally said that the most important was that the tool be valid and reliable for the assessment, and that no compromise could be made on its scientific credentials.

Tools need to be valid and reliable (All trainers)

No point in having a tool that is not scientifically reliable-the assessment will be completely pointless (Trainer 2)

On asking about further qualities, variation was noted between the trainers, where some felt that the increased detail in the ACE is essential, while others felt that the same detail made it overwhelming.

ACE- Very detailed, breaks down the skill into many parts (trainer 4). Assesses knowledge-that’s good, but we are assessing endoscopic skill here (trainer 2) Very detailed and complete-but perhaps difficult to do on a formative basis regularly (trainer 5)

Trainers also felt that the best tool needs to assess not only the endoscopic procedure itself but sedation, safety, consent and communication with the patient and para-medical staff before and during the procedure and post-procedure tasks such as report generation, and capturing of images as well.

Tools need to assess not only the psychomotor skills, but the procedure in totality-the pre and post procedure experience as well (trainer 1)

DOPS- Covers most aspects of a good assessment (trainer 2) It’s a good complete tool-does what it’s supposed to do (trainer 4)

GAGES- Isolates the procedure from the other aspects (trainer 3) Gives you a broad assessment (trainer 5) Too procedure focused only (trainer 6)

User-Friendly Tool

Trainers said that the most important thing for them to assess regarding the utility of the tool was its ease of use. This was one of the most important factors that word dictate whether they would continue to use it in the future. They defined ease of use as the form not taking long to fill, not very wordy, with an open layout and easy readability with regard to font size and type.

Tools need to be easy to use, quick to fill, assess all major skills (trainer 2)

The GAGES was super-quick. DOPS was next-the ACE was good, but just too cumbersome (trainer 5)

You have to see which tools can be used repeatedly, so that they don’t become overwhelming and time-consuming (trainer 1)

You have to look for a happy medium-not too long, but not too short, and easy to understand (Trainer 3 and 4)

Comprehensive Tool

The trainers were asked their opinion regarding the resources that were required to complete the tools. All of them felt that there was no difference in material resources. However, the time that was required to fill the tools was more for the ACE as compared to the GAGES and DOPS.

I didn’t require anything different in completing one tool or another (trainer 4)

Is time a resource? -it is, right? Then the ACE is definitely more resource intensive than the other 2 (trainer 6)

Pros and Cons of Each Tool

The trainers were asked to discuss the advantages and disadvantages of every tool in detail. This was asked in turn from all six trainers. General comments about any tool were not adverse. All the trainers thought that all three tools were generally very good, assessed the skills in detail, and were good to use on their own.

However, they all had their differences which made the overall experience of using them different for the trainers.

GAGES- Quick and to the point, but too brief (trainer 1)

DOPS- Reasonably quick, but takes a bit of initial understanding (trainer 3) Nice, open layout (trainer 5) Measures what it’s supposed to (trainer 6)

ACE- Detailed assessment, but time consuming (trainer 4) lengthy (trainer 6) Overwhelming (trainer 3) way too much detail-difficult to do regularly as formative assessment (trainer 5)

Discussion

The traditional approach to teaching endoscopic procedures involves the apprenticeship model, which relies on procedure numbers and training time. However, this approach is risky and unpredictable, potentially endangering patient safety. Furthermore, this training method lacks standardization and is subject to variation based on the trainer, trainee, and patient. To ensure a consistent and objective evaluation of competence, training must be standardized and structured, with frequent formative assessments.³⁹ In this study, we aimed to identify, through a scientifically rigorous and structured approach, the most effective assessment tool for this purpose.

Our results were in line with the values obtained in the initial validity studies of these tools. For ACE, the reliability coefficient was calculated for its individual components and reported as 0.79 for cognitive part and 0.88 for the motor part.⁴⁰ GAGES had Cronbach’s alpha value of 0.89 for upper GI endoscopy and 0.95 for colonoscopy.¹² Its IRR was 0.96 for upper GI endoscopy and 0.97 for colonoscopy.¹² DOPS had a Cronbach’s alpha value of 0.95⁴¹ and reliability of 0.81.¹¹

A key concept in the interpretation of internal consistency is that a low alpha value might result from a limited number of questions, weak correlations among items, or diverse underlying constructs.³⁴ Conversely, an excessively elevated alpha could indicate item redundancy, where certain items essentially examine the same concept but are phrased differently.³⁴ Our study determined that in EGD skill assessment, the highest internal consistency was for the GAGES, followed by the DOPS tool. As per the key interpretation concept mentioned above, the GAGES tool would be expected to score higher, given the fact that it is a 5-point scale and therefore a wider range, resulting in a greater standard deviation. In contrast, DOPS and ACE each are 4-point scales, hence having ranges narrower than GAGES, resulting in lower standard deviation and scores. Additionally, ACE has diverse underlying constructs, resulting in lower alpha values. The tools performed in a similar fashion for colonoscopy assessment as well. Trends remained the same as EGD for both analyses in colonoscopy in all three tools.

As discussed earlier, all 3 tools are very robust and can be used safely in the assessment of GI endoscopic skills. The GAGES tool is easy to administer, is consistent and meets high standards of reliability and validity.¹² The DOPS tool is a very good in-training competency assessment tool,⁴² while the ACE tool is also very sound and can meaningfully monitor fellows’ progress throughout the entire training continuum.⁴³

GAGES is designed to measure technical skill,¹² which it does very well. However, in our study, raters felt that a good endoscopic skill assessment tool should include the pre- and post-technical skill attributes as well, such as communication with the patient and staff, consent, and recognition of findings and complications. Given this, and the relative subjectivity in the scoring, the GAGES tool is our study was thought to be a good tool only if procedural assessment was required but not if the goal was training an endoscopy learner in the overall aspects that make up the procedure.

The ACE tool was extremely detailed and reduced subjectivity the most, thus having the highest IRR. However, in the FGD, raters felt that the tool was very extensive and detailed, and while all felt that may be a good thing, they felt it did not lend itself to repeated use. A few raters felt that there were so many entries to make, which took away from the time they would like to give to the trainee in providing feedback. Some felt that the ACE may serve well for advanced trainees as summative assessment, but for formative assessment they would prefer a briefer tool.

The DOPS tool scored well in the reliability analysis. It had favorable reviews in the FGD as well. Raters felt it did not ignore the pre- and post-procedure steps of an endoscopic procedure, was not time-consuming, and could be easily performed as formative and summative assessment. The scale was easy to understand and seemed to have the best mix of the affective, knowledge and technical attributes required for effective skills assessment of a gastroenterology trainee. These findings are also supported by Siau et al in their study about nationwide evidence on validity and reliability evidence to assess DOPS as an assessment tool⁴² and in the study on pediatric EGD.⁴⁴

Our study had main strengths, discussed as follows. To minimize bias and reduce sources of error in our study, we ensured that each trainee’s performance evaluation was rated by at least two individuals. The assessment tools we used for scoring the trainees’ performance were all validated tools identified from literature.⁴⁵

We utilized a standardized patient to evaluate the consent taking process, and a nurse for procedural assistance. We did this so that all facets of the tools could be evaluated, to avoid stereotyping and maintain fidelity, in keeping with best practices in simulation.

Prior to conducting the performance assessment, raters were trained through a discussion of the tools. This training, combined with independent assessments by the two raters, helped to establish the inter-rater reliability demonstrated in our study results.

The primary limitation observed in the study is the small sample size, despite the inclusion of all trainees. This was compensated for by the qualitative aspect of the study which had data from the FGDs with all the raters which, in turn, may have brought a subjective bias into the study. Additionally, the study was conducted exclusively within a single institution.

The results of our study have demonstrated the DOPS tool to be the most consistent assessment tool out of the three, in terms of internal consistency and inter-rater reliability as well as the themes identified from FGDs. However, our results also present an array of options for choosing the best tool based on the nature of the assessment being done. GAGES would be best applicable to situations in which only the procedural skill is being assessed. DOPS would be the best for overall procedural competence in a comprehensive yet compact manner, including both the knowledge and skill of the trainee, whereas ACE is most likely to have utility in summative examinations in which detailed assessment is being done for all aspects of procedural competence in depth.

These results have important implications for teaching and training in GI training programs. It will benefit trainees by identifying the best tool to provide them with vital feedback for learning. Selection of an appropriate assessment tool will help faculty by making scoring easier, more objective and efficient. This will improve training programs by ensuring standardization of assessment through valid and reliable tools to ensure high-quality training. Additional studies could be conducted to investigate the effect of training using different assessment tools, by examining their correlation with the clinical performance and quality outcome indicators of trainees in real-world clinical settings.

Conclusion

Our study demonstrated that all three assessment tools were found to have their advantages and drawbacks, and it would be inappropriate to pick one as the best for all contexts. In our study, the DOPS tool performed favorably in the qualitative assessment, and performed adequately in the psychometric evaluation to warrant being considered the most balanced of the tools. Therefore, we propose that the DOPS tool be used for endoscopic skill assessment in gastroenterology training programs whenever the setting resembles conditions portrayed in our study. However, we propose that gastroenterology training programs match their learning outcomes with the available assessment tools to decide which may be the most appropriate one for their need and context.