Evaluation of the learning environment of urology residency training using the postgraduate hospital educational environment measure inventory

Introduction

Urology residency training in Saudi Arabia is a 5-year structured program; it is hospital-based with central supervision from Saudi Commission for Health Specialties (SCHS) in which the 1st year is for General Surgery and Surgical Intensive Care rotation. Residents who complete their residency ultimately receive certification as a specialist.

The educational environment is an important measure which has a large impact on the satisfaction and success of medical education.¹ The establishment of a supportive learning-oriented culture is of utmost significance in creating competent physicians.² A measurement instrument of good quality to assess that learning environment is a prerequisite to allow accurate assessment of the learning environment and to identify areas requiring attention.³

The postgraduate hospital educational environment measure (PHEEM) is a self-administered 40-item inventory that assesses metrics of the level of autonomy, quality of teaching, and social support during the hospital-based training period for postgraduate students.⁴ PHEEM is a reliable and validated instrument to evaluate the quality assurance process as well as strengths and weaknesses within a certain educational environment.^4,5

The objective of the present study is to evaluate the learning environment of urology residents and factors influencing their perception of this environment. The study also evaluates associations of educational-environment perception with stages of the residency program (R2, R3, R4, R5) and training in different regions of Saudi Arabia and different health care sectors. To our knowledge, this is the first study to evaluate the urology residency program since its implementation in Saudi Arabia.

Methods

This is a cross-sectional study using the English version of PHEEM instrument to assess the educational environment of the urology residency program. Respondents were asked to indicate their agreement with each statement using a five-point Likert-type scale, which ranged from strongly agree (4), to agree (3), unsure (2), disagree (1), and strongly disagree (0). The four negative statements (questions 7, 8, 11, and 13) were scored in reverse so that the higher the score, the more positive the environment. Information on sex, residency level, training region in Saudi Arabia, and main training hospital were also included as part of the questionnaire.

The perceptions of teaching subscale contains 15 items with a maximum score of 60; perceptions of autonomy subscale contains 14 items with maximum score of 56; and perceptions of social support subscale includes eleven items with a maximum score of 44. The summation of these scores has a combined maximum of 160 and minimum of zero. Higher levels of agreement were correlated with more beneficial educational environments. Global scores of 0–40 indicate a very poor educational environment, 41–80 indicate plenty of problems, 81–120 indicate more positive than negative but room for improvement, and 121–160 indicate an excellent educational environment.⁴

In 2011, after Institution Board approval, questionnaires were emailed to all residents registered in the urology training program by an independent third party, and the identity of the collected data was kept anonymous to the researchers. Global mean scores, for individual respondents, were calculated with missing values scored as 2 (the midpoint on this 0–4 scale). Scores for each item and domain were calculated and entered into a spreadsheet. Raw scores were coded and calculated.

Descriptive statistics were reported as median, mean, and standard deviation. The comparative statistics used the nonparametric method of Kruskal–Wallis⁶ to compare PHEEM-derived data from respondents in different levels of residency training (excluding R1 residents) as well as to compare residents from different regions of Saudi Arabia and from different main sectors of the health care system.

The reliability analysis was performed using the Cronbach’s alpha coefficient to measure the internal consistency of the whole questionnaire and each of the three subscales.⁷ Using the “alpha if item deleted”, Cronbach’s alpha was used to identify questions whose exclusion would improve the reliability.

Results

Out of 72 registered residents, 38 (53%) completed the questionnaire. Apart from one female, all were males with a mean age of 29.1±2.3 years. The number of residents in each residency year varied from four to 12. Out of a possible 1,520 responses to the 40 questions, only 26 (1.7%) were missing, suggesting that the questionnaire was simple and practical. Responses of residents in the 1st residency year (4) were excluded being involved in general surgery rotation year. Table 1 illustrates demographic details of the included respondents, their region of training in Saudi Arabia, and their main health care sector.

Table 1 Characteristics of study respondents

Median and mean scores (± standard deviation) for each item of the inventory for the three domains and for the overall inventory are summarized in Table 2. There were no statements which were highly rated (mean value >3), and 22 statements were poorly rated (mean value 2 or less).

Table 2 Mean and median of each question, overall and subscales scores Abbreviations: Q, question; SD, standard deviation.

There was no statistically significant difference (P>0.05) in perception scores of educational environment among residents in different training regions of Saudi Arabia or different stages of residency program.

Significant differences were identified between residents from different main sectors of the health care system regarding the total score (P=0.024) as well as the teaching subscale (P=0.017). Residents from National Guard health care sector responded more significantly positive to questions 2, 3, 12, and 21 of the teaching subscale; responded more significantly positive to questions 4 and 32 of the autonomy subscale; and responded more significantly positive to question 24 from the social support subscale (Table 3). They also responded significantly positively to questions 10 and 33 as did residents from Armed Forces and Security Forces health care sectors, respectively. Security Forces health care sector residents responded more significantly positively to question 29 of the autonomy subscale (Table 3).

Table 3 Comparison of PHEEM scores of respondents from different health care sectors Abbreviations: PHEEM, postgraduate hospital educational environment measure; Q, question.

Cronbach’s alpha coefficient was computed to measure the internal consistency of the overall questionnaire and each of the three subscales. For the whole questionnaire, Cronbach’s alpha was 0.892. For the teaching and training subscale, Cronbach’s alpha was 0.847; for autonomy it was 0.750; and for the social support subscale it was 0.478. Questions 9, 17, and 32 of the autonomy domain, 16, 19, 20, 25, 26, and 36 of the social support domain, and 21, 27, and 37 of the teaching domain were not consistent with the rest of the scale and could be deleted (Table 4). On deletion of these factors from the reliability analysis, the Cronbach’s alpha coefficient increased for the entire inventory and for the social support domain to 0.911 and 0.564, respectively. Teaching and autonomy domains showed a little elevation to 0.867 and 0.757, respectively.

Table 4 Reliability analysis of the overall questionnaire Note: *Questions 9, 16, 17, 19, 20, 21, 25, 26, 27, 32, 36, and 37 are not consistent with the rest of the scale. Abbreviation: Q, question.

Discussion

The educational environment plays a crucial role in the learning process. PHEEM can be used to identify strengths and weaknesses of a medical residency program.⁵ It has been used to assess the educational environment of the residents in different studies.^5,8–13 We used PHEEM as a tool to evaluate the educational environment of the postgraduate urology residency program in Saudi Arabia for the first time since its implementation in 2000.

In this study, residents in the final stages of the residency program (R4 and R5) represented more than one-half of the respondents. Although it is expected that residents in final residency stages have more ability to judge the program, the stage of the program did not seem to have an effect on how residents perceive their educational environment (P>0.05). Similar results were also seen in another study.¹¹

Likewise, although residents from the central region of Saudi Arabia represented the majority of the respondents (47%), training region in Saudi Arabia had no effect on residents’ perception of their educational environment. A study of rotating interns in Australia showed that the rural performed better than urban locations on the teaching, autonomy, and support subscales.¹⁰ However, only one urban hospital was approached to be the comparator site and it is possible that adding more tertiary urban hospitals could change the results. We believe that no great cultural differences exist between different main regions in Saudi Arabia, especially given that all current urology training program hospitals are urban hospitals.

Residents from different main sectors of the health care system differed significantly regarding the total score (P=0.024) and the teaching subscale (P=0.017). National Guard health care sector residents showed a significantly higher satisfaction with their teachers, safety of their environment, and access to and available time for education. Similarly, residents of Security Forces health care sector significantly perceived more involvement in team work (Table 3). This mostly represents local differences among different sectors of the health care system as the urology curriculum is universally applied all over Saudi Arabia.

Our study showed no overall real positively rated points (mean score of 3.5 or more), while 22 statements were poorly rated, with a mean value of 2 or less, and should be examined more closely as they indicate problem areas.⁴ The lowest recorded score was 1 for item 26 (There are adequate catering facilities when I am on call), a situation that can be easily solved by proper training-site management. Poorly rated questions included questions 1, 4, 5, 11, 14, 17, 32, and 34 from the autonomy domain, which assess the presence of clear clinical protocols in the post, information provided by the contract regarding working hours, informative program, and work load in the job. Furthermore, the residents had doubts that the training in the post would make them ready to be specialists/consultants (question 34, mean score 1.7±0.9). Other poorly rated questions included questions 2, 3, 9, 15, 21, 22, 23, 27, 28, 33, and 39 from the teaching domain, which mainly define the role of teachers, ie, their teaching skills, enthusiasm, organization, and their ability to set clear expectations, as well as their ability to provide their residents with feedback on their strengths and weaknesses. Poorly rated social domain questions included questions 19, 20, 25, 26, and 38 which involve mainly hospital accommodation and catering facilities as well as no-blame culture. Similar results were shown in 101 residents in the Saudi family medicine training program.¹³

The total inventory score in this study was 77.7 (±16.5) which, according to the criteria proposed by Roff et al,⁴ reflects an educational environment with plenty of problems. Perception of the role of autonomy (26.2±6.5) showed a negative view of one’s role. Perception of teaching by the residents (29.7±7.7) indicated that teachers are in need of some retraining. The social support perception (21.9±4.3) reflected an unpleasant environment. Such results should be taken into account by curriculum planners as they consider improvements to the urology training program. Level of supervision provided by superiors, flexibility and freedom in the job, and level of participation in decision making are important areas of development and enhancement. Inclusion of residents’ perceptions of their training experience as a part of the quality assurance for accreditation might be beneficial. Course enrollments and free access to electronic journals are fundamental educational sources. Supervisors should be supportive and provide effective supervision and mentoring for residents.

Cronbach’s alpha scored high at 0.892 for the 40 statements and 0.847 for the teaching domain. It scored good at 0.750 for the autonomy domain and poor at 0.478 for the social support domain. The overall inventory showed a high internal consistency and reliability. A high reliability of PHEEM questionnaire was also shown in different studies with feasible sample sizes.^4,5,9–14 However, when our data was analyzed to exclude each question in turn, using the “alpha if item deleted”, 12 questions were found to be inconsistent with the rest of the scale and could be deleted (Table 4). Furthermore, the Cronbach’s alpha coefficient for the entire inventory and its three subscales increased on deletion of these items, suggesting reduction of the questionnaire items provided that the same results could be reproduced in a larger sample size. Items 1, 7, 10, 13, 15, 25, and 28 were previously found to be uncorrelated with total score in other reports.¹⁴

Our study is limited by the small sample size and rarity of females in the Saudi urology residency program, which precluded comparison of sex perception. However, we believe that this study, being the first of its nature, represents a good chance to evaluate the current urology training programs and could help in improving them.

Conclusion

PHEEM survey is an applicable and valid instrument for assessing the educational environment of the urology residency program in Saudi Arabia. Urology residents perceived the urology residency program educational environment as less than satisfactory. Perceptions of the educational environment did not change significantly among different stages of the program or different training regions of Saudi Arabia. Some sectors of the health care system are doing better than others. The residents’ educational environment needs close attention in all of its domains. Clear clinical protocols, work load, working hours, quality of teaching supervision, and supportive hospital environment are areas which need development and enhancement. It is hoped that the information provided by this study could be used to improve the educational environment of the urology residency program in Saudi Arabia.

Acknowledgment

This study was supported by a grant from the College of Medicine Research Center, Deanship of Scientific Research, King Saud University, Riyadh, Saudi Arabia.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	Genn JM. AMEE Medical Education Guide No 23 (Part 1): Curriculum, environment, climate, quality and change in medical education-a unifying perspective. Med Teach. 2001;23(4):337–344.
2.	Hoff TJ, Pohl H, Bartfield J. Creating a learning environment to produce competent residents: the roles of culture and context. Acad Med. 2004;79(6):532–539.
3.	Cassar K. Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Med Teach. 2004;26:260–264.
4.	Roff S, McAleer S, Skinner A. Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK. Med Teach. 2005;27(4):326–331.
5.	Vieira JE. The postgraduate hospital educational environment measure (PHEEM) questionnaire identifies quality of instruction as a key factor predicting academic achievement. Clinics (Sao Paulo). 2008;63:741–746.
6.	Tomkins CC. An introduction to non-parametric statistics for health scientists. University of Alberta Health Sciences Journal. 2006;3:20–26.
7.	Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011;2:53–55.
8.	Wall D, Clapham M, Riquelme A, et al. Is PHEEM a multi-dimensional instrument? An international perspective. Med Teach. 2009;31(11):521–527.
9.	Gooneratne IK, Munasinghe SR, Siriwardena C, Olupeliyawa AM, Karunathilake I. Assessment of psychometric properties of a modified PHEEM questionnaire. Ann Acad Med Singapore. 2008;37(12):993–997.
10.	Auret KA, Skinner L, Sinclair C, Evans SF. Formal assessment of the educational environment experienced by interns placed in rural hospitals in Western Australia. Rural Remote Health. 2013;13(4):2549.
11.	Al-Marshad S, Alotaibi G. Evaluation of Clinical Educational Environment at King Fahad Hospital of Dammam University Using the Postgraduate Hospital Education Environment Measure (PHEEM) Inventory. Education in Medicine Journal. 2011;3:6–14.
12.	Al-Shiekh MH, Ismail MH, Al-Khater SA. Validation of the postgraduate hospital educational environment measure at a Saudi university medical school. Saudi Med J. 2014;35(7):734–738.
13.	Khoja AT. Evaluation of the educational environment of the Saudi family medicine residency training program. J Family Community Med. 2015;22(1):49–56.
14.	Aspegren K, Bastholt L, Bested KM, et al. Validation of the PHEEM instrument in a Danish hospital setting. Med Teach. 2007;29:498–500.