Evaluating the Effects of Training to Improve Teaching Skills of Health Sciences Educators in Sudan

Background

The World Health Organization has identified Sudan as one of countries suffering an acute health care workforce crisis.^1,2 The crisis has negatively impacted the country’s ability to provide essential, life-saving interventions such as safe pregnancy, service delivery for mothers and child immunizations.^3,4 Maternal and child health indices in Sudan have remained very poor due in part to the lack of properly trained, competent health professionals who can address maternal and child health needs.^5,6

The Sudan Public Health Training Initiative (PHTI), a project led by The Carter Center (TCC) and the Sudan Federal Ministry of Health (FMoH),) aims to strengthen the capacity of Sudanese training institutions to produce more well-trained frontline health workers to meet the health maternal and child health needs of rural and urban communities (Figure 1: PHTI objectives). The initiative has launched a capacity building programme for health sciences training institutions, namely, the Academy of Health Sciences (AHS), Midwifery Schools (MS) and Centres for Continuing Professional Development (CPD).

Figure 1 Objectives of public health initiative.

Training Course on Learning and Teaching

Ongoing changes in management and clinical practices, health system organization and financing mean that future health care providers need to be sufficiently trained to meet the needs of the population in the country.^7,8 Trainers of the health sciences are from different fields of health care with different backgrounds and affiliations that include but not limited to the FMoH, medical schools, public health institutions, nursing and midwifery schools, They occupy important positions in health sciences training institutions and in the health system more broadly. However, the overall capacity is inadequate to carry out effective teaching functions, such as designing appropriate training courses, assessing student learning, providing student support and evaluation of training.^9,10 Continuing development of teaching staff would therefore play a vital role to enhance the capacity and skills of the teachers as well as to contribute to the motivation and retention of teaching staff at the health sciences training institutions.¹¹

In recognition of the need to enhance professional development of teaching staff working in health sciences institutions, PHTI has launched short courses in Sudan on “how to teach” delivered for the target teaching staff (referred to as trainees) from the eight target states, each course lasts for a period of 2 weeks. Accordingly, fourteen short courses on “learning and teaching” were designed and delivered to 310 teaching staff for all the eight PHTI target states as shown in Figure 2 (PHTI Target states).

Figure 2 PHTI target states.

These short courses were meant to address key human resources issues, particularly in enhancing the pedagogical skills of trainers at health science training institutions and to develop, adapt, and standardize health learning curricula and materials.

The learning outcomes and content of the training courses are shown in Table 1.

Table 1 Learning Outcomes and Content of the Short Courses

Training Evaluation

Learning and teaching short courses are intended to add valuable benefit to the trainers, students and the health system as presented in Figure 3 (Value of training in learning and teaching).

Figure 3 Value of training in learning and teaching.

This evaluation was the first step to ensure the training courses are worthwhile and are a valuable investment for both trainers and students to positively affect the health system and health status of the population.

There are a number of evaluation techniques and models that are used to measure the benefit of training, which include experimental/quasi experimental; logic; the context/input/process/product (CIPP); and Kirkpatrick four-level.¹² The latter, Kirkpatrick four-level model gives clear focus on the programme outcomes and clearly describes outcomes beyond simple learner satisfaction. This model has been used to evaluate similar training programmes in a number of countries.^11–13

In order to draw evidence regarding the outcome of the training courses in learning and teaching that were conducted in Sudan, this evaluation focused on levels three and four of the Kirkpatrick’s four-level model, ie, impact and results, while levels one and two (participant reaction and learning) were evaluated in a separate study.¹⁹

Aims and Objectives of the Evaluation

This evaluation aimed to determine the effectiveness of the one-off, singular learning and teaching courses and to assess their effects on the performance of trainees and learning environment by:

a) determining the trainees’ application of knowledge and practices acquired from learning and teaching training, and

b) documenting the views of managers of training institutions and beneficiaries (students) on the performance of trainers following the training courses.

c) Determining any changes that have been made at the organizational level as a result of the learning and teaching training.

Methodology

The study was a cross-sectional, institution-based qualitative investigation¹⁴ conducted in Academy of Health Sciences (AHS), Midwifery Schools (MS) and Continuous Professional Development (CPD) Centers in the eight states of Sudan that are supported by TCC through PHTI. It took place from October 2017 to January 2018. The study subjects included managers of the program and training institutions, students from training institutions and training staff who participated in previous PHTI learning and teaching short courses, using purposive sampling.¹⁵

First, in order to assess the actual performance of trainers, the evaluation included direct observations of selected teaching sessions that were delivered by trainers who attended learning and teaching courses using specific observation protocol used for this purpose. The observations were carried out by researchers who received appropriate training on how to carry out the observation and how to fill the reflection\comment part properly. The observations were conducted in four out of the eight PHTI target States, which included North Kordofan, Gezira, Khartoum and River Nile. A total of 8 teachers from AHS and CPD were randomly selected for observations. The evaluators (observers) focused their observations on the following practices:

• Written specific, meaningful, appropriate, realistic and testable (SMART) learning objectives.
• Planning the sessions and delivering high quality presentations and lectures.
• Selecting and using appropriate teaching methods to meet the learning objectives.
• Supporting learning in a range of appropriate ways.
• Planning and applying students’ assessments for constructive feedback where appropriate.

Secondly, students enrolled in the PHTI targeted States were recruited in focus group discussions to solicit their perceptions on the quality of teaching, presentations, assessments and student support. Eight focus group discussions were conducted, in four out of the eight PHTI target States, ie North Kordofan, Gezira, Khartoum and River Nile, with two FGDs in each state. A structured set of questions in Arabic were used to guide the focus group discussions at the different sites. Discussions were recorded and notes taking during the FGD. Recorded discussions were transcribed in Arabic and then translated into English.

Thirdly, eight key informant interviews (KII) using structured questionnaire were conducted involving program managers of training institutions (AHS and CPD) from four states, namely, North Kordofan, River Nile, Gezira and Khartoum. The aim was to explore their views regarding the effectiveness of training courses and overall performance of colleagues who attended short courses on learning and teaching, as well as their opinions on any organization level changes they had observed that might be attributed in part to the learning and teaching courses. Likewise, interviews were recorded and notes taking during the key informant interviews. These were later transcribed in Arabic and then translated into English. Table 2 summarizes different tools used, including areas of investigation for the evaluation.

Table 2 Sources and Type of Data and Evaluation Tools

Prior to the start of the evaluation, individuals assigned for data collection (FGD and KII) were provided a unified training on the objectives the study, methods and how to conduct the interviews using questionnaire guide, which were pre-tested by the data individuals in similar institutions before the actual data collection.

A thematic analysis was applied to the datasets, Figure 4 shows the process entailed. By analyzing similar and contrasting accounts from the coding frame, codes consistent with the evaluation objectives were developed into themes.^16–18

Figure 4 Thematic data analysis.

Ethical clearance was obtained from AHS and CPD at the Federal level and informed consent was obtained from each study participant. The consent included the approval for publication of anonymized responses. Participants names were not recorded as part of the questionnaire or interview. Participation in the study was voluntary and participants were informed that they were free to withdraw from the study at any time in the data collection process.

Limitations

Generalization

The small sample size and methodology prevented generalization of the findings. Furthermore, the diversity in different States in terms of resources and capacity across Sudan prevented exploration of these broad perspectives. Nonetheless, generalization was not intended rather, emphasis was placed on determining in-depth perspectives from participants.

Bias

The facilitators of the learning and teaching course along with program implementers led the evaluation; that the evaluation was not conducted by an independent entity and therefore there could be some inherent biases.

Results

The results of the qualitative evaluation focused on levels three and four of Kirkpatrick framework which was used to measure application of learning in their job (ie behavior changes related to job performance), and to a lesser extent, organizational impact.

Behavior Changes

This level of Kirkpatrick model measures the behavior changes as the result of learning. It measures the application of the newly gained knowledge and skills from the course and evaluates the actual job performance. In this evaluation, behavior change was evaluated using qualitative methods to monitor the job performance after training, targeting the trained teaching staff and their students. This paper focuses on qualitative aspects, as findings of quantitative evaluation are published elsewhere.¹⁹

The findings in regard to behavior changes are presented under the following two themes:

• Trainer’s performance through observations.
• Student’s perceptions of their educational experience.

Theme One: Trainers Performance Through Observation

Observations were conducted for the trained instructors teaching sessions to collect data on the impact of the training on their teaching process. The observation focused on topics covered during the training courses, ie the use of learning objectives to guide the teaching session, teaching methods including presentation skills and approaches used by the trainer to give feedback to students. A summary of the findings and observers’ comments are provided in Table 3

Table 3 Observation of Teaching Sessions

Theme Two: Students’ Perceptions

This part of evaluation involved focus group discussions to obtain data from students regarding their perceptions on certain aspects of the teaching process that linked to topics covered during the course. The selected students were taught by trainers who attended the learning and teaching courses. The findings are presented under six themes, as illustrated in Figure 5.

Figure 5 Themes of student perceptions.

Statement of Session Learning Objectives

Learning objectives are statements that define the expected goal of a curriculum, course, lesson or activity in terms of demonstrable skills or knowledge that will be acquired by a student as a result of instruction. They help to put learners at the centre and equip them with the knowledge of what is expected from them as the result of learning.²⁰ During FDGs, students mentioned that some of their instructors clearly presented the objectives of the session when they started teaching. They either presented on a PowerPoint slide, wrote on a flip chart/blackboard or mentioned them verbally. One of the students stated:

My teacher mentioned the lecture objectives at the beginning of the session and clearly stated what we as students are expected to learn, followed by what she wants to present.

They also referred to other instructors who often started their teaching without stating what they expected students to learn from the session. This was happening particularly in practical or clinical sessions rather than in theoretical ones, as illustrated by one of the students and agreed by other members during FGD:

In the practical session, most of our teachers start teaching new skills, for example ‘prepare a woman for labour’ without telling the students what the objectives of the session are.

Training Curriculum Design

Curriculum design is a term used to describe the purposeful, deliberate, and systematic organization of instructional blocks within a class or course. It is a way for teachers to plan their teaching. When teachers design curriculum, they need to base it on what will be done, who will do it, and what schedule to follow.²¹ Often a curriculum is reviewed and redesigned following evaluation of training programs and student assessments, which should go beyond the measurement of newly gained knowledge, skill, and attitudes. Teachers design each curriculum with a specific educational purpose in mind. The ultimate goal is to improve student learning.²²

The FGDs also tried to capture students’ perceptions of the way their training program is designed. Most of students shared that their curriculum was comprehensive and included all the information needed to help students to become skilled and competent health professionals. However, some students expressed concerns in the sense that they felt the curriculum included more theoretical sessions than practical ones. They added that practical sessions started late and some modules required intense revision and re-organization. They also felt that certain parts of the curriculum focused more on basic sciences instead of actual clinical subjects as reflected in the following quote from one of the students:

In the medical assistant program, we did not study all the diseases that patients present when they come to the health centers.

Another student mentioned: “They taught us the drug sciences in semester four before we studied the diseases”, which emphasize the point regarding the need for reviewing and re-organizing the training curriculum.

Instructors Presentations Skills

Based on students’ responses, PowerPoint slides are widely used in classroom teachings, which are complemented with the use of a blackboard for further explanation by some of the teachers. Students acknowledged several advantages when teachers use PowerPoint slides in their presentations. They mentioned that PowerPoint presentations help them to understand the training material better, especially when they contain pictures, flow charts and video clips. Having those additional illustrations tends to be valued by students as they make the presentation interesting, attractive and motivating as illustrated by this quote:

In the last lecture, the teacher showed us a video of wound dressing. It was very informative and has helped with our understanding of how to do the procedure correctly.

On the other hand, students also mentioned some disadvantages of PowerPoint, especially when the instructors try to cover too much ground, when the slides are crowded with too much information and when the teacher mainly reads from the slides. They found that speaking too fast by instructors and using jam-packed slides may reduce participation of students and make it difficult for them to follow the session. A student said:

The negative point of Power Point presentation is that when the teachers move the slide very fast, we can’t catch the information and understand the significance of the illustrations.

Effective Teaching Methods

In the AHS, students are aware that there are variety of teaching methods and aids, but it was left for the teacher to decide to use the ones that are appropriate for the lesson. Most of students said they were taught through lectures, group work, seminars, practical sessions, brainstorming and role plays. However, this was not the case in midwifery schools, where students reported that teaching methods such as seminars, group work and role play are not widely used.

All students in FGDs agreed that the sessions were clearer and more useful when the instructors used a combination of different teaching methods. They found that seminars were particularly effective in delivering positive learning experiences. They also mentioned the advantages of seminars, which include developing students’ communication skills, self-confidence, ability to work together, searching for the appropriate material and peer education skills. This is illustrated by a comment from one of the students who said:

Seminars were the important teaching approaches, they made us search for the information and enhance the collaboration and competition between the groups.

Another teaching method included group work, where students were divided into small groups and given a clear task to accomplish. They felt that working in groups encouraged them to learn from each other and share their knowledge. However, they felt that both seminars and group work were of no relevance, if the teacher does not plan properly and all students do not take their individual contributions seriously.

According to the students, role play was conducted in both theoretical and practical sessions and has made it easy for them to understand and retain concepts and processes learned as well as influencing their thinking and behavior. Similarly, practical sessions which were conducted in skill labs and hospitals were an immensely important method to help them acquire and develop skills that are fundamental to their profession. However, they found that practice sessions provided in hospital settings were inadequate because of limited resources.

Student Assessment

Students mentioned that in the AHS, they take examinations, where they are expected to demonstrate knowledge and skills at the end of the course or a semester. They mentioned that, those examinations consist of two parts:

a) Written and oral examinations, also known as Objective Structured Practical Examinations or OSPE. This tool is used to objectively assess practice-based components of the course and is marked by scoring. The scores are A, B⁺, B, C, C⁺, or F.

b) Continuous assessment, which involve participation and assessments, including student-led seminars, regular class attendance and participation in discussions as well as home-work assignments.

Students acknowledged that some of their trainers informed them in advance about the assessment tools to be used both in the course and during the academic year. But this is not uniform with all trainers. They gave an example of the second year when students asked their trainers how they will be assessed, and the trainers were reluctant to tell them. This shows lack of harmony and consistency between the practice of different trainers in terms of informing students of the assessment methods in advance and in providing feed-back on students’ academic performance.

Regarding exams, respondents mentioned that exam questions were a mixture of direct questions which focus on what they have learned from the course, and indirect questions that also included what was studied in previous years or in other subjects. Nevertheless, students were generally satisfied with the way they were assessed and the marks that were given. One said:

80% of the questions in the last exam were from the course but there were about 20% of the questions, which were out of the course, which constituted difficult indirect questions.

Students were of the view that a good exam should be designed to assess the content learned in the class, and that it is a mistake to make a test with the intention that students fail it. They felt it is important that the exam has varied types of questions, eg multiple choice, open-ended questions, written tasks. In this way, the students have the opportunity to improve all their abilities and demonstrate what they have learned.

Supportive Academic Supervision

The specific roles of supervisors may differ depending on the academic discipline and departmental practice. In most of the training institutions, academic supervisors are also known as a research degree supervisor and their role is to assist and support a student throughout their academic studies. Their duties involve ensuring compliance with all institutions’ policies and regulations, including the key policies and procedures relating to supervision and examination.²³ In the context of AHS, students see their academic supervisor as the person who is responsible to arrange the training courses, especially the practical training that takes place in the health centers and hospitals. Most of the students were critical of the performance of their supervisors. They mentioned that while some of the supervisors were supportive, others were weak in providing supervision in academic activities and research.

Organizational Changes

This level of Kirkpatrick’s model measures the organizational changes that happened as the result of acquisition and application of learning by teaching staff following their training course on learning and teaching. When measuring changes at organizational levels of the AHS, CPD and MS, it is very hard to associate those changes with staff training alone. There are numerous confounding factors that need to be considered, such as the availability of educational resources, work environment, financial context, management systems and political climate, among others.^24,25

In this evaluation, organizational changes were assessed using specific qualitative (in-depth interviews) targeting the managers of training institutions where the trained instructors were based. The interviews were with managers of AHS and CPD in Gezira, North Kordofan, River Nile and Khartoum States. The evaluation considered changes related to the training programs and their effect on AHS graduates and CPD trainees. Thematic analysis was used to analyse the data, under three main themes as illustrated in Figure 6.

Figure 6 Themes of analysis for organizational changes.

Development of Training Materials

As training materials are fundamental to improving the educational and academic processes at AHS, CPD and MS, trainers’ supervisors and managers in those institutions explained that all training materials are designed at the federal level and distributed to the training institutions. The Educational Development Center (EDC) of AHS and the Curriculum Section of CPD at the federal level are the responsible bodies to develop, design and revise the programs’ curricula. However, trainers at the state level are invited to provide input and their ideas and comments are taken into account in developing, reviewing and updating training curricula.

At the state level, the main role of trainers was to deliver the curricula, and to prepare and design their lessons. Therefore, the trainer is a mediator between curriculum and students. No guidelines and manuals are in place to guide the trainers in developing their training materials and to refresh their knowledge, the quality of teaching delivered in any institution always depends on the curriculum guidelines and continuous training that are offered to trainers.

However, trainers working in those institutions in the four states who attended training on learning and teaching were able to design tools to measure participants’ reflection and satisfaction after each training course or module. Following the learning and teaching short courses, trainers have shown enthusiasm and made legitimate and essential changes in regard to (1) preparing lesson plans guided by the learning objectives, (2) preparing the teaching materials using available technology, (3) applying different teaching methods such as small group work, seminars and student-led presentations.

One of the managers said that:

I have seen our trained teachers reviewing their presentation (e.g. slides) and aligning their teaching to course objectives as the result of what they learned in the course.

A CPD manager mentioned:

Our trainers are regularly seeking support from IT colleagues to improve the design of the slides and insert videos and pictures in their PowerPoint slides.

Assessment of Learning and Supervision

Student assessment is essential to measure the progress and performance of individual students, plan further steps for the improvement of teaching and learning, and share information with relevant stakeholders.²⁶ AHS managers felt that the curriculum always determined the assessment plan and the distribution of grades between the continuous and final assessment. They agreed that as the result of learning and teaching courses that their trainers have attended, they can see evidence of improvement in their skills in assessing students, especially in developing and administering both oral and written questions. They reported noticeable discrepancies between trained and untrained trainers when it comes to student assessments. One AHS manager mentioned:

In the AHS, setting and administering oral and open-ended written exams was one of our challenges in assessing the students. However, after attending learning and teaching short courses, our trainers became more competent in assessing their students.

Evaluation of Training

Training evaluation is a systematic process to analyze if training programs are effective and efficient. At the AHS and CPD, managers and trainers use training evaluation to assess if the training programs are achieving their desired objectives, ie training and production of health care professionals with the skills and knowledge to serve in the health system.

Training evaluation includes the assessment of the effectiveness of the training programs, including by the trainers themselves to evaluate their teaching either formally by administering end of session or course evaluations and/or through discussion with the students regarding their level of satisfaction. Managers of the training institutions repeatedly mentioned that following the learning and teaching training, the trainers together with the management of the institutions designed an end of the course evaluation questionnaire which is now administered to all students. As far as they are concerned, this was an achievement they have waited to happen for a long time. Trainers also started to analyse the questionnaires that are completed by their students and use the information to revise their teaching content and methods.

Discussions

The observations demonstrate that the trainers who attended the learning and teaching course are keen to have lesson plans for their sessions. However, in some cases they do not have a contingency plan. Although the lesson plan is an important resource for teaching, there are however occasions when there are gaps between what is planned and what happens in actual classroom teaching. Therefore, having contingency plans would help trainers to come to terms with unexpected events, for example, in the case of load shedding which is common in Sudan, and how to deal with those real time interactions and interruptions when such occasions arise.²⁷

Most of the trainers observed were clear about the learning objectives and clarified them to their classes. They discussed the expected learning objectives with their classes even when they did not have them written down. They were also able to link the session with the previous session. Observing the presentation skills of the eight trainers revealed that the teachers spoke clearly and presented themselves in a professional manner.

In terms of the teaching aids and materials, PowerPoint was used properly in line with the guidance provided during the learning and teaching courses. It is clear that PowerPoint is a powerful teaching tool with both advantages and disadvantages, depending on how it is used.²⁸ Students mentioned that they found PowerPoint to be more effective when trainers are familiar with its use and make sure that they simplify the number of words on each screen, use key phrases and include only essential information. However, only half of the trainers used other teaching aids. Using different teaching aids is important since students may have varied learning styles and needs. Therefore, different teaching aids should be considered during didactic lectures to improve the students’ understanding and ensure that they are able to learn the material.²⁹

Teaching methods refer to the general principles, pedagogy and management strategies used for classroom instruction to achieve the learning objectives of the teaching session. When considering the use and application of different teaching methods, the observation showed that seven out of the eight observed sessions used different teaching methods in accordance with the learning objectives of their sessions (eg role play, small group). The observer noticed that in two observations, the trainers used the older teaching aids (chalk and the blackboard) to complement their teaching methods such as role play, small group work, and use of skill models. Studies in other settings have shown that students have different learning styles and therefore benefit from different teaching methods. Moreover, different outcome domains require different teaching methods as employed by some of the trainers who were observed in this evaluation.³⁰

Literature suggests using teaching methods that increase understanding, decrease anxiety, and increase satisfaction through combination of technology, both audio and video, written materials, and demonstrations. Teaching is often most successful when various teaching strategies are used in combination.³¹

Teaching methods definitely play an important role in student learning. The choice of a method often would depend on the learning objectives, teaching context and competency of the teacher. Different learning domains require different teaching methods. However, there is a general consensus among pedagogy experts of the importance of using a combination of teaching methods, (eg seminars and group work help students to develop communication skills, teamwork and problem-solving abilities). Likewise, using videos, group discussions and practice teaching in combination of multimedia teaching strategies significantly improve students’ understanding and learning as well as enhance the students’ memory of and learning about the concept being taught.³²

In general, assessments are meant to judge the extent to which both the teaching, and curriculum implementation process have contributed to the achievement of learning objectives. They need to be fair, objective and valid, which have to be guided by articulated assessment criteria to judge the level of student performance and to be able to define the grades each student will achieve.³³ However, the students of AHS were not aware of assessment criteria and unable to comment on how those criteria were applied in practice.

This study illustrates that students perceived academic supervision to be equal to that of service supervision, (ie regular check-up of how students are performing in the clinical settings). This demonstrates the fact that students’ academic supervision was a weak part of the AHS system and requires urgent action by the Ministry of Higher Education (MoHE) to accelerate the establishment of a strong academic supervision system, as well as to provide the necessary mechanisms to develop the skills of teaching staff in this area.

Conclusion

This assessment is a key part to ensure that the training courses on learning and teaching have improved the trainers’ capabilities and skills and consequently enhanced student learning. This evaluation focused on the knowledge gained and application of new skills, which are the key elements of the course learning objectives.

The observations and focus group discussions with students revealed that the learning and teaching course has had positive effects on the trainers’ ability to have a clear, well-stated learning objective, their presentation skills, and their use of the different teaching methods and aids. Moreover, the observations showed that those who attended the learning and teaching course are providing chances to the students to ask questions and provide them with feed-back.

The findings revealed that trainers were able to apply their learning in their teaching without any difficulties. Moreover, students and direct managers noticed the improvement of trainers’ teaching abilities and considered the courses as valuable investment to trainers, students, and training centers.

In particular, managers expressed their satisfaction with the performance of their colleagues who attended the training course in learning and teaching as evidenced from high level of satisfactions from participants and very high number of students who are achieving higher grades in their exams. They were happy with the way trainers are able to accommodate both the positive and negative results from student evaluations and felt that trainers improve their skills in the areas of teaching, assessment and student support. Some organizational changes were also instituted as a result of the learning and teaching short courses, including the development of end of course questionnaires that are now implemented across all of the classes.

It would be highly desirable to improve the educational and academic supervision by trainers, including how to guide students and mentor them in their field work and research. Lobbying with the Ministry of Higher Education to develop supervision guidelines to be widely used in all training institutions could help with this process. It is also high time to expand the learning and teaching short courses to all states to improve the trainers’ capabilities and produce competent allied health professions serving the health system in Sudan.