Evaluation of teachers training workshop at Kirkpatrick level 1 using retro–pre questionnaire

Introduction

Medical education comprises 3 key components: a curriculum, an educational environment and the teachers or faculty members. Teachers not only influence the teaching and learning process but also play an essential role in shaping the other 2 components: a curriculum and an educational environment. To improve and maintain the quality of medical education, faculty development programs are indispensable. Teachers training focusing on teaching learning methodologies, assessment tools, and approaches that motivate the teachers and enhance their confidence, and it is one of the fundamental programs of faculty development. This enriches the capacity of faculty in teaching and learning and assessing students’ performance.^1–5

The evaluation of teachers training program documents the enhancement of knowledge and teaching skills of the trained teachers.⁶ Retrospective post-then-pre questionnaire design is one of the instruments used to evaluate the efficacy of training program. It assesses the learners’ self-reported changes in knowledge, awareness, skills, confidence, attitudes, or behaviors. It is used to avoid the sensitivity of pretest and reduce the speculation.^7,8

Health Professionals Education and Research Center, Chitwan Medical College (CMC), Bharatpur, Nepal, has been organizing teachers training workshop since September 2014 in order to strengthen the capacity and competence of faculty members in teaching learning and assessment methods and to boost their level of confidence in this aspect. The main objective of this study was to assess the self-reported perceived confidence of faculty members after participating in basic teachers training workshop at Kirkpatrick level 1 using retro–pre questionnaire, but the perception at level 2a and, to some extent, knowledge/skills at level 2b were also assessed. The rationale of the study was as follows: 1) participants’ reactions, perception, and self-confidence can usually be measured with retro–pre questionnaire immediately after the training program, and 2) it is believed that the positive self-reported perceived confidence may motivate teachers and bring positive behavioral changes among them for teaching learning, educating the students, and assessing their performance.^9,10

Methodology

The retrospective post-then-pre design questionnaire was used for assessing faculty members’ self-reported perceived confidence after completing basic teachers training workshop at Kirkpatrick level 1 (evaluation of reaction). In addition, the perception at level 2a and, to some extent, knowledge/skills at level 2b were also assessed with the same questionnaire. There are 4 levels of Kirkpatrick’s model of evaluation – level 1: evaluation of reaction; level 2: evaluation of learning (level 2a: attitudes/perceptions and level 2b: knowledge/skills); level 3: evaluation of change in behavior; and level 4: overall impact of training (level 4a: organizational practice, level 4b: student benefit and level 4c: patient benefit).^9,11

After piloting the retro–pre questionnaire in the 2nd teachers training workshop, it was subsequently used for the assessment during the 3rd, 4th and 5th teachers training workshop. Each statement was assessed using a 4-point Likert scale: not confident, somehow confident, very confident and highly confident for both pre and post responses.

Written informed consent was taken from the participants.

Each teachers training workshop was of 6 days duration. The 3rd, 4th and 5th teachers training workshops were held at Health Professionals Education and Research Center, CMC, Bharatpur, Nepal, during July 26–31, 2015; February 27–March 3, 2016; and September 5–10, 2016, respectively. Altogether, 60 (18 in the 3rd and 21 each in the 4th and 5th workshops) faculty members participated in the teachers training workshop from clinical sciences, basic sciences, public health, and nursing departments of CMC. The national and regional resource persons facilitated the training besides local resource facilitators.

The main sessions conducted were on educational objectives, curriculum and role of faculty member, principle of adult learning, behavioral objectives, feedback skills, communication skills, facilitation skills, interactive teaching/learning, small group discussion, role play, clinical skills teaching/learning, community-based learning, ethics for professional educators, microteaching, lesson planning, audiovisual aids, problem-based learning, and assessment tools. Most of the sessions contained a tutorial on the subject with brainstorming, group work in small groups (3 small groups with 6–7 participants in each) and presentation of group work in a large group with discussion.

The collected data were checked for completeness, accuracy and consistency. The collected data were coded and entered in Epi Data 3.1 (EpiData Association, Odense, Denmark) and exported to IBMS SPSS version 20 (IBM Corporation, Armonk, NY, USA) for analysis. Data were categorized as 0 for not confident, 1 for somehow confident, 2 for very confident and 3 for highly confident. Later, the “somehow confident” and “very confident” categories were merged into the “confident” category. Altogether, 30 statements were used, and the maximum possible score was 90 for each participant. The data were analyzed by frequency, percentage, mean (standard deviation) for continuous variables and median (interquartile range) for ordinal variables, and p-values were calculated for significance. p-values were computed using Wilcoxon signed-rank test for the pre and post scores of self-perceived confidence of the participants, while Mann–Whitney U-test was used for computing p-values for comparison between the 2 groups.

The details of the groups were as follows:

Group I – received some sort of training on teaching learning before joining CMC.

Group II – received no training on teaching learning before joining CMC.

Results

Out of 60 participants (18 in the 3rd and 21 each in the 4th and 5th workshops), 58 responded to all statements included in the retro–pre questionnaire. The response rate was 96.67%. Males accounted for 70.7% of the participants, while females were 29.3%. The mean age of the participants was 34.14±5.15 (min, 25 years and max, 54 years). More than half of the participants (51.7%) were engaged in teaching learning prior to joining the CMC, while the remaining (48.3%) joined CMC immediately after receiving a postgraduate qualification. Seven (12.1%) participants received some sort of training on teaching learning before joining CMC.

The self-reported perceived confidence on 30 statements/items were analyzed for 3 categories (not confident, confident and highly confident) using a Likert scale. There was a marked difference found in the level of confidence of the participants after training (Table 1).

Table 1 Self-reported perception and confidence of the faculty members before and after teacher training workshop on various sessions conducted during training Abbreviations: OSCE, objective structured clinical examination; OSPE, objective structured practical examination.

Overall, the self-reported perceived confidence level of the participants was significantly increased after teachers training workshop (Table 2).

Table 2 Overall self-reported confidence level of the participants before and after teachers training workshop Note: p-value is computed using Wilcoxon signed-rank test.

Participants (n=7, 12.1%) who received some sort of training on teaching learning before joining CMC were comparably more confident prior to teachers training than those who received no training on teaching learning before joining CMC (n=51, 87.9%) (p-value <0.001). Although the level of confidence after teachers training was comparably increased more in those participants who received no training on teaching learning before joining CMC, this was not statistically significant (p-value =0.141). The observed median percentage change in self-reported perceived confidence was 29.63% after teachers training in those who received no training before (n=51, 87.9%); this was 2 times more than those who received some sort of training prior to teachers training (n=7, 12.1%), and the result is statistically significant (p-value 0.018) (Table 3).

Table 3 Comparison of self-reported perceived confidence level of the participants who received no training on teaching learning compared to those who received some sort of training (before and after teachers training workshop) Note: P-value is computed using Mann–Whitney U-test.

Discussion

Faculty members represent one of the key resources at any medical college. The academic success of a medical college depends upon the performance of faculty members.¹² Teachers training may improve quality, professionalism, and effectiveness of teaching and improve the self-confidence of the teacher, thus enhancing their performance.¹³ Our study assessed the self-reported perceived confidence of faculty members after participation in a basic teachers training workshop using retro–pre feedback questionnaire at Kirkpatrick level 1 (evaluation of reaction) besides the perception at level 2a and, to some extent, knowledge/skills at level 2b. It is anticipated that the positive self-reported perceived confidence may motivate teachers and bring positive behavioral changes among them for teaching learning, educating the students, and assessing their performance.¹⁰

Our study revealed that after teachers training, the highest percentage of participants perceived themselves as being highly confident in using audio–visual aids (58.6%), developing a lesson plan (44.8%) and conducting oral examination – viva (43.1%), whereas the least percentage of the participants perceived themselves as being highly confident in community-based learning(10.3%), designing problem-based learning (18.9%), knowledge on curriculum and role of faculty (19%), principles of adult learning (19%) and designing modified essay questions for examination (19%). The low score could possibly be due to less time allocated for these sessions.

Overall, the self-reported perceived confidence level of the participants of the 3rd (pre-training median 57, post-training median 70, p<0.001), 4th (pre-training median 51, post-training median 67, p<0.001), and 5th (pre-training median 51, post-training median 68, p<0.001) training workshops was significantly increased after training.

The retro–pre questionnaire was used to assess the learners’ self-reported changes by Rajalakshmi et al¹⁴ at Kirkpatrick level 1 (evaluation of reaction). They evaluated participants’ reaction (self-perceived) at the end of workshop on communication skills for health care providers and reported significant self-perceived improved changes in communication skills of the participants.¹⁴ Masood and Usmani¹¹ evaluated medical teacher training program through Kirkpatrick’s model at all 4 levels. Participants in their training perceived that their learning considerably improved after training at Kirkpatrick level 1 (evaluation of reaction).^9,11

Our study also showed a comparably increased level of self-reported perceived confidence after teachers training workshop among the participants who received no training on teaching learning prior to the training workshop (percentage change: 29.63) compared to those who had received some sort of training (percentage change: 10.96). Statistically, this difference is significant.

The findings may not be generalized as this study was conducted in 1 institution. This type of research must be conducted in other institutions organizing similar faculty development programs.

Conclusion

Overall, the self-reported perceived confidence level of the participants was significantly increased after teachers training workshop. A significant change in the level of self-reported perceived confidence was noticed after teachers training workshop among the participants who received no training on teaching learning prior to the teachers training workshop. Further research is required to explore the behavioral changes among the teachers and their teaching performance after teachers training program at Kirkpatrick level 3 and in terms of organizational practice (Kirkpatrick level 4a), student benefit (Kirkpatrick level 4b) and patient benefit (Kirkpatrick level 4c).

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Harden RM. International medical education and future direction: a global. Acad Med. 2006;81(Suppl 12):S22–S29.
2.		McLean M, Cilliers F, Van Wyk JM. Faculty development: yesterday, today and tomorrow. Med Teach. 2008;30(6):555–584.
3.		Davis MH, Harden RM. Planning and implementing an undergraduate medical curriculum: the lesson learned. Med Teach. 2003;25(6):596–608.
4.		Steinert Y, Mann K, Anderson B, et al. A systematic review of faculty development initiatives designed to enhance teaching effectiveness: a 10-year update: BEME Guide No. 40. Med Teach. 2016;38(8):769–786.
5.		Harden RM. Presentation on staff development. Presented at: 6th Asia Pacific Medical Education Conference, National University of Singapore, Singapore; February 21–22, 2009.
6.		Chapman A. Donald Kirkpatrick’s Learning Evaluation Model 1959; review and contextual material Alan Chapman 1995–2007. Available from: http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm. Accessed on May 01, 2018.
7.		Program Development and Evaluation. Using the Retrospective Post-then-Pre Design, Quick Tips #27 [Internet]. Madison, WI: University of Wisconsin Extension; 2005 [updated July 25, 2005; cited September 17, 2013]. Available from: https://fyi.uwex.edu/programdevelopment/files/2016/04/Tipsheet27.pdf. Accessed May 1, 2018.
8.		Piryani RM, Shankar PR, Piryani S, et al. Assessment of structured physical examination skills training using a retro-pre-questionnaire. J Educ Eval Health Prof. 2013;10:13.
9.		Leslie K, Baker L, Egan-Lee E, Esdaile M, Reeves S. Advancing faculty development in medical education: a systematic review. Acad Med. 2013;88(7):1038–1045.
10.		Lee SS, Dong C, Yeo SP, Gwee MC, Samarasekera DD. Impact study of a faculty development programs for positive behavioural changes among teachers: a case study. Korean J Med Educ. 2018;30(1):11–22.
11.		Masood RQ, Usmani MAW. Evaluation of medical teacher’s training program through Kirkpatrick’s model. Khyber Med Univ J. 2015;7(2):76–80.
12.		Ries A, Wingard D, Morgan C, Farrell E, Letter S, Reznik V. Retention of junior faculty in academic medicine at the University of California, San Deigo. Acad Med. 2009;84(1): 37–41.
13.		Breckwoldt J, Svensson J, Lingemann C, et al. Does clinical teacher training always improve teaching effectiveness as opposed to no teacher training? A randomized controlled study. BMC Med Educ. 2014;14:6.
14.		Rajalakshmi M, Suguna E, Dongre AR. Evaluation of workshop on communication skills for health care providers in Pondicherry. Natl J Res Community Med. 2016;5(4):212–216.