A Proposed Framework to Develop, Describe and Evaluate Peer-Assisted Learning Programs

Introduction

Peer-assisted learning (PAL) is increasingly used in different fields of education, including medical education, due to its established advantages.^1–3 It refers to the acquisition of skill and knowledge from matched companions of similar social group via active support.³ These companions are not professional educators or teachers.¹ In medical and health professions education, the large amount of literature research about the advantageous role of PAL reflects its success as a learning strategy.^4–8 This is expected with the several well-documented benefits for both the learners and institutions in addition to being well-received and accepted by many students.⁹ PAL programs allow more time for individualized learning between students, and it allows direct interactions between same-level students that promotes active learning and comfortable discussions.¹⁰ PAL also allow a greater understanding of curricula as they are shared by the tutors and tutees.¹⁰ From the teachers’ perspective, it saves more time for the teachers to move to the next lessons and it encourages the students to ask the questions they all share a difficulty in their understanding.¹¹ From the financial perspective, PAL has a lower cost than hiring teachers or professional staff members for the educational process.¹²

Though PAL programs have been documented to have several advantages, the outcomes reported are inconsistent among different studies with few authors even reporting reduced outcomes ie in the students’ scholastic achievements.^9,13 Such inconsistency reflects an issue in its design or implementation. Each PAL program had a specific design, specific implementation strategy and specific monitoring techniques.^14–20 Differences in any of these aspects may explain the inconsistency of reported outcomes among literature reports.

The Faculty of Medicine at Umm Al-Qura University (UQU) has implemented many PAL programs,^21,22 but these programs have never been evaluated in depth. Evaluation of any educational or learning program is essential for identification of defects, strengthening the weak points and improvement of the outcome.²³ In the literature, however, only scarce data are available relating to evaluation of PAL programs, and to the best of our knowledge, there are no best practice guidelines to improve PAL programs. Though many schools have shared their PAL experiences,^{15,20,24–26} it is difficult to compare these experiences with that of others and with our own institution due to variability in their description and components, rendering us unable to conclude what factors may have what effects and to what contexts do these effects manifest. Therefore, our aim in this work was to devise a framework to describe and develop PAL programs that helps universities, schools and other institutions to compare their experiences and evaluate their implemented programs.

Approach to Framework Development

Literature Review

To build a framework that can describe, develop and evaluate a PAL program, we initially conducted a scoping review via searching Pubmed, ERIC and google scholar using the keywords: Peer-assisted learning, PAL, medical education, framework, and PAL educational intervention. Of 897 related results, the titles were quickly screened to choose articles that describe PAL programs experiences and articles that explore the underpinning theories of PAL interventions. Of a 234 articles related, the abstracts were read for further filtration. Then a final 52 results were evaluated for criteria selection.

We chose the framework proposed by Ross et al¹⁹ because it focuses specifically on the planning and implementation of PAL programs. We also included the Haji et. al model of program evaluation²⁷ to help us lay down the theoretical ground on which we based our framework.

Framework Development

The framework we propose was developed in four steps. The first step was generating criteria to evaluate PAL programs, we did that through interviewing two of the faculty members who lead the PAL programs at the Faculty of Medicine at UQU. In addition to the interviews we looked in the literature for suitable criteria for the evaluation of PAL programs. The interviewed with the faculty members were structured containing three questions. The first question was “what are the factors that you consider when developing a PAL program?”. The second question was “what are the factors that you consider when implementing a PAL program?”, and the last question was “what are the factors that you consider when evaluating a PAL program?”. At the end of the interview, the interviewer would go through the answers of the faculty member to confirm them.

The second step was the development of the framework template where we utilized the Haji et al model of program evaluation.²⁷ The model developed by Haji et al comprises seven essential elements for holistic evaluation of educational programs. These elements emphasize not only evaluating the outcomes but to also taking into consideration the context, the planned and emergent processes and outcomes, and the planned and emergent theory.²⁷ Therefore, the evaluation goes beyond whether the implemented program worked or not to assessment of how and why it worked, and what else happened,²⁷ this aligns with the complex and sensitive nature of PAL interventions which we elaborate on in the following section. After adjusting the order of these elements, we ended up with a template comprised of four main categories, those are context, theory, outcomes and processes and emergence. The answers we obtained from the first step (faculty members interview and literature review) were fitted under the most relevant main category. The third step for development of our framework was sending the developed framework to a third PAL program leader at the Faculty of Medicine at UQU for testing the questions, and then another interview was made with the faculty members to have their feedback. Lastly, we compared and contrasted our work with the framework developed by Ross et al¹⁹ as it is the only model that focuses on the planning and implementation of PAL programs from our literature review. This step finalized our framework.

The Resulting Framework

Table 1 briefly describe the components of our proposed framework. The framework is composed of thirteen criteria distributed under four categories ie context, theory, outcome and process, and emergence. In the following section we will provide a detailed account of all the criteria of the framework; Table 2 contains a summary of the framework.

Table 1 A Brief Overview of the Proposed Framework for Description, Development and Evaluation of a PAL Program

Table 2 A Detailed Description of the Proposed Framework for Description, Development and Evaluation of a PAL Program

In this manuscript, our aim is not to decide or review what the best practices for developing, implementing and evaluating a PAL program are, as the evidence about this is still lacking. Our main aim is to propose a systemic framework that provide a comprehensive approach about the different strategies and methods that have the potential to improve the PAL intervention and to avoid certain practices that could hinder its effectiveness.

Context

In an educational program, the context refers to the variable conditions of learning that can change from one setting to another such as the curriculum, the facilities, the equipment, the learning style, and the level of learner.²⁸ In our proposed framework, we suggest evaluating three aspects of the context those are the institution, the curriculum and the students.

Regarding the students aspect, there are numerous ways in which it can impact the PAL program. The context of teaching has been reported to affect the learning process; and the more experienced the educator is the more capable they are to make use of the context and environment in the educational process.²⁸ In PAL programs, the students teach each other; and being novice to the educational process, their use of the context is expected to be limited. Two other examples of students context from Bennett et al study which can affect the PAL program relate to the students exam-focused mindset and their preference for teachers.⁴ Bennett et al found that many students are exam-focused, and this mindset can negatively impact a PAL intervention if the students deem it to be of no value to their grades.⁴ In the same study by Bennett et al, the students viewed learning with the faculty as the primary purpose of their clinical rotation; this lack of insight that the student showed regarding the benefits of being taught by peers might explain the undesired outcome.⁴ Therefore, defining the context of a PAL program is essential to determine its impact on the witnessed outcome.²⁹

The curriculum aspect of the context can also impact the PAL program, an illustration of that can be evident when planning to adopt a PAL program in a student-centered curriculum is different than adopting it in a teacher-centered one; in student centered-curriculum, students are generally better equipped to receive and engage with PAL.³⁰ This can lead to better outcomes as opposed to students whom are oriented to teacher-centered curriculums, those students will require much more effort to make them fully embrace PAL.³⁰

The third aspect of the context is the institution; this can be demonstrated in the way that some schools might adopt PAL widely because of its reduced cost in comparison to traditional teaching. This approach might be harmful to the students learning because many experts consider PAL as a supplemental strategy that cannot replace traditional teaching, despite its many documented benefits.³¹

For a detailed evaluation of the context, we added the objectives of learning in our proposed framework under this section. It is well-established that appropriately-defined and well-articulated learning objectives are fundamental in any educational program to help the students to focus their efforts on, to guide their learning strategies and to direct their activities.³² Moreover, learning objectives clearly define the expectations from the educational program assigned, help evaluation of learners and curricula, and direct the methods of education.³³

The other criteria that fall under the context is the logistics criteria which has many aspects and can be unique to each program. In our proposed framework, we included the shared points ie the venue where program will take place, the administrative work conducted and the financial aspects such as funding of the program.

Theory

Variable definitions for PAL programs exists in literature, which might have confused organizations about the actual constituents of the PAL programs and might have lead subsequently to different implementation strategies and different outcomes.¹⁸ Callese et al in their commentary about the advancing theory of PAL, thoroughly discussed this issue and attempted to overcome it by revisiting the definition of PAL.³⁴ They sought to construct a more descriptive definition of PAL and stated that: PAL is a social practice of mutually beneficial personal and professional development among learners interacting as status equals, characterized by safety, comfort, motivation through relevance, and intellectual risk-taking.³⁴ This definition revealed the complexity of a PAL intervention via embracing the educational theories behind it.

This section of our proposed framework is theory evaluation. PAL is based on two educational theories ie cognitive congruence and social congruence.³⁵ Cognitive congruence reflects the ability of tutors to engage with students at an appropriate level, drawn from a shared understanding of the material under discussion, and social congruence refers to the willingness to become involved with students in an authentic way.³⁶ In our proposed framework, evaluation of these two theoretical concepts was emphasized through multiple steps including the criteria presented in Table 1.

When designing and implementing a PAL program, it is essential to adhere to the educational theories in each step of the designing and implementation to achieve the favorable outcomes.^34,37,38 The educational theories underlying PAL include the cognitive congruence and the social congruence.³⁶ In our framework, we emphasized including both theories for designing and implementation of a successful PAL program. We included eight items under the section of theory to fulfill the cognitive and social congruence criteria ie the content (subject, purpose, type and level), the program design, the number of tutees and tutors, the participation and attendance, the program relation to the curriculum, the duration and time of the program, the tutors recruitment and preparation, and the faculty involved. Each of these items might have an impact on the program outcome.^19,39

The content type and nature of a PAL program might explain why some PAL programs are more effective than others.² Some studies suggested that PAL is significantly more effective when the course being taught is clinical in nature as opposed to a theoretical nature.⁴ The theory of social congruence can be more pronounced during practical skills teaching because tutees will be less stressed and more comfortable with peer teachers as opposed to expert teachers which will promote learners’ confidence.³⁶

The tutors’ and tutees level is another significant contributor to the PAL program outcome.⁴⁰ In our proposed framework, we included several items to describe and decide the tutors and tutees involved in the educational process ie tutors recruitment and preparation, number of tutors, and the participation and attendance. Preparation of tutors is the responsibility of the program leaders.⁴¹ The role of the faculty during PAL sessions is facilitating and monitoring the relationship between their students and to ensure delivery of the correct contents.⁴² When the tutors are at the same level, the spoken language will be at more appropriate level to the tutees which will promote the process of knowledge delivery, and they will feel more confident to ask questions even if it’s seems stupid.⁴³ This reflects the implementation of both the cognitive and the social congruence theory during the educational process ie the tutee will receive knowledge from a cognitively-matched partner, will not feel embarrassed when asking a seemingly stupid question, and will be motivated to participate actively in educational sessions.³⁶ This, however, may not be true in the case of older tutors (eg from a higher academic level) or in a large group of students.²² If a student is going to give a didactic lecture, for example, the dynamics of the group are expected to be considerably different than if a teacher was giving the lecture. The students will be more open to discuss their opinions, ask about debatable points and feel free to express and exchange their ideas.⁴³ Many factors should be considered during determination of the numbers of tutors and tutees in a group such as the students’ level, the subject learnt, the nature of the course given, the program design and the resources needed.⁴² In our proposed framework, we included the number of students as an item under the section of theory using the classification proposed by Olaussen et al for the ratio of the tutors to tutees a peer mentoring design a ratio of (1:1–2) which requires a lot of resources, but it is beneficial when targeting disadvantages students. On the other end of the spectrum there is peer didactic in which a larger ratio can be utilized (1:>10), this style of PAL usually used in situation where the resources are very limited, and/or when the focus of the program is on improving the peer tutors skills of teaching and presenting because they will have to prepare and present in front of a larger groups. The third classification proposed by Olaussen et al is peer tutoring and it hold a middle ground between the previous two classes, its favorable ration is (1:3–10) this style of PAL is more common in practice, because it requires less resources, and it increases the ability of the institute to track the performance of the participants involved in the programs.

Successful PAL programs do not rely only on the tutors and tutees involvement, but also on the program design and the involvement of faculty members.^19,20 Therefore, we emphasized including these items in the program description. We included the program instructional design, the program relation to the curriculum being educated, the duration of the program and the faculty involvement. Program design, duration and content have a non-doubtful effects on the outcome.¹⁹ In literature, the duration of the implemented PAL programs are highly variable (from few hours to several months) which makes the comparison between these programs challenging.^44,45 Therefore, detailed description of such an item is essential for future determination of the best practice guidelines development. The role of faculty members should also be clarified on designing and evaluating a PAL program eg planning, organization, implementation or monitoring.⁴⁶ It is important to keep in mind that the less explicit the involvement of the faculty members, the more likely the student-centered nature of the initiative will be maximized.⁴⁶ Another important point to be taken into consideration is that the junior faculty staff (eg teaching assistants) should not be considered as peer tutors even if their level is close to the students “cognitive congruence”, because this might interfere with the principle of social congruence.⁴⁶

Outcomes and Implementation Processes

Evaluation of the implementation process and the outcome of any educational program is integral during the process of assessment to delineate the defects and the difficulties or challenges that hinder its appropriate implementation and optimized outcome.^47,48 For evaluation of the implementation process, we suggest collecting feedback from students (tutors and tutees), faculty members involved in the PAL program implementation, other concerned faculty members and faculty leadership; for assessment of outcomes, we proposed devising a pre-and post-intervention assessment of the students and to keep all the data for future comparison (Table 2).

Though the PAL programs have been increasingly used among different educational fields in different countries, there remains a lack of a unified clear strategy for implementation, monitoring and evaluation of outcomes. This again adds to the confusion when an organization decides to adopt a new PAL program in its educational programs. In our framework, we emphasized including the strategy set for implementation, monitoring and evaluation of the implemented educational program to allow accurate comparisons with other programs. Under this section, we included feedback evaluation and students assessment.

Feedback is integral in any educational program to identify the strength and weak points, to maximize the students’ benefits at any stage of learning and to take prompt actions to solve evolving challenges.⁴⁹ For a feedback to be effective, it must be taken from both involved and from non-involved individuals. Feedback from involved individuals is crucial to identify the challenges during implementation, uncovering the weak and strength points and to evaluate the effectives of the implemented program.^50,51 Collecting feedback from non-involved individuals will also enhance identification of their un-biased views about the results of the program.⁵² In this proposed framework, we ensured collecting feedback from students (both tutors and tutees), faculty members involved in PAL programs, non-involved faculty members and faculty leadership.

The outcome is the final and most important aspect in any educational program, and evaluation of the outcome is an important means for comparing different programs. In PAL literature, diverse outcomes are reported from different studies because the outcome items are not unified. Examples of reported outcomes in literature included improvement in scholastic scores, impact of PAL on total cost of the course.^53–55 It is important to unify the outcome measurements among PAL programs to allow comparisons. An outcome that holds a substantial significance is students’ academic achievement. In our proposed framework, we suggested devising a specific way for assessment of the students before and after joining the PAL program, and to have documented records with the pre and post-intervention scores for future comparisons.

Emergence

In the emergence section, our concern here is mainly about emergent events that evolve during the process of implementation and the unplanned outcomes. For a more detailed description of that we suggest recording data about the program evolution eg program age, revisions and updates. Good and bad experiences are to be documented at this section as well. The evolution record is meant to keep track of the past experience, the revisions needed, the updated forms and the program age. Monitoring programs evolution over time help leaders to make reflections about the best practice strategies learnt over time.⁵⁶

Value of the New Proposed Framework

This new proposed framework provides a practical solution for programme designers looking to introduce PAL into their curriculums or evaluating existing ones. Though Ross et al’s framework is an excellent effort focusing on PAL planning and implementation, our framework is rather a more specific and practical framework that provides more details on what makes PAL an effective intervention. Moreover, our framework can give the designers better insight when designing PAL interventions. The framework also focused on stress the importance of the educational theories underlying the PAL and how these theories are still evolving. The effectiveness of a PAL programme is a result of a delicate group dynamic, namely social congruence and cognitive congruence; a PAL programme reporting positive outcomes without evidence of social and cognitive congruence can be regarded as coincidental thus not reliable. For evaluation of this proposed framework and identification of its potential deficiencies, future conduction of a Delphi study is recommended.

Conclusion

Development of a clear well-defined framework for description, implementation and evaluation of a PAL educational program can provide a foundation to unify the terms organizations use to communicate the parameters of PAL programs and overcome the jargon about PAL in the literature. It also can provide comparisons between the programs in an attempt to set best practice guidelines in the future for PAL program developing and implementation. Our proposed framework is intended to be a practical and straightforward tool that will give the designers valuable insight about the factors that they need to consider in their PAL program.