Impact of a National Journal Club and Letter Writing Session on Improving Medical Students’ Confidence with Critical Appraisal

Introduction

Critical appraisal is the task of assessing the quality and relevance of research in a particular context.¹ It is an important skill for clinicians of the future, who have a responsibility to harness academic medicine to improve clinical care.² Since global scientific output within healthcare is expected to double every 9 years, this task will become increasingly complex.³ Critical appraisal skills are therefore an essential outcome for medical schools and postgraduate programmes to develop.

One such programme, the Academic Foundation Programme (AFP), was developed in the United Kingdom (UK) to provide graduating doctors with a protected period of time in which to conduct research. Applications to this programme have risen by 20% in the last 4 years, from 1490 in 2016 to 1804 in 2019. This suggests that 22% of graduating doctors applied to the AFP in 2019.⁴ The selection process for this programme in many regions involves an interview which tests the applicant’s critical appraisal of a scientific abstract.⁵ Critical appraisal can also be tested in postgraduate interviews for specialty training. These skills are therefore increasingly essential to the career progression of future doctors.

Despite this, previous studies have shown that medical students in the UK have limited confidence with critical appraisal, and have little opportunity to participate.^6–8 This has been attributed to factors such as restricted time for critical appraisal teaching within the curriculum,⁹ lack of previous research experience,⁶ and inability to take part in an intercalated degree.¹⁰

Numerous studies have shown that critical appraisal tutorials and study days can have a positive effect on health care professionals’ evidence-based medicine skills.^11–16 One example approach is the journal club, where individuals have the opportunity to discuss and critically appraise a journal article in a collaborative environment. The discussion points can subsequently be drafted and submitted to the journal as a “letter to the editor.”

When combined with the use of a critical appraisal checklist, this approach can be an effective strategy for teaching critical appraisal skills.^17–19 Journal clubs provide the opportunity to actively engage students with critical appraisal in a stimulating, educational setting. Students are able to learn from their peers through group-based discussion. Learners have the chance to see first-hand the importance of developing critical appraisal skills at an early stage in their development towards becoming a clinician. By taking the traditional journal club one step further and submitting a response to the editor, students are able to gain confidence and experience in manuscript submission and develop from the positive or negative feedback from editors.

Journal clubs which extend beyond just one hospital or university can host a larger capacity and be more accessible to students who do not have a local journal club, thereby encouraging participation. Additionally, national events can also foster collaboration and exchange of ideas between medical students who might not have otherwise interacted.²⁰

Whilst the effects of local or specialty-specific journal clubs have been studied,^17,19 research on national journal clubs targeted specifically at medical students is limited.²¹ The aim of this study was to evaluate whether a national journal club session could improve UK medical students’ confidence with, and understanding of, the critical appraisal process.

Methods

The authors organised a free half-day conference for UK medical students on 30^th March 2019. This consisted of 90 minutes of interactive critical appraisal teaching (Table 1). All teaching was delivered by junior doctors who were in their first or second year of practice after medical school. The conference was sponsored by the Medical Defence Shield. The content of the conference was designed based on an initial informal focus group with six medical students. The goals of the conference were:

1. To improve medical student’s confidence with critical appraisal and writing letters to journal editors.
2. To demonstrate the importance of critical appraisal skills to students.

Table 1 Critical Appraisal Teaching Programme

Students were divided into groups of 3–6 and each group had the opportunity to discuss a unique, recent journal article under the supervision of one or two tutors. They were also given guidance on how to write up their discussion points and encouraged to submit a letter to the editor. Their tutor was available after the course to provide mentorship and guide them through the submission process.

Articles were selected from journals that allowed letters to be submitted to the editor without charge, up to 2 months after the original article was published. Students were given copies of the abstract, full text of the article and standardised critical appraisal worksheets based on freely available information adapted from the Oxford Centre for Evidence-Based Medicine.²²

The day also included a dedicated lecture on how to become involved in research, a keynote speech from a senior academic and a poster presentation competition which students were able to submit to in advance. These components were requested by medical students during the initial informal focus group.

An online cross-sectional pre-course survey was administered to students one month before the course. This consisted of demographic items, questions on previous academic experience and five questions relating to confidence with, and understanding of, critical appraisal. A post-course survey was administered immediately after the course and comprised the same items on critical appraisal, as well as the opportunity to provide feedback on the course. Feedback was collected through the use of free text comments and subsequently collated. The design of the course is summarised in Figure 1.

Figure 1 Flowchart depicting design of the critical appraisal course.

Outcomes and feedback were measured using a validated Likert scale, with a score of “1” representing strong disagreement with a statement, and “5” as strong agreement. Data were treated as non-parametric and analysed using the Wilcoxon matched-pairs signed-rank test, with statistical significance defined as p<0.05.

Explicit online informed consent was taken from participants before and after the course for their anonymised responses to be used in research. The University of Manchester Ethics Decision Tool was adhered to which states that formal ethical approval was not required as the study was an evaluation of a teaching method. A risk assessment was also completed which deemed the study to be of low overall risk.

Results

The course was attended by 98 students from 12 different medical schools in the UK, of which 74 students (76%) completed both the pre and post-course survey. Demographic data are reported in Table 2. Students were predominantly in their second and third year of medical school and only 12% had previously attended a critical appraisal workshop.

Table 2 Participant Characteristic Frequencies (%)

After the course, students reported increased confidence with critically appraising research articles (median score 2 vs 4, p<0.01), increased knowledge in assessing the methodology of research articles (median score 3 vs 4, p<0.01), increased confidence in writing letters to journal editors (median score 2 vs 4, p<0.01), and an increased understanding of why critical appraisal was important to their careers (median score 3 vs 5, p<0.01).

Students found that the course increased their understanding of critical appraisal overall (median score 5, IQR 4–5), was useful to their careers (median score 5, IQR 4–5), and found the resources provided to be useful too (median score 5, IQR 4–5). Mean values are presented in Table 3.

Table 3 Mean Results*

Analysis of free text feedback from students is presented in Table 4. Students commonly appreciated the small group format with practical opportunities to appraise a research article, as well as the structured teaching lecture at the start of the session. The most common suggestion for improvement was more time overall for the workshop, as well as the chance to be sent the specific journal articles in advance of the session.

Table 4 Analysis of Themes from Participants’ Free Text Feedback

Discussion

To the authors’ knowledge, this is the first study to demonstrate that a single national journal club session can improve UK medical students’ confidence with the critical appraisal process. Practical opportunities to learn critical appraisal in a journal club setting were valued by the participating medical students. The subsequent submission of discussion points in a letter to the editor is also important to students, as is a small group teaching format.

The results of this study align with previous studies of local journal clubs for allied health professionals in the United States of America. Green and Johnson demonstrated that an 11-week evidence-based practice course and journal club was received positively by chiropractic students.²³ Students on the course were found to have high evidence-based medicine assessment scores after the course. Landi et al also found that a series of three journal club sessions for pharmacy students improved confidence in critically evaluating clinical research.²⁴ Another study, by Maloney et al, found that a series of four journal club sessions and didactic teaching to paramedic students increased self-reported ability to find, evaluate and apply medical research articles.²⁵

These examples demonstrate that extended journal clubs in local settings are received positively by non-medical healthcare professionals. This study highlights that the journal club can also provide value to medical students. Additionally, a single journal club session can improve confidence with critical appraisal even without the presence of follow up sessions.

In the UK, Edwards et al showed in 2001 that a journal club as part of a three-week evidence-based medicine course for medical students at the University of Newcastle received positive feedback and resulted in the publication of 26 letters to the editor over three and a half years.¹⁹ However, this study involved a series of sessions and student confidence with critical appraisal was not assessed. There have since been no further published studies of journal clubs for UK medical students.

Multiple journal club sessions are harder to replicate and require a greater commitment from both students and trainers when compared to a single session. Furthermore, the course proposed here is outside of formal teaching available to students, and so does not encroach upon valuable time in the curriculum. This study demonstrates that students from across the country showed interest in this course and were willing to travel to attend. A national journal club can therefore work towards reducing the regional barriers to academic medicine, since some medical students may not have adequate local critical appraisal teaching.²⁰

An additional benefit of this course was that teaching was delivered exclusively by junior doctors. Previous research has shown that critical appraisal teaching from near-peer tutors (newly graduated doctors) provides similar test scores to teaching from more senior tutors, but is rated more highly by students.¹² This approach also meant students had access to near-peer role models to encourage them to engage with critical appraisal, and in turn with academic medicine. Moreover, the tutors themselves were able to gain valuable teaching and mentoring skills. Furthermore, they were able to refresh their own critical appraisal skills.

However, this study also has limitations. The participants of this course may not reflect the wider UK medical student population, since not all medical schools were represented. Additionally, the students attending may have been more inclined towards academia and therefore may have been more likely to have perceived benefit from the course. Nonetheless, Griffin et al found in 2010 that only 14% of 515 UK medical students had submitted an article for publication,⁷ which aligns with the publication experience of students on this course. Additionally, attendees were drawn evenly from all years of medical school (aside from final year students, for whom the course may have clashed with examinations or elective periods).

It was also not possible to fully standardise the teaching method. Although all of the critical appraisal teaching was based on the same lecture notes and teaching slides, they were delivered by different tutors who may have had varied teaching styles. Tutors were given proformas, timetables and predetermined discussion points for the journal appraisal however it was not possible to verify whether these materials were adhered to. Moreover, each group was required to appraise a different journal article, in order to ensure that subsequent letters would be unique. Therefore, some articles may have been more challenging to appraise than others.

Analysis of qualitative feedback indicated that the course could have been improved by allowing students to read and reflect on the journal article in advance of the session. Pre-allocation of article groups was not performed initially due to the risk that some students would not attend, leaving groups imbalanced. In future, this could be resolved by asking students to provide a refundable deposit to book onto the course, whilst still keeping the course free of charge. Additionally, future sessions could be run over a full day instead of half a day, to enable more time and smaller group sizes as suggested by the students. Despite these limitations, students still felt that the course improved their knowledge in assessing the methodology of research articles.

Critical appraisal has recently been identified by UK final year medical students as the most important research competency to obtain and is essential for student engagement with academic medicine.⁶ It is imperative that doctors are able to critically appraise and evaluate research in order to provide better patient care. Barriers to acquiring research and critical appraisal skills include lack of suitable academic mentors and lack of time within the medical school curriculum.^9,26 This study addressed both of these challenges and demonstrates the benefits of a poorly explored but easily replicable method to improve medical student confidence with critical appraisal.

Further research is needed to ascertain whether such courses can objectively improve critical appraisal knowledge and skills, and in turn provide more robust research output from new doctors to influence ongoing patient care. Future research should also explore the optimum format and year of medical school in which to deliver such interventions. This effort will require input from medical schools, doctors and academics. However, students themselves will be the most important voices to ensure that ongoing enthusiasm for acquiring critical appraisal skills permeates to future work.