Back to Journals » Advances in Medical Education and Practice » Volume 6

Workplace-based assessment; learner and assessor perspectives

Authors Nair B , Parvathy M, Wilson A, Smith J, Murphy B

Received 17 January 2015

Accepted for publication 2 March 2015

Published 13 April 2015 Volume 2015:6 Pages 317—321

DOI https://doi.org/10.2147/AMEP.S79968

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 3

Editor who approved publication: Dr Md Anwarul Azim Majumder



Balakrishnan (Kichu) R Nair,1,2 Mulavana S Parvathy,2 Amanda Wilson,3 Justine Smith,1 Brooke Murphy1

1Centre for Medical Professional Development, Hunter New England Local Health District, Newcastle, NSW, Australia; 2School of Medicine and Public Health, 3School of Nursing and Midwifery, Faculty of Health, University of Newcastle, NSW, Australia

Objective: To examine the acceptability and educational impact of the workplace-based assessment program for international medical graduates on candidates and assessors.
Method: A grounded theory-based qualitative analysis of the experiences of 17 candidates and eleven assessors using focus groups, interviews, and surveys.
Results: Both candidates and assessors identified positive opportunities for improved performance of international medical graduates. Their integration into the workforce was facilitated by improved communication and peer acceptance, from ongoing multifaceted feedback and time to practice skills.
Conclusion: This study showed a high level of acceptability of the Newcastle workplace-based assessment program among candidates and assessors.

Keywords: performance, competency, formative, feedback, international medical graduates, Australian Medical Council


Introduction

In Australia, the traditional path for registering international medical graduates (IMGs) has been the province of the Australian Medical Council (AMC) and the Medical Board of Australia. They are responsible for the registration of all doctors in Australia by setting the relevant standards, codes, and guidelines.1 Doctors from the UK, USA, New Zealand, and Ireland can fast-track medical registration under a “Competent Authority Pathway”.1 However, doctors from other countries are required to follow a “Standard Pathway” of examinations in English, medical knowledge, and clinical skills.

The AMC Clinical Examination is a 16-component multistation assessment of clinical skills in medicine, surgery, obstetrics, gynecology, pediatrics, and psychiatry. Most stations use role players and the entire assessment lasts for 180 minutes.1 The “Standard Pathway” pass rate is approximately 50%, and this coupled with a substantial backlog of candidates means it can take years for IMGs to obtain registration.2 This deprives IMGs of valuable work experience and the Australian medical system of much needed practitioners. Additionally, there is evidence that this assessment may not be optimal as IMGs reportedly receive more official complaints and have more adverse findings made against them compared with non-IMG doctors.3

The Newcastle workplace-based assessment (WBA) program was developed in 2010 with the aim of addressing problems associated with assessing and integrating IMGs into the Australian medical system.4 The WBA is a novel method of IMG assessment. IMGs who have been granted limited medical registration and work in an accredited Australian hospital, due to a doctor shortage, are eligible for this assessment. Details of this program have been previously published.4,5 A recent Australian Parliament Committee into registration processes for overseas doctors has recommended WBA to be used nationally.2

The WBA program consists of a 6-month assessment process focusing on clinical performance. This comparatively long assessment period provides assessors with multiple opportunities to observe IMGs in the workplace. Candidates receive ongoing feedback and time to improve their performance. It also provides time for IMGs to become familiar with the Australian medical system and the needs and expectations of patients.

Van Der Vleuten’s utility formula for assessment is a validated equation which combines the elements of validity, reliability, educational impact, cost effectiveness, and acceptability (Table 1).6 The utility of the WBA as an effective evaluation tool has been assessed by addressing each of these elements. The utility formula is as follows:

Table 1 Positive themes: assessors
Abbreviation: WBA, workplace-based assessment.

where R = reliability, V = validity, EI = educational impact, A = acceptability, and C = cost.

The face validity of the WBA is demonstrated through clinical skills assessment which includes detailed and repeated patient assessment, including history taking, examination, and planning management. Communication is an essential element of effective medicine which is adversely affected by language, cultural, and social barriers. The WBA uses 360° assessments or multisource feedback where members of the health team, including medical, nursing, and allied and administrative staff, provide feedback on the candidate’s communication, teamwork, and professional skills.4

The reliability of the assessment tools used in the WBA are well established and described elsewhere.4 Cost of the WBA is also detailed in another paper.5 In the present paper, we focus on the acceptability and educational impact of the WBA program by exploring the views and experiences of assessors and candidates.

Methods

A grounded theory qualitative methodology was used to explore the experiences and views of assessors and candidates in the WBA program.7

Participants

Candidates from five WBA cohorts (n=95) were invited by email to participate in one of two candidate focus groups. Interested candidates who were not able to attend the focus groups were offered telephone interviews. Sampling was purposive in that some candidates were included on the basis of their country of origin, sex, and area of work. All assessors in the WBA program (n=72) were invited by email to attend an assessor focus group or interview.

Data collection

Participants provided informed consent and the study was approved by the Hunter New England Ethics Committee on May 14, 2010. Interviews and focus groups were conducted by two trained researchers (UP, JS). Each session was audio recorded and the recordings transcribed for the purposes of analysis. Identifying information such as names and country of origin was removed from transcripts in order to preserve anonymity. Topic guides were used; however, the format of the interactions was informal and conversational, and participants were encouraged to express their views freely.

Data analysis

The transcribed text from the interviews and focus groups was entered into NVivo 10.0 (QSR NVivo version 10; QSR International Pty Ltd, Victoria, Australia). The source texts were closely read and reread multiple times, independently, by three researchers (UP, JS, AW). Themes were identified and coded for concepts, with coded segments then analyzed and categorized thematically. Coded segments of text were compared for similarities and differences and then categorized. One-third of all text was coded separately by two of the coders to ensure consistency in coding. Saturation of themes was reached after six interactions of focus groups and interviews with no new themes emerging.

Results

A total of 26 candidates and eleven assessors participated in the study. The candidates’ countries of origin included India, Pakistan, The Philippines, Malaysia, Pakistan, Egypt, Germany, Jordan, People’s Republic of China, Indonesia, South American countries, and South Africa. Their specialties included medicine, psychiatry, oncology, emergency medicine, and general surgery. Assessors’ specialities were psychiatry, obstetrics and gynecology, medicine, pediatrics, general practice, surgery, and emergency medicine.

The major themes arising from the two groups were similar. Both assessors and candidates saw great potential in the use of WBA due to the in-depth and broad knowledge gained by the candidates as well as the multifaceted approach to assessments. Both groups like the concept of having collegial relationships and looked forward to future working relationships on this basis. However, the negative side of these relationships included difficulty in assessing someone who works as a registrar at an intern level. The themes have been divided into positive and negative and are discussed in terms of assessors and candidates.

Positive themes

Opportunities

The majority of candidates and assessors were positive about the program and the direct benefits to IMGs as well as the indirect benefits to the health service itself (Tables 1 and 2). Assessors felt WBA raised the reputation of Australia and the Hunter New England Local Health District in educational terms while increasing the numbers of doctors interested in working in regional Australia. Candidates felt welcomed and valued the advice and education they received during their training.

Table 2 Positive themes: candidates

Communication skills

Candidates and assessors found the ongoing revision of education and knowledge improved clinical and communication skills, as well as developing expertise and workplace excellence. Cultural and communication differences between their country of origin and Australia were greater than most candidates had expected. The appropriate way to address patients and colleagues, especially senior staff, was in contrast to most candidates’ expectations. Time was needed to understand and accept the use of humor and informality of Australian culture. Candidates knew it was essential to quickly gain the necessary communication skills to overcome these differences in order to pass registration and to practice effectively, and they felt the WBA was an excellent means of achieving this.

Acceptance

Candidates and assessors acknowledged the advantage of being accepted as part of the health team. They described how the WBA program increased their confidence, networking, and team building. Candidates were impressed and surprised at the accessibility of their assessors and felt included and supported in their work. Assessors felt less pressured knowing the candidate had time to improve their skills and felt candidates were open to constructive criticism.

Holistic education

Education through ongoing feedback provided candidates with important clinical and practical information. Candidates felt that having a variety of assessors providing multifaceted advice and expertise improved their knowledge, especially when the information was demonstrated on a real patient. Workplace-based training provided candidates with information they would not have otherwise received or known that they needed, such as administrative input regarding patient records. At no time before the WBA were candidates taught anything about medical records, paperwork, or how to access electronic patient data. The time requirement for WBA was viewed positively by most candidates who felt it provided time to prepare for assessments and more opportunity to gain clinical skills.

Negative themes

Inconsistency

There was a recurrent theme among candidates that assessors needed more consistency in terms of competence and objectivity during assessment procedures. While candidates should be examined at an intern level, many felt their assessments were harder because their assessors knew they were registrars. The assessors admitted feeling conflicted examining at an intern level if the doctor was working as a registrar or resident (Tables 3 and 4).

Table 3 Negative themes: candidates

Table 4 Negative themes: assessors
Abbreviation: CMOs, Career Medical Officers.

Intimidation

Issues of intimidation, both psychological and physical, were raised by assessors. The most serious were experienced by female assessors providing negative feedback to male candidates. It was felt this situation could be addressed by having assessment sessions video recorded. There were also instances where candidates felt demeaned by consultants who referred to WBA as a “soft” or “easy” option as opposed to the AMC process. Some candidates said the assessment period was too long and created burdens, such as being away from family and the increased cost of living in maintaining homes in different cities.

Discussion

The WBA is a novel approach to assessing IMGs which acknowledges the complex mix of knowledge and skills that defines medicine. What is important for patients and the society is not competency (what the doctor can do) but performance (what the doctor does).8 WBA is a performance-based assessment and the candidates and assessors acknowledged and embraced this approach. Assessment drives learning and students will always focus on learning what will be assessed. However, WBA is assessment for learning.8 The longitudinal assessment by multiple examiners with feedback was built into each clinical assessment. This design provides the candidates with the opportunity to modify and improve their performance over time.

This study showed that candidates and assessors felt the WBA process was positive and presented good opportunities for performance improvement and integration of the IMG into the Australian health care environment. This was especially so in the areas of communication, peer acceptance, multifaceted feedback, and time to process and practice their skills.

A major response in this study from candidates and assessors was that assessments performed on real patients, in a workplace by practicing clinicians, had true validity. All the tools used in WBA are reliable from previous studies. We have discussed the cost of WBA in another paper5 and it is reassuring to note the cost for the Organisation is only $10,000 (per candidate), and this is a small price to pay for a long-term investment in the safety of the patients and future of the IMG.

Limitations

The WBA is currently only available for doctors already working in the health care system in accredited sites, and most candidates are not in this situation. This raises the possibility of selection bias since these doctors have gone through a recruitment process and have knowledge of the health care system. WBA is also a labor- and time-intensive process for candidates and assessors.

Implications

Given the supply shortage of doctors in Australia, the Newcastle WBA program is an innovative response to improve the pathway for IMG registration. This study showed the program is acceptable to both the candidates and assessors while reinforcing its educational impact and validity. It is cost effective and ensures the successful integration of the IMGs into our health care system and the community. It needs to be extended to more sites and IMGs. Moreover, many of the postgraduate training bodies are introducing WBA and the lessons learnt from this work can influence future assessments.

Acknowledgments

The program was funded by the Health Workforce Australia. The work was carried out in accordance with the Declaration of Helsinki, including but not limited to there being no potential harm to participants, the anonymity of participants is guaranteed, and the informed consent of participants was obtained.

Author contributions

BRN, MSP and BM designed the workplace-based assessment program. All authors were involved in the design, data collection, analysis and the writing of this study. All authors agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.


References

1.

Australian Medical Council. Applying to the Australian Medical Council A guide for non-specialist and specialist international medical graduates. Kingston: Australian Medical Council Limited; 2012. Available from: http://www.amc.org.au/joomla-files/images/publications/applying-to-the-amc.pdf. Accessed April 6, 2015.

2.

House of Representatives Standing Committee on Health and Ageing. Lost in the Labyrinth: Report on the inquiry into registration processes and support for overseas trained doctors. Canberra: The Parliament of the Common wealth of Australia; 2012.

3.

Elkin K, Spittal MJ, Studdert DM. Risks of complaints and adverse disciplinary findings against international medical graduates in Victoria and Western Australia. Med J Aust. 2012;197(8):448–452.

4.

Nair BR, Hensley MJ, Parvathy MS, et al. A systematic approach to workplace-based assessment for international medical graduates. Med J Aust. 2012;196(6):399–402.

5.

Nair BK, Searles AM, Ling RI, Wein J, Ingham K. Workplace-based assessment for international medical graduates: at what cost? Med J Aust. 2014;200(1):41–44.

6.

Van Der Vleuten CP. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1(1):41–67.

7.

Strauss A, Corbin J. Basics of Qualitative Research Techniques and Procedures for Developing Grounded Theory. 2nd ed. London: Sage Publications; 1998.

8.

Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010;341:c5064.

Creative Commons License © 2015 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.