Stakeholders’ Views and Confidence Towards Indonesian Medical Doctor National Competency Examination: A Qualitative Study

Introduction

Along with the increasing number of medical schools, each with its own distinct excellence and shortfalls, and the increasing mobility in the medical workforce, the Medical Licensing Examination (MLE) can be seen as an attempt to ensure the graduate’s ability in delivering quality care and improving patient safety.¹ The implementation of MLE differs between countries in terms of policy, regulations and the methods utilized. Despite the context-specific technicalities, some common features in the enactment of the MLE are shared. Most countries enact MLE as the requisite to grant the MD degree or as provision for registration and medical practice. Some countries even have a policy to limit the attempts to retake this exam rendering it as a high stakes assessment for medical student’s career. Multiple-choice questions (MCQs) are the most common form used in the MLE while some opt to add performance assessments in the form of the Objective Structured Clinical Examinations (OSCE) as part of the licensing examination.^2–5 Most countries conduct MLE at the end of the clinical training phase.^4,6,7

The Indonesia Medical Doctor National Competency Examination (IMDNCE) has been administered since 2007 in the form of 200 MCQ items, and later in 2013, 12 stations of OSCE were added. This national examination is expected to ensure the quality of medical graduates from all Indonesian medical schools. It is conducted at the end of clinical training, which was done in the final year of medical college as students before passing the final year examination, and each student has to pass before he/she is inaugurated as a medical doctor. Those who pass the IMDNCE will receive a competency certificate as one of the requirements to apply for the medical license to practice.

Standardization is crucial in a country such as Indonesia that has a significant number and yet very diverse medical schools. Currently, there are 83 medical schools in Indonesia which are varied in many aspects, including ownership (public vs private), new vs old, student body (small vs big), and accreditation level (A/B/C). Some medical schools are located in an area where there are several other medical schools that forces them to compete with each other in finding teaching hospitals, which in the end will influence their teaching and learning quality. Other important variations include the entrance selection process and number of qualified teaching staff and facilities. Although there is the written document of the Indonesia Medical Doctor Competency Standard that must be referred to for curriculum development by all medical schools, in practice, all those above variations will certainly affect the quality of the teaching learning process. These conditions could be a significant source of concern and worry for some of the stakeholders of medical graduates.

As a high stakes summative assessment, the IMDNCE demands application of quality standards for defending its validity and reliability. This requires both an internal and external quality assurance process.⁸ The internal quality assurance should be conducted by the exam provider to defend that the test instrument is representing what it is intended to measure. The provider should also guarantee that the examination process is also conducted in a proper way so that the decision made from the test result is defensible. The Panitia Nasional Uji Kompetensi Mahasiswa Program Profesi Dokter (PNUKMPPD), or the National Committee of Competency Examination, as the provider of IMDNCE has utilized a quality framework called We PASS with the score of A to satisfy this quality provision.⁹

Moreover, the external quality assurance conveys the understanding that the test and its provider are subject to undergo review process, feedback or appraisal from an independent third party and/or stakeholders regarding the quality definition employed. While the idea of accreditation of test providers is a popular topic in the education literature and conferences, the research of the stakeholders’ views on IMDNCE is yet to be evaluated. The evaluation from stakeholders will be representing the generalization and extrapolation of evidence produced in the Kane validity framework.¹⁰

There are 3 main stakeholders involved in IMDNCE: consumers (patients and consumers in watchdog organizations), doctors (individuals/candidates and collective/professional bodies), and employers and other organizations including but not limited to the government, universities, and insurance companies. Each stakeholder has their own expectations toward IMDNCE, with some expectations such as patient safety and quality care shared between all stakeholders while others can be varied.¹¹ There are 3 key elements that contribute to acceptability: 1) stakeholders’ wishes, 2) stakeholders’ beliefs and, 3) the ratio of reward and cost.¹¹

To date, there has not been any research in Indonesia that explores the stakeholders’ perceptions toward IMDNCE. Therefore, this study aims to explore how the stakeholders have perceived the impact of IMDNCE.

Materials and Methods

This research was a qualitative study using a phenomenological approach. The study was conducted from July to December 2017. The sampling technique was purposive sampling and the participants consisted of four groups. Group 1 consisted of representatives from consumer organizations, the National Health Coverage, the Ministry of Health, and the Indonesian Medical Association. Group 2 consisted of employers (hospital and health center directors). Group 3 consisted of supervisors of internship and supervisors of first year residents and group 4 consisted of patients.

In groups 2, 3, and 4, participants came from 6 regions based on geographical locations throughout Indonesia as depicted in Figure 1. In each region, researchers determined a major city as a center of data collection. In every major city, researchers identified health centers and hospitals that employ medical doctors who have passed the IMDNCE as interns. The identification was based on the Internship databases of the Ministry of Health. The directors of health centers and hospitals were invited to participate as group 2 participants. In addition, the researchers also contacted the medical faculties that have residency training programs in each area to send first year resident supervisors as group 2 participants. Meanwhile, the supervisors in those health centers and hospitals were invited to participate as group 3 participants. Participants of group 4 were patients served by the intern doctors at the chosen health centers and hospitals.

Figure 1 Geographic distribution of 6 regions for data collection of groups 2–4.

Data for groups 1–3 were obtained through focus group discussions (FGDs). The FGD guide focused on the exploration of stakeholders’ perceptions of: 1) perceived roles of the IMDNCE and, 2) confidence toward the performance of medical graduate who passed the IMDNCE. We used FGDs for groups 1–3 since FGD encourages discussion dynamic and enables us to obtain rich data. Table 1 shows the number of FGDs for each group. Data for group 4 were obtained through interviews to make the data collection process more feasible.

Table 1 Number of FGDs and Interviews

Figure 2 shows the data analysis process. Trustworthiness of this study has been developed and achieved through: triangulation of methods, information resources, frequent peer debriefing during data analysis, thick description of the phenomenon under scrutiny, and an audit trail toward the research process, data collection, and data analysis. Verbatims from the FGDs and interview recordings were coded. The codes were reviewed and sorted into categories. Lastly, the categories were analyzed and put into general themes.

Figure 2 The process of the qualitative study.

The research was a nation-wide project involving a huge research team from multiple health professions educational institutions across the country. The research was then divided into sub-topics. The ethical clearance were obtained from the Health Research Ethical Committee of RSUD Moewardi/Faculty of Medicine Universitas Sebelas Maret 1063/XII/HREC/2016. Thank you, The IRB is the institutional home base of one member of the larger research team. The particular member was then assigned in the other sub team with different sub topic under the same research project apart from this sub topic. The consent for participating in the study has been obtained from the participants of before each of the Focus Group Discussion and interviews by providing them with written information in regards to the study. The participants who gave consent to proceed with the data collection were required to sign the consent form. All of the participants who were involved in the study had provided informed consent to participate in the study and allow for their responses to be written anonymously in the report.

Results

The total number of FGD participants from the 6 regions was 73 people, while the total number of interview participants from the 4 regions was 17 people. Interview participants from regions 1 and 6 were excluded because the patients only provided yes or no answers without further explanation although interviewers did attempt to probe for more detailed responses. The distribution of participants is shown in Table 2.

Table 2 Distribution of Participants for FGDs and Interviews

There are 3 general themes that emerged from data analysis.

Theme 1. Indonesia Medical Doctor National Competency Examination (IMDNCE) as an effort to standardize doctor graduates in Indonesia

Competence examination as a method for national standardization of medical graduates

Most of participants in groups 1, 2 and 3 described the IMDNCE as an important step to standardize medical graduates. Some examples are as follows:

“Assessing standard competency of medical doctor, how far the competency has been achieved”, R1_Y_PK

“There are numerous medical schools, mushrooming everywhere. When the students graduate, the competency examination can be used as competency standards that will be given to each doctor after graduation. The targets in every university are different, but maybe with this standard, at least for general practitioners there are standards that should be achieved” N1_Ja_PK

“In my opinion, the competency examination is one way that can be used to standardize the process of medical education in Indonesia, it is expected, wherever the doctor has been educated, the outcome is almost the same.” B_S_Pr

Competency examination as a filter examination for certification

Participants in group 2 (hospital and health center directors) added that the Competency examination has a role in the National administrative requirement for the applying of the Registration Certificate, as mentioned below:

“Requirement for practice” N1_Ja-PK

“Those who cannot pass the examination, cannot work.” I_Me_Pr

Competency Examinations are also known as the assessment tools to define the quality of graduates from aspects of knowledge and skills, which are objective, as indicated below.

“Because we can see it nationally and to avoid subjectivity, so it is more objective. Because the examination is using computer, so there is no conflict of interest” M_Ja_Pr

Theme 2. The results of the national examination as means to reflect the quality of medical education in Indonesia

Many participants described how the quality of those who pass examination competency can be attributed to the quality of medical faculty and also the performance of each individual prospective physician. This feedback encourages the medical faculty to monitor and improve its quality ranging from the students’ admissions to time of graduation. For examples:

“As a controller of medical schools that produce doctors, with the competency examination, medical schools can evaluate and monitor their graduates, so that the graduates can be ready to plunge into society. R_M_P

“Competency examination is important. [We] need to maintain [the medical schools] with improved quality. [The exam results] can be [used] as a reflection for medical schools on selecting their prospectus students.” D_S_Pr

“Now with a standardized competency examination, between private and state medical schools, we can compare their quality, one of which is through this examination. We do not need to physically assess their building; by looking at this competency examination results, one can compare [the quality of graduates].” S3_Ja_ST

Participants also mentioned that the IMDNCE is an attempt to standardize medical education and minimize the disparity of the quality of medical education in Indonesia, and to measure the expected competencies. Two participants underscored the importance, as follows:

“I think that competency examination [is] a parameter so that for universities in Java i.e the most populated and developed island in Indonesia], the quality is not very different with medical schools outside Java. At least, the quality of graduates in Java and outside Java is closer”. SS_B_PK

“To standardize [the quality of medical education]. Although, it depends more on individual factors than the attributes of medical schools.” AY_Me_Pr

Many participants from the employer and stakeholder groups also added that those who passed the IMDNCE were expected to able to compete with overseas graduates, i.e. due to ASEAN (Association of South East Asian Nations) free trade. They also mentioned that the quality of the education process in medical schools was increasing so that the graduates were expected to be ready to compete with doctors from other countries. Some examples are as follows:

“We are optimistic. The preparation to ASEAN free trade is a longtime effort, [and it is] different with the students’ preparation for OSCE (relatively short time). From the beginning, the lecture [for example], can be adjusted, at minimum [with both] English. Other international languages [can] also be taught. More prepared the education process, the better results we can expect.” R4_Y_PK

“Indonesian students have a similar chance with Australian and Singapore students hence we must be optimistic that we can compete with them. If the education process is better, more standardized, we do not need to worry with how the quality of private medical schools or eastern Indonesia medical schools are. Earlier, simulation learning is still limited, but now it is better.” S3_Ja_ST

Most participants from the employer group articulated that the IMDNCE needs to be developed to improve the quality of graduates so that they are recognized at the international level. One participant highlighted the importance as follows:

In my opinion, IMDNCE is the only standard of the eligibility of medical doctor graduates in Indonesia. For competition at the ASEAN level, it needs a specific parameter for medical doctor graduates and an ASEAN level competency exam. Our students need to take the ASEAN competency exam so that it can be determined whether the quality of our graduates is equal or not”. SS_B_PK

Theme 3. Competency examination as an effort to improve health services in Indonesia through the quality of graduates

In this theme, the quality of health services was defined based on the data collection techniques as follows: (a) FGDs from groups 1, 2 and 3 about the quality of graduate doctors in communication skills, clinical reasoning, management and professionalism including patient safety; and (b) Patients’ interviews about graduate doctors in communication skills and professionalism and ethical issues.

Graduates’ Quality in Communication Skills

Effective communication is one of the 7 competency areas of doctors based on the national standard of competence for medical doctor in Indonesia or Standar Kompetensi Dokter Indonesia (SKSI). Therefore, communication skills become one of the focuses in the exploration and analysis process. Most participants from the employer and clinical supervisor groups mentioned that communication skills of the graduates’ doctor have a variety of results; some are good enough and some are still needing to improve. They also noticed that the communication skills of graduates’ doctor will develop during their work experience. Some examples are as follows:

“Now, the medical students are trained in skills laboratory to analyze patient, [and] introducing themselves. Therefore, their communication skills, I think, is better.” I_Me_PK

“Maybe our weakness is in communication. How we educate, and empathize to the patient is needed” Z_S_PK

From the patients’ points of view, we also found various levels of communication skills: some graduates showed good communications skills, and some others did not. For these skills among the medical graduates, some of the patients perceived them as follows:

“The language ‘sounds normal’, easy to understand.” (B-PC-NH40, B-PC-NH42)

“I think the information is clear enough, [the doctor] communicates patiently and seems that he is not in a rush.” (J-NH)

“Well, the doctor doesn’t clearly explain the reason why I should have the examination, he only tells me to lay down on the bed.” (B-PC-NH38, B-PC-NH39)

Graduates’ Quality of Clinical Skills

The participants of groups 1, 2 and 3 expressed perceptions about graduates’ quality of clinical skills which are varied. Some participants mentioned about the lack of clinical skills and others have different points of view, as shown in the following:

“From all the competence aspects, I saw the lack of clinical skills performed” PI-J

“They already have a good skill learning in their education program, right? I think they are ready enough to perform in hospital. In the beginning, they seem not confident, I think it is okay.” ST-J

Graduates’ Quality of Clinical Reasoning

Most participants in the employer and clinical supervisor groups perceived that the quality of clinical reasoning and diagnostic findings of graduates were initially poor, but the performance increased with experience and exposures to patients and clinical environment, as follows:

“Sometimes they make incorrect diagnosis.” PK-KR-Y3

“Sometimes, clinical supervisors complain about their diagnostic abilities” PK-KR-Y5

Graduates’ Quality of Patient Management

According to participants’ perception especially in the employer and clinical supervisor groups, the quality of patient management has to improve more, as indicated by the following:

“Because there are a few medicines that seem unfamiliar for them, the incorrect management or wrong prescribing can happen” PK-K-RS1

“They make some mistakes in prescribing drugs.” PK-K-PS2

“I think they are good enough in applying their knowledge, even though there are 1-2 graduates who have some problems during prescribing” PK-KR-Y3

As an effort to improve health services in Indonesia, participants perceived that the competency examination has an important role to prepare graduates to improve patient safety, as indicated by the following:

“I see that this competency examination is a state effort to protect the public by producing competent doctors” S1_Ja_ST

The role of the competency examination encourages the aspect of professionalism and a strong sense of humanity, by treating patients as “human” and not objects.

“The impact is very much. Therefore, by passing the competency examination, they are at least competent in applying ethics. In addition, when they treat their patients, they are more competent.” N2_Ja_Pr

From the patients’ perceptions, we also asked about the professionalism and ethical issues such as how they feel about graduate doctors’ responsibility, empathy and other ethical issues, as shown below:

I think the doctor is responsible enough. We were satisfied; when I and my son came [to the hospital], he examined [my son], and then the laboratory [examinations] were done and we got the medicine.

[The doctor] is a good listener. She gave eye contacts when she was telling me how to exercise my back.

I think the doctor was careful to examine [me]

Participants mentioned that passing the competency examination is not the only basis to trust graduates’ skills. Employers and clinical supervisors seemed to agree that a continuing competence development through graduates’ work experience also plays an important role in their trust, as indicated by the following:

“For me, if they pass the competency examination, I believe in their diagnostic skills. Nevertheless, honestly, if they were left to work independently, they are not yet ready. As dr R said at the beginning, experiences are important.” R3_Y_PK

“May be similar with that doctor’s opinion. … to be able to assess whether their performance relates with their competency examination, perhaps, it is not correlated directly, because all of this is a process. Including when they were learning from lectures, clinical practice. It is a unity. So, if it was said, it is the impact of competency examination, … no it can’t, as it is a unity.” F_S_Pr

Discussion

In general, the IMDNCE has been able to meet the expectations and beliefs of all stakeholders involved in the study. It has been perceived that the IMGNCE is giving reassurance to the employers and medical professionals regarding the medical graduate’s competency. It is also ensuring to the patients that the graduates are able to provide quality care. This finding is commensurate with research conducted in other countries that use an MLE. Performance in licensing examinations, e.g. in the United States Medical Licensing Examination (USMLE), has been considered in the doctors’ career,^12,13 ensuring greater patient safety and improved quality of care.^14–16 Despite the limitation of such studies and given challenges in directly linking the results of those national licensing examinations with the patient outcomes, the correlation analyses have highlighted the value of competency achievement.¹⁷ In our setting, the IMDNCE has been placed as an exit exam from a medical school, not a licensing examination per se, in order to assure the alignment of curricula in our medical schools and to increase accountability of each medical school in preparing the future graduates who are fit for practice.

Furthermore, in our study, medical professionals feel that the IMDNCE also serves as a feedback mechanism for medical schools to monitor and improve their quality in providing education. This feedback is relevant with one of our findings that sees the IMDNCE as an important standardization tool for medical school. There are various measures to decrease disparities of quality among medical schools, including national student selections, accreditation, and national examination. This research shows that stakeholders perceive the IMDNCE to have a critical role in narrowing the gap of quality standards among the large number of medical schools. It is under the assumption that medical graduates who pass the IMDNCE are considered to achieve the required competency regardless of his/her educational background. The stakeholders perceived that despite meeting the minimal standard for medical doctors, those who passed the IMDNCE also are reflecting the educational process in their medical school. This finding underlines the basic conception of the assessment driven learning paradigm which argues that exam results are by their very nature a measure of the quality of teaching and a reflection of the staff in the school.^18–21 This finding is similar with the results of several studies that compared and analyzed the different results of USMLE between US medical graduates and international medical graduates.^22–24 These differences are claimed to be associated with different accreditation and educational systems between countries, such as differences in curricula, learning experiences, standard of care, and culture. It is suggested that medical schools should develop their educational programs to improve students’ performance in national competency examinations.

Another important role of the IMDNCE is to protect and prepare medical graduates from the influx of doctors from other countries. This role is similar with the function of the USMLE in filtering medical graduates, both with national and international background, to provide standardized patient care in the US.^23,24 In recent years, there is increasing awareness among countries to facilitate free movement of doctors across borders.^25,26 This is also noticeable in the ASEAN region, but the mobility of doctors among ASEAN countries is still hindered by several problems, such as a lack of standardized recognition of qualification, medical licensing examination methods, language barriers, and unified medical curricula.²⁷ Our study shows that stakeholders have similar expectations for the role of the IMDNCE in preparing medical graduates to compete in regional and international levels.

This study also shows that stakeholders’ confidence toward the performance of medical graduate are varied. Non-patient Groups (Groups 1–3) agreed that the performance of graduate tends to be influenced by his/her “on the job” experiences. Patients’ perceptions toward the medical graduates were mostly positive, even though some have a negative perception especially in the professionalism aspect. This view of graduate doctors recalls the existing discourse on physician professionalism where the general public might have different appraisals on physician’s performance depending on the various experience that they received.^28–30 In-depth exploration showed that communication skills tended to be the most favorable aspect, while clinical reasoning and prescribing ability tended to be the least favorable aspect in medical graduates’ performance. This finding is similar with a previous study³¹ which also found that the prescribing and case management competences have become problematic during the transition period from learning in the classroom to real clinical work experience. Curriculum development in medical schools across the country is recommended to close the gap and improve stakeholders’ satisfaction.

The specific expectations of stakeholders in this study influenced their confidence toward medical graduates’ performance. In turn, this confidence will influence their acceptance toward the IMDNCE as the standardized national assessment. The acceptance of performance by stakeholders is an evolutionary process,²³ thus the gap between expectation and reality noticed by stakeholders is part of the evolution process of acceptance. Furthermore, Indonesia has a large power distance for their cultural approach that leads to low individualism among students of medical education.³² This cultural aspect especially the prevalent paternalistic view about young doctors’ performance can possibly become the main influencing factor of stakeholders’ confidence toward medical graduates’ particularly among employers and clinical supervisors who are mostly senior doctors. One study about perceptions of stakeholders in different contexts shows that the USMLE Step 1 and 2 scores are regarded as highly important criteria in selecting medical students for postgraduate training.¹²

Triangulation regarding sampling, participants and time commitments have been done. In addition, variations in the origin of the subject areas have also been considered. However, in the actual implementation, some prospective respondents could not attend all FGDs for various reasons. This limitation to participant contribution allows for reduced opportunities to obtain new information from prospective respondents who were not present. This study also does not explore the perceptions of the individual candidates toward the IMDCNE. Further studies are also encouraged to investigate the long-term impact of the IMDNCE towards medical education development in Indonesia in regards to educational management, curriculum development as well as learning resources. It is also recommended to include candidates who had recently passed or failed the examination, or medical students who are about to take the examination.

Conclusion

In general, communities of stakeholders, employers, professionals, and patients see the competency examination as able to standardize medical graduates. The competency examination is also seen as an effort to improve health services. In addition, the competency examination is a reflection of the quality of medical education in Indonesia. The competency examination needs to be further developed to improve the competence of graduates to be recognized at the international level, in order to prepare physician graduates to face competition with graduates of other countries and to engage in cross-cultural sharing about medical best practices.