A Pilot Study Assessment of Medical Student Knowledge and System Citizenship Attitudes Pertaining to Health Systems Science

Introduction

Health systems science (HSS) has been described as the third pillar of medical education, and as being complementary to the traditional disciplines of basic and clinical science.¹ Introducing HSS content early in the medical school curriculum is thought to contribute to learners’ ability to work in teams, enhance understanding of the larger health care delivery system, and add value to care during their educational experiences in the clinical setting.² Further, while interprofessional education and collaborative, team-oriented clinical practice are topics that pre-date the current emphasis on HSS, these topics are highly synergistic with this emphasis. While HSS has been described as a broader construct that includes interprofessional education (IPE),³ this in no way minimizes the importance of IPE; indeed, it has been stated that the types of changes needed to both health professions education and health care systems “rely on interprofessional approaches to optimize education and value”.⁴

Additionally, HSS is described as especially relevant to the physician training process and to the critical leadership roles that physicians occupy in the delivery of health care. It is suggested that, due to the rapid pace of change in the present health care system and the need for greater economic efficiencies in the system, physicians have a duty to become system citizens who not only practice clinically but also “view themselves as stewards of the system…. [who] contribute to continuous evolution of the health care system itself to help achieve optimal results for patients and populations”.⁵ The HSS literature contains many references to an expanded role for physicians, who must become practitioners of systems thinking and help lead a transformation of our health care systems from the sole provision of clinical care to entities that apply “scientific advances…. with the goal of improving the health outcomes of the population”.⁶ A foundational article pertaining to HSS in medicine defines systems thinking as a construct that involves “an awareness of the ‘whole’, not just of the parts; and the ability to recognize multidirectional cause-effect relationships with all causes emerging as the effect of another system dynamic”.⁷

Parallel to the call for physicians to play expanded roles in health care systems are also calls for “a re-evaluation of health education, which may require changes in curricula and additional competencies…. [including] new curricula that encourage team science and interprofessional education”.⁶ Through its Accelerating Change in Medical Education consortium project, the American Medical Association has served as a catalyst for curricular revision processes that emphasize the inclusion of HSS in the medical school curriculum.⁸ The construct of HSS is also consistent with graduate medical education, as one of the six general competencies of the physician, systems-based practice, has been promulgated by the accrediting body for physician residency and fellowship training since 1998.⁹ A schematic crosswalk showing alignment between HSS and other ACGME competencies (professionalism, interpersonal and communication skills, practice-based learning and improvement) has also been recently published.¹⁰ However, HSS concepts and systems-based practice competencies have not been completely congruent and have not been adequately translated into workplace practice patterns for resident physicians.¹¹ Finally, it is recognized that an expanded focus on HSS must include professional development and ongoing training of practicing clinicians from a variety of disciplines.¹² Thus, the emphasis on HSS is one that applies across the continuum of physician training, from initial training in medical school through residency and/or fellowship training and into continuing professional development/education.

As part of a comprehensive curriculum renewal project described elsewhere,¹³ our school has embarked upon the development of a new, four-year HSS curriculum for medical students, including the incorporation of new content into an existing interprofessional education curriculum involving first-year medical, nursing and physician assistant students. In partnership with another health professions college wherein we have established a strong foundational curriculum in interprofessional education,¹⁴ we worked with faculty colleagues to introduce this new content (labeled as health systems science and interprofessional practice, or HSSIP) beginning with the fall 2020 semester. The new curriculum will eventually encompass all 4 years of the medical school experience. The initial roll out for year 1 provided HSS content in a variety of formats including pre-assigned readings, lectures, and small group learning sessions focused on clinical scenarios and other topics. Each learning session was built upon specific objectives related to some aspect of HSS. The curriculum for year 1 was delivered in four different instructional blocks, and a syllabus for each block specified the different HSS topics covered. Topics included in these four blocks were as follows: overview of health system science concepts, health care leadership, roles and scopes of professional practice, teamwork in health care, organizational and clinical ethics, health care disparities, population health, comparative health systems and US health care policy. The course typically involves from 100 to 130 students per year (variable depending on enrollment in the nursing program), and approximately 35 different faculty members from a variety of disciplines and backgrounds.

We recognized early in our curriculum reform and evaluation process that the issue of assessment of medical students’ knowledge of HSS and attitudes toward system citizenship would be of critical importance. Nearly simultaneous with our curriculum reform efforts focused on HSS, the National Board of Medical Examiners (NBME) introduced a new subject exam on HSS in 2019. The development of the exam was a collaborative effort between the NBME, and the American Medical Association consortium of medical schools interested in HSS and related educational reform.¹⁵ To assess learner knowledge of HSS, we chose to use the new NBME HSS subject exam as part of our pilot study protocol. Additionally, we wanted to gain insight into how medical students would view the idea that physicians should become system citizens. However, after conducting a literature review, we could find no existing attitudinal survey related to the construct of HSS system citizenship, likely because this construct was so new. Thus, we developed a survey that focused on what we felt were the key components of system citizenship.

Our study hypotheses were that medical students who underwent the new HSS curriculum in year 1 would 1) achieve higher scores on the NBME subject exam and 2) would exhibit more positive attitudes toward system citizenship when compared to our medical students who did not experience the new curriculum.

Materials and Methods

To test these hypotheses, we implemented an IRB-approved pilot study research protocol (Virginia Tech IRB #20-069, judged exempt from further review) over a 2-year period. The overall purpose of our study was twofold: to gather program evaluation data that would assist curricular leaders in ongoing formative evaluation of the new HSSIP curriculum; and to measure baseline and ongoing medical student knowledge and system citizenship attitudes pertaining to HSS.

We identified a total of four student groups over two academic years (2019–2020 and 2020–2021) for participation in the study as follows:

Study year 1: M4/fourth-year students, no participation in the new HSS curriculum, N=42.

M1/first-year students, no participation in the new HSS curriculum, N=44.

Study year 2: M4/fourth-year students, no participation in the new HSS curriculum, N=40.

M1/first-year students, participated in the new HSS curriculum, N=48.

For two consecutive academic years, all fourth- and first-year students completed the HSS subject exam and system citizenship surveys toward the end of their respective academic years. Exams were administered to first-year students at the end of a designated exam period, and to fourth-year students during a “transition to residency” curriculum block. Approximately 2 weeks ahead of time, students in all groups received email messages which explained the purpose of the study. The email messages informed the students that the HSS exam would not be part of any official course grade and that advanced study or preparation for the exam was not necessary. They were subsequently provided the identical information again, along with the informed consent document, immediately preceding the administration of the exam and survey. The consent document informed the students that their completion of the HSS exam and system citizenship survey was required for the evaluation of the new HSS curriculum, and asked students to “opt out” if they did not wish to include their data in the research study.

The NBME subject exam consisted of approximately 100 multiple-choice items in a single-best-answer format. The exam blueprint and scoring categories were reviewed on the NBME website,¹⁶ with items being characterized in two major groups: HSS core domains (six categories) and HSS cross-cutting domains (five categories). Although the NBME Health Systems Science examination is designed to be broadly appropriate as part of overall examinee assessment, we recognized that course objectives as reflected in our new HSS curriculum were not perfectly congruent with examination content; thus, performance scores were not used in determining grades. However, given that national faculty with expertise in health systems science wrote and reviewed the items, we chose to use this exam during our study to provide us with an early measure of students’ understanding of health systems science. Moreover, the subject exam provides performance feedback including score reports, content area summaries, student performance profiles, and year-end reports. Scores are statistically equated across exam administrations, which ensures that all scores have the same meaning and are comparable over time on different forms of the same exam. For our analyses, we used equated percent correct scores to reflect an examinee’s mastery of the content domain, calculated as a percentage of items in the total content domain that were answered correctly based on an examinee’s proficiency level.

The original version of the system citizenship survey consisted of 12 items (see Table 1) and was constructed with the intent of assessing student attitudes toward various aspects of HSS. Items were selected based upon a review of several existing resources, as well as a literature review which sought to identify how the construct of system citizenship had been previously defined and/or measured in the context of health professions education. As part of our review, we examined a recently published grid of HSS competencies¹⁷; the HSS learning modules offered by the American Medical Association¹⁸; a published set of milestones for the science of health care delivery at Mayo Medical School¹⁹; Transitional Year Milestones (2019) as promulgated by the Accreditation Council on Graduate Medical Education²⁰; the Entrustable Professional Activities (EPA) list of competencies and other information related to EPA number nine (Collaborate as a Member of an Interprofessional Team) and EPA number 13 (Identify System Failures and Contribute to a Culture of Safety and Improvement), as promulgated by the Association of American Medical Colleges²¹; and the existing M3 clerkship interprofessional education objectives from our medical school curriculum. Our literature review identified two key articles that informed the development of our survey: a survey of nursing faculty on efficacy for teaching systems thinking to nursing students²²; and a validation study of a questionnaire designed to measure students’ learning of systems thinking as part of an undergraduate systems engineering program.²³

Table 1 Health System Citizenship Scale Virginia Tech Carilion School of Medicine

Based on our review of these resources, we constructed the original 12-item version of the system citizenship survey. Students were asked to “please indicate the extent to which you agree with each of the following items” using a Likert rating scale (1=strongly disagree to 5=strongly agree). All students in both study years completed this version of the survey.

The original 12-item attitudinal survey was altered for the second year of the study. Three of the original 12 items were re-worded and the survey was expanded to include four additional items, bringing the total number of items to 16 (see Table 1). The updated version of the survey was approved as a study modification via the IRB review process. The modification of the original attitudinal survey meant that we could compare only nine of the items across both study years. Students in study year 2 were asked to complete both the revised attitudinal version and the two 16-item versions of the survey, which addressed the extent to which students felt that they 1) had experienced HSS concepts as part of their medical education and 2) felt prepared for various health system components. This modification of the original survey was undertaken to gather more specific information from students as an additional curricular evaluation measure. It was felt that the expanded versions of the survey would provide additional insight into the effectiveness of the new HSSIP curriculum. The same Likert rating scale was used for all versions of the survey as previously indicated above.

We compared HSS exam scores and survey results between groups of students as shown in Figure 1.

Figure 1 Comparisons of HSS exam scores and survey results by student group. *Students had not experienced new HSSIP curriculum. +Students had experienced new HSSIP curriculum.

Group mean scores on the HSS exams were compared using two sampled t-tests with a significance level of alpha=0.05. Survey item means were also compared using the same statistical procedure. Calculated effect sizes are reported based on Cohen’s d statistic²⁴; for survey items, effect sizes are reported only for statistically significantly different items.

Results

For study year 1, 24 of 42 (57%) fourth-year students and 42 of 44 (95%) first-year students provided informed consent for their data to be included in the study. For study year 2, 32 of 40 (80%) fourth-year students and 28 of 48 (58%) first-year students provided informed consent. Thus, a grand total of 56 study eligible fourth-year students (68%) and 70 (76%) study eligible first-year students participated in the study. No students in year 1 of the study had received any of the new HSS curricular content. For students in year 2 of the study, only the first-year students received new HSS curricular content.

HSS Subject Exams

We first compared exam scores for students in year 1 of the study. Fourth-year students who had not experienced the new curriculum (N=24, mean=69.5, standard dev=6.10) scored statistically significantly higher than first-year students who had not experienced the new curriculum (N=42, mean=65.2, standard dev=7.35); p=0.02. Cohen’s effect size value (d=0.61) suggested a moderate practical significance.

We next compared exam scores for first-year students from both years of the study. First-year students from study year 1 who had not received the new curriculum scored higher on the exam (N=42, mean=65.2, standard dev=7.35) than first-year students who had experienced the new curriculum (N=28, mean=55.5, standard dev=13.76); p=0.003. Cohen’s effect size value (d=0.79) suggested a moderate to large practical significance.

Finally, we compared exam scores for students from year 2 of the study. Fourth-year students who had not experienced the new curriculum (N=30, mean=68.9, standard dev=12.10) scored statistically significantly higher than first-year students who had experienced the new curriculum (N=28, mean=55.5, standard dev=13.76); p=0.0002. Cohen’s effect size value (d=0.85) suggested a large practical significance.

Reliability data for the HSS subject exam was not available at the time of our study, so we cannot report on the psychometric properties of the exam as reflected in our student population. In the only study we found related to use of the new HSS subject exam, an alpha reliability coefficient of 0.83 was reported based on a national sample of medical students from 15 schools who took the exam.¹⁵

System Citizenship Surveys

Scale internal consistency was estimated by pooling observations and calculating Cronbach’s alpha with this sample and based on the intended use of the scale.²⁵ We completed reliability analyses for the two different student samples based on study year. Reliability coefficients were strong, with the 12-item version (α = 0.83, year 1; α = 0.98, year 2) having a slightly lower reliability score than the subsequent 16-item versions (α = 0.98, experience items; α = 0.99, preparation items).

We compared system citizenship survey scores between the following groups:

Fourth-year versus first-year students from study year 1 using the original, 12-item version of the scale (see Table 2). None of these students had experienced the new HSS curriculum. Two items were statistically significantly different, with first-year students rating both items higher: “healthcare is more than a function of the individual physician–patient relationship” and “physicians should sometimes lead a clinical team but at other times should defer to a non-physician team leader”.

Table 2 Comparison of Attitudes Toward System Citizenship Fourth-Year versus First-Year Medical Students (Study Year 1, No HSS Curriculum Experience)

Fourth-year versus first-year students from study year 2, using the 16-item version of the scale which asked students about their perceived experience with health systems science (see Table 3). Two items were statistically significantly different, with fourth-year students rating both items higher: “health system science impacts on the delivery of care to patients” and “physicians sometimes leading a clinical team but at other times deferring to another health care team leader”.

Table 3 Comparison of Student Perception of Experience with Health Systems Science (Study Year 2, Fourth-Year Students/No HSS Curriculum versus First-Year Students/Had New HSS Curriculum)

Fourth-year versus first-year students from study year 2, using the 16-item version of the scale which asked students about their perceived preparation for health systems science (see Table 4). Five items were statistically significantly different, with fourth-year students rating all five items higher: “leading a clinical team and at times deferring to another health care team leader”, “discussing medical errors/unexpected outcomes openly to facilitate learning and patient understanding”, “using multiple viewpoints to understand/analyze a problem encountered in the clinical setting”, “demonstrating a duty to ensure that economically disadvantaged people have an equal opportunity to be healthy”, and “measuring and monitoring clinical performance related to health equity”.

Table 4 Comparison of Student Perception of Preparation for Health Systems Science (Study Year 2, Fourth-Year Students/No HSS Curriculum versus First-Year Students/Had New HSS Curriculum)

First-year students from both years of the study, using only the nine items from the original 12-item version of the scale that were worded exactly the same on the scale during both years (see Table 5). Six of nine items were statistically significantly different, with students who had not experienced the new HSS curriculum rating all six items higher: “an understanding of health systems science impacts how I view the delivery of care to patients”, “health care is more than a function of the individual physician–patient relationship”, “the design of a health system has a strong influence on patient outcomes”, “physicians should serve in leadership roles in systems thinking and patient safety”, “physicians should identify and address systems level factors that contribute to ethical dilemmas”, and “physicians should discuss medical errors/unexpected patient outcomes openly to facilitate learning and patient understanding”. It should be noted that both groups of first-year students rated all scale items very highly (from 4.00 to 4.74 on the five-point scale).

Table 5 Comparison of Attitudes Toward System Citizenship Both Study Years, First-Year Medical Students Who Did Not/Did Experience New HSS Curriculum

Discussion

Health systems science has been identified as a new emphasis within health professions training, especially for medical students.²⁶ Students undergoing a new HSS curriculum at one medical school reported positive impacts on both their knowledge of health systems and their perception of physicians playing a vital role in leading and improving systems of care.²⁷ Their experiences are consistent with an ever-growing cascade of calls for physicians to “be better prepared to work in, and contribute to the continual improvement of, health care systems”.²⁸ However, some medical students have reported that the instructional time devoted to systems topics was inadequate and that preparation for practice in the changing environment of the US health care system should receive greater emphasis.²⁹

Our study examined two important issues: medical student knowledge about HSS, as measured by a new NBME subject exam; and student self-reported attitudes toward, experiences with and preparation for health systems science as a new curricular emphasis. We conducted our study during the very early stages of implementing a new HSS curriculum, which was designed to be gradually rolled out over a 4-year period beginning with the 2020–2021 academic year. Our study revealed findings that were both expected and surprising. Such results often occur when attempting to make changes to the medical school curriculum, due to factors related to curricular reform and how best to introduce a process of change.^26,30

Regarding the rate of student participation, across both study years a slightly larger percentage of our M1 students than our M4 students agreed to have their exam and survey data included in our study. However, the differences in participation between the two study years was striking, with a higher percentage of M1 students agreeing to participate during study year 1, but a higher percentage of M4 students agreeing to participate in study year 2. We especially note the fact that fewer M1 students from study year 2, ie, those who had experienced the new HSS curriculum, agreed to participate. This lower participation by the only group of students to experience the new HSS curriculum could have impacted our findings, due to the small sample sizes of the student groups. Although small sample sizes are common in pilot studies, the different levels of study participation by our student classes presented challenges in interpretation of results. Effect sizes pertaining to exam data were moderate to large, and were moderate for survey items that were statistically significantly different. We suggest that, notwithstanding the small size of our study populations, the moderate to large effect sizes indicate a genuine difference in exam performance and survey items between student groups.

One of our findings pertaining to the use of the new HSS subject exam was not unexpected: that M4 students during both study years outperformed M1 students from both study years. Regardless of whether formal and deliberate exposure to HSS content occurred, it seems intuitive that M4 students would have gained more knowledge of at least some aspects of health systems than M1 students by virtue of their clinical experiences. We are also pleased that our M4 students appear to have subject exam scores that were aligned with those of a national sample of students who also took this new exam.¹⁵

Two findings pertaining to the HSS subject exam were surprising: that M1 students during study year 1 scored nearly as high on the exam as did M4 students, with neither group having experienced the new curriculum; and that M1 students from year study year 1 (who had not had any formal HSS content) outperformed M1 students from study year 2 (ie, those who had experienced the first year of the new HSS curriculum). Regarding the first finding, we are unsure as to why our M1 students scored nearly as high as our M4 students on the exam during study year 1. It is possible that class characteristics could account for the higher score by the M1 students in study year 1. For example, if a higher than usual number of students in this M1 class had prior experience as clinical scribes, emergency medical technicians or similar types of involvement in health care settings prior to medical school, these factors could account for the score difference in that those students had greater insights into systems of care. This possibility is consistent with a qualitative study wherein “influential experiences before medical school” were found to have an impact on the aspirations of medical students related to future involvement in efforts to impact health systems.³¹ Whether these experiences prior to medical school could also impact the scores of a given group of medical students on the HSS subject exam is an intriguing question. Although our study protocol did not allow us to examine these class characteristics, this would seem to be an important consideration for future studies. Regarding the second finding, both the afore-mentioned class characteristics as well as the difference in sample size between the two groups of first-year students who gave consent to include their data in the study (ie, 42 in study year 1 versus only 28 in study year 2) could have impacted the exam scores, since the number of students who took the exam after experiencing the new HSS curriculum and gave informed consent for study participation was approximately one-third less than the comparison group of first-year students. Taken together, the statistically significant difference in mean scores and the moderate to large effect size would seem to indicate that the magnitude of the difference in performance on the exam was genuine and not due to chance. Another possibility is that, despite our instructions to the students that no preparation for the exam was necessary since their scores would not be part of any course grade, first-year students during study year 1 could have nevertheless prepared for the exam in some fashion, with such preparation resulting in higher scores.

Our students’ scores on the HSS exam are consistent with the previously mentioned larger study pertaining to performance on this new subject exam.¹⁵ In that study, which involved a national sample of 1837 students from fifteen medical schools, the average score on the HSS subject exam was 67%; this is consistent with the scores attained by three of the four student groups in our study. However, the study authors noted that “students taking the exam after instruction scored significantly higher than students who took it before or during instruction”, and that M3 and M4 students scored higher than M1 and M2 students. This finding was dissimilar to the results of our study in that one of our M1 student groups scored nearly as well on the exam as both groups of M4 students. We were surprised at the mean score achieved by our M1 students from study year 1 and speculate that this finding could possibly be an outlier due to the factors indicated above or other unknown reasons. Another possibility, however, is that the mean score achieved by M1 students from study year 1 could indicate an issue pertaining to exam validity and reliability. If, for example, these results were felt to be generalizable to all first-time takers of the HSS exam, it could indicate a need for further scrutiny of the exam itself. Another related but important point is that the HSS subject exam was designed to be taken by M4 medical students. While our study design did not allow us to examine generalizability or item-level differences in performance that could have contributed to the higher overall scores of M1 students from study year 1, we nevertheless suggest that, as further refinement of the HSS exam occurs, it is important to consider findings such as ours as part of an overall quality assurance strategy for exam development.

Regarding the use of our new system citizenship survey, we were pleased with the reliability of the instrument across study years and the alternative versions of the instrument. There was a general trend observed wherein the ratings of most items on the 12-item general attitudes toward system citizenship scale were high. Although we noted statistically significant differences on some survey items pertaining to students’ general attitudes toward HSS, only a single-item mean was below the rating of 4 on our 5-point rating scale; we found highly similar response patterns between the two groups of M1 students, with both groups assigning high ratings to the survey items (albeit with some statistically different item means noted). These results raise the possibility of response bias: did our students answer the survey items as they thought they should? At least one study had similar findings, with the authors noting “a ceiling effect with very high levels of interest in HSS among MS1 students that may have been due to social acquiescence bias”.³² When we modified the scale in study year 2 to ask more specifically about students’ experience with and preparation for HSS content, greater differences were noted between responses from M4 versus M1 student groups. This latter finding seems to more likely reflect genuine differences based on the year of medical school and greater exposure by M4 students to a variety of health system/clinical care settings.

One survey item of particular interest asked students to agree or disagree with the idea that “physicians should sometimes lead a clinical team but at other times should defer to another health care professional team leader”. Student opinion on this item fluctuated depending on the context in which it was raised. For students in study year 1, where neither group had experienced the new HSS curriculum, M1 students were more likely to agree with this statement than M4 students. For study year 2; however, M4 students were more likely to report having experienced this situation and being prepared for it. Our findings related to this survey item are interesting and worthy of further exploration in future studies; and perhaps greater insight into this area could also be achieved through more qualitative research techniques. Indeed, one recent study used qualitative techniques to examine how medical students “envision their future professional identity in relation to the system” and found that there are a variety of factors that contribute to student views on system citizenship, including the fact that some students were “more likely to view systems expertise as distributed within interprofessional teams”.³¹ This study finding appears to be congruent with the fact that M4 students in our study were more likely to agree that other health care professionals can sometimes lead a clinical team.

The construct of system citizenship is not yet well-defined and is therefore challenging to assess. Do we know precisely what systems citizenship really means in terms of medical students, resident physicians and/or other clinicians involved in the delivery of health care services? Is the key defining characteristic the ability to engage in systems thinking as applied to health care? If so, how is systems thinking defined within a health care context? Or is system citizenship something broader than systems thinking? Four medical students who engaged in an HSS curriculum at one medical school reported that their perspectives on what it means to be a physician were broadened to include the belief “that improving the well-being of our health care system is our duty as future care providers”.²⁷ An important conceptual linkage in this regard is that of professional identity formation (PIF), described as “the process through which a student transforms into a physician”.³³ While a complete discussion of PIF is beyond the scope of this paper, it is becoming increasingly apparent that the process of becoming a physician must include ensuring that future physicians possess the characteristics of health system citizenship.^5,34 A qualitative study of second- and fourth-year medical students at four different schools revealed that past, present and future-oriented experiences and considerations shaped how the students viewed their own future identities as clinicians engaged with the health system, with the added complexities of social context, affinities, abilities and practice context.³¹ Related aspects of the goal to produce system-ready clinicians include the need for medical education to reinforce the social contract between the medical profession and society at large, the ongoing examination of clinical care delivery models and their impact on various constituent groups, and the concept of communities of practice and interprofessional teamwork in the workplace.^4,35 Perhaps, our pilot study using a newly developed survey instrument designed to gain insight into students’ views on health system citizenship will make a contribution to ongoing dialogue on this topic within the medical education community.

Limitations

Our study has limitations that should be highlighted. First, the small sample size of our research subjects, as reflected by the lack of full “opt in” particularly for M1 students in study year 2, make generalizability of our results difficult. Although the sample size was low, the samples were representative of our classes. Second, our protocol did not allow us to investigate class characteristics and their possible impact on our findings. Third, because the HSS subject exam is still new and evolving we did not have access to reliability data for the exam itself and the reporting of results is limited in this regard. Fourth, it is likely that there is incongruence between our new HSS curriculum and the new HSS subject exam. The dimensions of the new HSS subject exam were determined by a national cohort of experts. Although the exam may be wider in content scope and therefore not correspond perfectly with our HSS curricular learning objectives, it is a measure that adequately samples from the universe of HSS content and thus still provides an estimate of our students’ ability from that domain of knowledge.

Future Studies

We will perform the same measurements over the next 4 years, comparing these early results with that of future classes that receive the new curriculum across all 4 years of medical school. We also plan to examine whether positive attitudes toward, and experience with, HSS by medical students correlate with performance on the HSS subject exam. Further validation studies are also needed regarding the system citizenship survey, and we would welcome collaborators at other medical schools for this purpose. Our ongoing research will provide greater insights into strengths and improvement areas for our new HSS curriculum as it continues to be implemented.

Conclusions

Our pilot study revealed differences between fourth- and first-year medical students concerning performance on a new HSS subject exam and regarding attitudes toward health system citizenship. It is hoped that our ongoing research will contribute to increased proficiency among medical educators regarding how to measure the impact of newly developed HSS content on student knowledge, and to determine whether our learners truly embrace the proposition that physicians must be system citizens who are obligated to work toward improvement of the overall health care systems in which they will practice.