Military Medicine

USN, Anthony R. Artino, Jr., MSC;(Ret.), William R. Gilliland, MC USA;PhD, David F. Cruess,;PhD, Steven J. Durning, MD,

doi: 10.7205/MILMED-D-14-00562pmid: 25850119

ABSTRACT In 1980, the Uniformed Services University of the Health Sciences (USU) graduated its first class of medical students. As a national university intended to produce “career-committed” military officers and future leaders of the Military Health System, USU functions as the service academy for military medicine and public health. More than 40 years after the school's charter and 5,000 graduates since the first class, we describe the original purpose of USU and provide an update on its achievements. In particular, we address the question of the “staying power” of the University's alumni—the degree to which graduation from the nation's military medical school is associated with long years of devoted service to military medicine. At a time when the MHS is confronting the challenge of extended deployments, rising health care costs, and a growing array of threats to our nation's health, we suggest that America needs USU now more than ever. On May 24, 1980, the Uniformed Services University of the Health Sciences (USU) graduated its first class of medical students. Coincident with this event, Francis D. Moore, MD, a member of the University's Board of Regents at the time, wrote an editorial entitled “First Class” in which he commented on the school's purpose and early accomplishments.1 Moore concluded his editorial by noting that the school's success would be judged by the “staying power” of its graduates. In the author's words, “if that staying power involves long years of devoted service, the school will have succeeded.”1 Five thousand graduates later, USU has met Dr. Moore's definition of success. Forty years ago, the U.S. military faced a severe physician shortage as a result of the Vietnam War and the end of the draft. Congress responded by establishing the Health Professions Scholarship Program (HPSP) to generate doctors who would typically serve a 4-year obligation, and a national university intended to produce “career-committed” military officers and future leaders of the Military Health System (MHS).2 Subsequently named for the Louisiana Congressman who championed its creation, USU's F. Edward Hébert School of Medicine functions as the service academy for military medicine and public health. Located adjacent to the Walter Reed National Military Medical Center in Bethesda, MD, directly across the street from the National Institutes of Health (NIH), USU graduates approximately 170 medical students annually, somewhat more than its originally projected size of 150.1 Tuition is free, and medical students are commissioned and paid as Ensigns or Second Lieutenants in the Navy, Army, Air Force, or Public Health Service. Like the military service academies, USU graduates incur a service obligation to repay the cost of their education. Thirty-two students were admitted into USU's first class, 9 (28%) of which were ethnic minorities. Three-fourths had prior military experience, and 6 (20%) were academy graduates. In 2012, the 40-year anniversary of the University's charter, 43 (25%) of our 171 matriculants were ethnic minorities, 55 (32%) had prior military experience, and 12 (7%) were academy graduates. The inaugural class had 5 (15%) women; today, women comprise approximately 30% of any given class. This percentage is below the mean for civilian medical schools but nearly twice that of female commissioned officers in the armed forces.3 USU's curriculum covers the same content of civilian medical schools and much more. Because USU graduates are educated to be military officers as well as doctors, our students receive an additional 600+ hours of supplemental instruction in military history, operational medicine, tropical medicine, humanitarian assistance, ethics, and leadership. At various points in their military medical education, students participate in field exercises that emphasize problem solving, teamwork, and medical decision-making under battlefield conditions. Whereas civilian medical schools often assign students to clerkships in local community hospitals, USU sends its students to military hospitals across the United States (Fig. 1). FIGURE 1. View largeDownload slide Clinical clerkship sites utilized by the F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, MD. FIGURE 1. View largeDownload slide Clinical clerkship sites utilized by the F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, MD. Despite the depth and breadth of material covered over 4 years, USU students perform well on their licensing examinations, and 93% go on to achieve board certification, compared to 88% of civilian graduates.4 USU's unique emphasis on military medicine paid off during the recent military conflicts in Iraq and Afghanistan, when our alumni deployed around the world to treat sick and injured service members and countless civilians. Over time, USU expanded its degree offerings to include Masters and Doctorate degrees in the biomedical sciences, public health, clinical psychology, health professions education, and health administration and policy. Twenty years ago, the University launched a Graduate School of Nursing, recently named for the late Senator Daniel K. Inouye. The Armed Forces Radiobiology Research Institute was formally aligned with the University in 2006. A multicampus postgraduate dental college was started in 2010. Although research was not envisioned as part of the University's original mission, it developed naturally to address urgent needs in military readiness and health. Today, USU's departments and centers conduct high-impact research on traumatic brain injury, post-traumatic stress, combat casualty care, radiological threats, disaster medicine, emerging infectious diseases, medical education, vaccine development, human performance optimization, and many other topics. The Department of Defense's investment in research at USU and its other laboratories paid dividends during the recent wars in Iraq and Afghanistan, when the MHS achieved the highest rate of survival from battlefield injury in the history of the world. The value of USU's science is increasingly recognized by the NIH and other funders. In a recent survey, the University achieved the largest gains in federal funding for research and development of any U.S. institution of higher education, with an 894% increase in federal funding from $12.5 million in 1999 to $124.3 million in 2009.5 Has USU met Moore's definition of “staying power”? Moore postulated that the military would consider the school a success if a large proportion of USU graduates stayed in service 20 years or longer.1 Results from a recent analysis suggest the school has achieved this goal. In the U.S. Navy and Air Force, approximately 45% of USU graduates remain on active duty for at least 20 years.6 This 20-year retention rate compares favorably to the estimated 15% retention rate among HPSP graduates. Altogether, these results support an older Government Accountability Office study, which concluded that “the longer expected retention of (USU) graduates is consistent with the legislative intent of providing long-term military medical officers.”2 USU was founded as a leadership academy. Since its creation, countless USU alumni have held senior positions of clinical and administrative leadership. For example, a 2013 analysis of Navy physicians indicated that although USU graduates represent a consistent 10 to 14% of annual accessions, USU alumni held 27% of commanding officer and executive officer billets—positions that represent the pinnacle of military medical leadership.7 Moreover, the proportion of Navy leadership positions held by USU graduates has continued to trend upward over the last decade.7 More broadly, across the armed services, 27 USU graduates have reached flag rank as generals or admirals, including the recent Surgeon General of the U.S. Air Force, two Deputy Surgeons General, and the Surgeon General of the Canadian Armed Forces. The value of USU alumni to the nation extends beyond their military service. A 2012 study of 2,750 USU graduates revealed that 71% of those who had retired from the military continue to work as civilian doctors in military clinics, Veteran Administration hospitals, and other federal facilities.4 Today, USU alumni practice in all 50 states and overseas. Because “staying power” is an important metric of USU's success, the University systematically monitors the career trajectories of its graduates through a Long-Term Career Outcome Study (LTCOS).8 The LTCOS has three primary missions: (1) collect and analyze educational data to generate evidence-based evaluations of the school's success in meeting its educational objectives; (2) generate scientific knowledge that establishes USU as a local, national, and international leader in the field of health professions education; and (3) support the translation of research in health professions education into improved policies and practices for the University and its affiliated educational programs (e.g., residencies and research centers). To date, members of the interdisciplinary LTCOS team have published more than 65 manuscripts on topics ranging from the effects of instructional authenticity on medical student learning to the assessment of diagnostic reasoning and clinical performance in residency training. This special issue of Military Medicine represents the “latest and greatest” from the LTCOS team. The issue is organized into the three phases of medical education: “before” medical school, “during” medical school, and “after” medical school. As indicated by the diverse set of studies presented in this issue, the LTCOS has a unique opportunity to answer many challenging questions about medical education and its potential effects on clinical outcomes. Because the MHS is a closed health care system, we can monitor the progress of our graduates, and their impact on patient care, for long periods of time.8 The insights produced by this work have the potential to inform medical education in civilian as well as military medical centers—a benefit Moore did not envision. Forty years and 5,000 graduates later, USU has achieved its founders' vision of a university dedicated to producing high-quality, career-committed physicians for the uniformed services. Our graduates and faculty have not only made important contributions to clinical practice but medical science as well. At a time when the MHS is confronting the challenge of extended deployments, rising health care costs, and a growing array of threats to our nation's health, America needs USU now more than ever. ACKNOWLEDGMENTS The authors would like to thank Dr. Arthur L. Kellermann, Dean of the F. Edward Hébert School of Medicine, for his thoughtful reviews and suggested revisions to earlier versions of this manuscript. REFERENCES 1. Moore FD First class. N Engl J Med  1980; 302: 1202– 3. Google Scholar CrossRef Search ADS PubMed  2. Government Accountability Office (GAO) Military Physicians: DOD's Medical School and Scholarship Program. GAO/HEHS-95-244 . Washington, DC, GAO, 1995. Available at http://www.gpo.gov/fdsys/pkg/GAOREPORTS-HEHS-95-244/content-detail.html; accessed November 17, 2014. 3. Office of the Under Secretary of Defense, Personnel and Readiness Population Representation in the Military Services: Fiscal year 2010 . Washington, DC, Department of Defence, 2010. Available at http://prhome.defense.gov/RFM/MPP/ACCESSION%20POLICY/PopRep2010/; accessed November 17, 2014. 4. DeZee KJ, Durning SJ, Dong T, et al.   Where are they now? USU School of Medicine graduates after their military obligation is complete. Mil Med  2012; 177: 68– 71. Google Scholar CrossRef Search ADS PubMed  5. The Chronicle of Higher Education Federal Science Funds Doubled at 28 Colleges , August 21, 2011. Available at http://chronicle.com/article/Federal-Science-Funds-Doubled/128258/; accessed November 17, 2014. 6. Dietrich EJ, Kimsey L, Reamy BV, Artino AR The Uniformed Services University of the Health Sciences: developing career-committed military medical officers. Mil Med  2015; 180( 4 Suppl): 172. Google Scholar CrossRef Search ADS PubMed  7. Dietrich EJ, Kimsey L, Artino AR The Uniformed Services University of the Health Sciences: a leadership academy for military medical officers in the U.S. Navy. Mil Med  2015; 180( 4 Suppl): 171. Google Scholar CrossRef Search ADS PubMed  8. Durning SJ, Artino AR, Dong T, et al.   The Long-Term Career Outcome Study (LTCOS): what have we learned from 40 years of military medical education and where should we go? Mil Med  2012; 177: 81– 6. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.

USA, Aaron Saguil, MC;PhD, Ting Dong,;USA, Robert J. Gingerich, MSC;PhD, Kimberly Swygert,;MC, Jeffrey S. LaRochelle, USAF;USN, Anthony R. Artino, Jr., MSC;PhD, David F. Cruess,;PhD, Steven J. Durning, MD,

doi: 10.7205/MILMED-D-14-00550pmid: 25850120

USA, Nathalie D. Paolino, MC;USN, Anthony R. Artino, Jr., MSC;USA, Aaron Saguil, MC;PhD, Ting Dong,;PhD, Steven J. Durning, MD,;USA, Kent J. DeZee, MC

doi: 10.7205/MILMED-D-14-00553pmid: 25850121

ABSTRACT Objectives: This article explores specific aspects of self-reported clinical and research experience and their relationship to performance in medical training. Methods: This is a retrospective cohort study conducted at the Uniformed Services University. The American Medical College Application Service application was used to discern students' self-reported clinical and research experience. Two authors applied a classification scheme for clinical and research experience to the self-reported experiences. Study outcomes included medical school grade point average (GPA), U.S. Medical Licensing Examination (USMLE) scores, and intern expertise and professionalism scores. A linear regression analysis was conducted for each outcome while controlling for prematriculation GPA. Results: Data were retrieved on 1,020 matriculants. There were several statistically significant but small differences across outcomes when comparing the various categories of clinical experience with no clinical experience. The technician-level experience group had a decrease of 0.1 in cumulative GPA in comparison to students without self-reported clinical experience (p = 0.004). This group also performed 5 points lower on the USMLE Step 2 than students who did not report clinical experience (p = 0.013). The various levels of self-reported research experience were unrelated to success in medical school and graduate medical education. Discussion: These findings indicate that self-reported technician-level clinical experience is related to a small reduction in typically reported outcomes in medical school. INTRODUCTION The selection of future physicians is a high stakes endeavor for students, educators, and society. It is vital that each student entering training is able to successfully graduate into independent practice, given the extensive resources used to educate students. Educators and admission committees have struggled to identify those applicants who will be able to achieve independent practice, as no single factor identified to date can accurately predict successful training with precision. Prior research has evaluated the extent to which factors such as personality, previous performance, personal statements, letters of recommendations, Medical College Admission Test (MCAT), gender, ethnicity, learning styles, and interviews may help to predict future performance.1 A previous meta-analysis reported that 23% of the variance seen in medical school performance is accounted for by previous academic performance; but such previous performance only accounts for 3% of the variance in postgraduate performance.1 It may be that prematriculation data more accurately predicts classroom performance typical of early medical school and not the bedside performance measures seen in postgraduation training. Few studies have addressed other components that are often considered during the application process and that may help to predict successful completion of medical training.1 Thus, the majority of variance in medical student performance remains unexplained when using prematriculation data. Other factors that have gained interest of late are previous clinical and research experience, which have been rated as “of high importance” and “of median importance,” respectively, in a recent study of allopathic medical school admission officials.2 Previously, our group has explored the relationship of self-reported prematriculation work experience on success in undergraduate medical education (UME) and graduate medical education (GME). We found that medical students who self-reported clinical experience, defined as either being employed, volunteered in a health care setting, or having performed an observership, did not outperform students who did not report having such clinical experience.3 Similarly, we also studied the effects of self-reported research experience on UME and GME performance, which demonstrated a small effect on performance. In particular, we found a trend which suggested that research experience correlated with better performance at the beginning of medical school, but worse performance in GME training.4 Both of our previous studies did not attempt to categorize or quantify the types of research and clinical experience reported, thus the small effects may have been as a result of inadequate granularity in our unit measurement; that is, we lumped all experiences together as opposed to looking for associations with specific types of self-reported experiences. This study expands on the previous research conducted by our group. The objective was to determine if a specific type of clinical experience had a positive correlation to UME and GME performance and if students who self-reported greater research experiences (like that associated with resultant publications), outperformed students with lesser or no research experience. We hypothesized that students with higher levels of clinical experience and/or research experience would outperform their less experienced counterparts. METHODS This investigation is a retrospective cohort study and is part of the Long-Term Career Outcome Study (LTCOS) conducted at the F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences (USU). The LTCOS was developed for the purpose of collecting and analyzing educational data to generate evidence-based evaluations of the school's success in meeting its educational objectives. One aspect of LTCOS's mission is to correlate preadmission data with subsequent student performance through medical school and into residency training. Nearly all students who matriculate into USU, upon graduation, proceed to military-affiliated residency training programs across the country. This facilitates the process of tracking students and enhances the school's ability to collect data on student performance after graduation. This study sample includes students who received their USU medical degree between 1993 and 1999 (N = 1,112) and is a sequel to previous studies published in 2012 that examined the relationship of research experience and clinical experience (broadly defined) on performance in medical school and internship.3,4 Study Variables and Procedures Data were gathered through two primary sources: the University's Registrar's Office and the School of Medicine Office of Recruitment and Admissions; and surveys completed by the program director of each trainee regarding the trainee's performance during his/her internship. Records from the registrar's office provided information gathered from the American Medical College Application Service (AMCAS) application, preclinical grade point average (GPA) (first 2 years of medical school), clinical years GPA (final 2 years of medical school), cumulative medical school GPA upon graduation from medical school, and U.S. Medical Licensing Examination (USMLE) Step 1 and 2 scores. All GPAs included in this study used the commonly accepted 4-point scale. The AMCAS application is currently used by most allopathic institutions in the United States and includes a section where applicants can list work experience, extracurricular activities, honor, awards, and publications to bring to the attention of the admission committees. A mixed-method, qualitative/quantitative analysis of this open-ended application item was conducted. Two authors (NP and KJD) reviewed the open-ended comments and developed a coding scheme. The coding schemes were compared and revised until no additional categories were discovered (i.e., saturation was achieved). Data regarding clinical and research experience were categorized by level of exposure. Clinical experience was segregated into 8 categories: no experience, observation/shadowing experience, volunteer experience, unspecified health care employment, health care technician, first responder experience, health care provider experience, and experience unspecified. The technician-level group of clinical experience were students who worked in the health care field, but did not have jobs requiring significant decision-making opportunities regarding patient care (e.g., licensed practicing nurses, laboratory technicians, radiology technicians, orderlies, and administrators). Table I gives examples of the reported experiences that were classified into the above categories. Students who listed experience without being specific on the type of experience were categorized as experience not otherwise specified. Research experience (Table II) was categorized as no experience, research experience without publications, and research experience accompanied with one or more publications (including abstracts). Two authors (NP and KJD) then coded each applicant's clinical and research experience using the rubric provided in Tables I and II, with differences resolved by consensus. An inter-rater agreement of 0.91 and 0.96 using Cohen's kappa coefficient was achieved for clinical and research experience category assignment, respectively, indicating excellent agreement. TABLE I. Frequency of Clinical Experience with Corresponding Examples Clinical Experience Categories  Examples  N (%)  None  No Experience Listed, EMT Course Taken, CPR Course Taken  349 (34)  Observation Experience  Internship, Operating Room Observation, Shadowing of a Physician  101 (10)  Volunteer Experience  Red Cross Volunteer, Volunteered at X Hospital  329 (32)  Health Care Experience Unspecified  Worked in Diabetes Clinic, Worked with Doctors, MASH, Worked in ER  20 (2)  Health Care Technician Experience  Cardiovascular Technician, Pharmacy Technician, Medical Technician, Licensed Practicing Nurse, Nurses Aid, X-ray Technician, Ward Clerks  116 (11)  Health Care First Responder Experience  EMT & Medic  78 (8)  Health Care Provider  Registered Nurse, Pharmacist, Occupational Therapist, Physical Therapist, Chiropractor, Optometrists, Social Worker  11 (1)  Experience, Unspecified  VA Hospital, Hospital ER, Insane Asylum, Medical Experience Without Elaboration  20 (2)  Clinical Experience Categories  Examples  N (%)  None  No Experience Listed, EMT Course Taken, CPR Course Taken  349 (34)  Observation Experience  Internship, Operating Room Observation, Shadowing of a Physician  101 (10)  Volunteer Experience  Red Cross Volunteer, Volunteered at X Hospital  329 (32)  Health Care Experience Unspecified  Worked in Diabetes Clinic, Worked with Doctors, MASH, Worked in ER  20 (2)  Health Care Technician Experience  Cardiovascular Technician, Pharmacy Technician, Medical Technician, Licensed Practicing Nurse, Nurses Aid, X-ray Technician, Ward Clerks  116 (11)  Health Care First Responder Experience  EMT & Medic  78 (8)  Health Care Provider  Registered Nurse, Pharmacist, Occupational Therapist, Physical Therapist, Chiropractor, Optometrists, Social Worker  11 (1)  Experience, Unspecified  VA Hospital, Hospital ER, Insane Asylum, Medical Experience Without Elaboration  20 (2)  This table demonstrates some examples of experiences listed in the AMCAS application by applicants as well as the frequency for each category. There was a Cohen's κ coefficient agreement score of 0.91 for this variable. View Large TABLE I. Frequency of Clinical Experience with Corresponding Examples Clinical Experience Categories  Examples  N (%)  None  No Experience Listed, EMT Course Taken, CPR Course Taken  349 (34)  Observation Experience  Internship, Operating Room Observation, Shadowing of a Physician  101 (10)  Volunteer Experience  Red Cross Volunteer, Volunteered at X Hospital  329 (32)  Health Care Experience Unspecified  Worked in Diabetes Clinic, Worked with Doctors, MASH, Worked in ER  20 (2)  Health Care Technician Experience  Cardiovascular Technician, Pharmacy Technician, Medical Technician, Licensed Practicing Nurse, Nurses Aid, X-ray Technician, Ward Clerks  116 (11)  Health Care First Responder Experience  EMT & Medic  78 (8)  Health Care Provider  Registered Nurse, Pharmacist, Occupational Therapist, Physical Therapist, Chiropractor, Optometrists, Social Worker  11 (1)  Experience, Unspecified  VA Hospital, Hospital ER, Insane Asylum, Medical Experience Without Elaboration  20 (2)  Clinical Experience Categories  Examples  N (%)  None  No Experience Listed, EMT Course Taken, CPR Course Taken  349 (34)  Observation Experience  Internship, Operating Room Observation, Shadowing of a Physician  101 (10)  Volunteer Experience  Red Cross Volunteer, Volunteered at X Hospital  329 (32)  Health Care Experience Unspecified  Worked in Diabetes Clinic, Worked with Doctors, MASH, Worked in ER  20 (2)  Health Care Technician Experience  Cardiovascular Technician, Pharmacy Technician, Medical Technician, Licensed Practicing Nurse, Nurses Aid, X-ray Technician, Ward Clerks  116 (11)  Health Care First Responder Experience  EMT & Medic  78 (8)  Health Care Provider  Registered Nurse, Pharmacist, Occupational Therapist, Physical Therapist, Chiropractor, Optometrists, Social Worker  11 (1)  Experience, Unspecified  VA Hospital, Hospital ER, Insane Asylum, Medical Experience Without Elaboration  20 (2)  This table demonstrates some examples of experiences listed in the AMCAS application by applicants as well as the frequency for each category. There was a Cohen's κ coefficient agreement score of 0.91 for this variable. View Large TABLE II. Frequency of Self-Reported Research Experience in Our Study Population Research Experience Categories  N (%)  No Research  620 (60)  Research Without a Publication(s)  375 (37)  Research Experience with Publication(s)  29 (3)  Research Experience Categories  N (%)  No Research  620 (60)  Research Without a Publication(s)  375 (37)  Research Experience with Publication(s)  29 (3)  Cohen's κ coefficient agreement score of 0.96 was achieved for this variable. View Large TABLE II. Frequency of Self-Reported Research Experience in Our Study Population Research Experience Categories  N (%)  No Research  620 (60)  Research Without a Publication(s)  375 (37)  Research Experience with Publication(s)  29 (3)  Research Experience Categories  N (%)  No Research  620 (60)  Research Without a Publication(s)  375 (37)  Research Experience with Publication(s)  29 (3)  Cohen's κ coefficient agreement score of 0.96 was achieved for this variable. View Large Students' GPA was obtained from transcripts. The undergraduate medical school GPA is a weighted average created by multiplying each course grade by the number of contact hours for the given course, summing the weighted grades across courses, and then dividing the sum by the total number of contact hours. The resulting averages were converted to a common 4-point scale (range: 0.0–4.0). Study outcomes included GPA for the first 2 years of medical school (preclinical GPA), GPA for the final 2 years of medical school (clinical GPA), cumulative medical school GPA (preclinical and clinical GPA combined), USMLE Steps 1 and 2 scores, and postgraduate year 1(PGY-1) expertise and PGY-1 professionalism scores (described below). Students in our sample completed the Step 1 examination after their first 2 years of medical school. Students completed the Step 2 examination during the last year of medical school. Scores on the USMLE Step examinations are three digits and range from 140 to 280, with a mean ranging between 200 and 220 and a standard deviation of 20 during the years studied. In this study, we used students' scores on their first attempt at each of the step examinations. The PGY-1 expertise and professionalism scores were obtained from a survey sent out to all program directors of USU graduating classes of 1993 through 1999. This survey was developed in the early 1990s and distributed to program directors at the end of each student's internship. Interdepartmental educators with expertise in GME and UME developed this form that was validated in 2005 with a Cronbach's α of 0.96.5 This survey had an annual response rate of 72 to 90%. Survey items were categorized as evaluating expertise (13 items) or professionalism (5 items). Each survey item was listed on a 5-point Likert scale, where 5 = outstanding, 4 = superior, 3 = average, 2 = needs improvement, and 1 = not satisfactory. These two constructs were used as separate outcomes in this study. A multiple linear regression analysis using STATA (version 11.2, College Station, Texas) was conducted for each outcome (dependent variable) to determine the predictive power of self-reported clinical exposure and research exposure while controlling for prematriculation GPA, a variable typically found to be related to performance. Dummy variables were used for clinical exposure and research exposure, with no experience being the referent category. Effect sizes were used to assess the strength of the statistically significant associations.6 An effect size is utilized to demonstrate the clinical significance of a variable that is statistically significant such that an effect size of less than 0.2 is essentially meaningless, one of 0.2 to 0.5 is small, 0.5 to 0.8 is considered moderate, and a reported effect size of greater than 0.8 represents a large clinical significance.6 Each group was compared to the no-experience group within their respective category. The USU's Institutional Review Board provided ethical approval for this study. RESULTS Of the 1,112 medical students who graduated from USU between 1993 and 1999, complete records were retrieved on 1,024 matriculants (92%) who had a mean prematriculation GPA of 3.41 (SD 0.3). Eighty-eight files had missing data, due most often to missing program director letters since some of the students attended nonmilitary residency programs. Our study population was predominantly male (75%) with an average age of 24.7 (SD = 1.6), which is representative of the medical student population at USU. Students who listed no clinical experience were the predominant category for this variable (349 students) followed by 329 students with volunteer experience. The frequency of each self-reported experience can be viewed in Table I. Of the 1,024 matriculants (Table II), 620 reported no research experience, whereas 375 stated they had performed some type of research activity while only 29 of the matriculants reported a research publication in a peer-reviewed journal. There were several statistically significant, but small differences when comparing the various categories of self-reported clinical experience with no clinical experience across all 7 evaluated outcomes. The technician-level experience group appeared to perform slightly worse during medical school with a decrease of 0.1 in cumulative GPA (based on a 4-point scale) in comparison to students without self-reported clinical experience (p = 0.004). However, the effect size for this difference was small (0.27). This group also performed 5 points lower on the USMLE Step 2 than students who did not report a clinical experience (p = 0.013), but again the effect size (0.26) was small. Table III demonstrates the coefficients of regression achieved with our linear regression model using no experience as the reference category. This table demonstrates that the technician-level experience group performed consistently lower during UME; however, this difference resolved during GME training. The effect size for each of these statistically different performances was small or very small (and thus not practically important) depending on the specific outcome. TABLE III. Impact of Clinical Experience on Outcome Measures Reflected by Multiple Linear Regression Models Clinical Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Observation Experience  −0.09  −1.8  −0.05  −1.8  −0.10  −0.004  0.12  Volunteer Work  −0.06  −0.9  −0.03  −1.0  −0.07  −0.15*  −0.12  Work Unspecified  −0.11  −3.7  0.01  2.7  −0.07  −0.09  −0.24  Technician Level Experience  −0.11*  −3.8*  −0.08*  −4.9*  −0.11**  0.004  −0.03  First Responder Experience  −0.16**  −2.0  −0.02  −2.2  −0.09  −0.15  −0.17  Health Care Provider Experience  0.07  0.3  0.06  7.0  0.05  −0.20  −0.29  Experience Unspecified  −0.10  −2.1  −0.11  −4.0  −0.10  −0.13  0.04  Clinical Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Observation Experience  −0.09  −1.8  −0.05  −1.8  −0.10  −0.004  0.12  Volunteer Work  −0.06  −0.9  −0.03  −1.0  −0.07  −0.15*  −0.12  Work Unspecified  −0.11  −3.7  0.01  2.7  −0.07  −0.09  −0.24  Technician Level Experience  −0.11*  −3.8*  −0.08*  −4.9*  −0.11**  0.004  −0.03  First Responder Experience  −0.16**  −2.0  −0.02  −2.2  −0.09  −0.15  −0.17  Health Care Provider Experience  0.07  0.3  0.06  7.0  0.05  −0.20  −0.29  Experience Unspecified  −0.10  −2.1  −0.11  −4.0  −0.10  −0.13  0.04  * p < 0.05 and ** p < 0.01 while correcting for undergraduate GPA. Reported regression coefficients are unstandardized. View Large TABLE III. Impact of Clinical Experience on Outcome Measures Reflected by Multiple Linear Regression Models Clinical Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Observation Experience  −0.09  −1.8  −0.05  −1.8  −0.10  −0.004  0.12  Volunteer Work  −0.06  −0.9  −0.03  −1.0  −0.07  −0.15*  −0.12  Work Unspecified  −0.11  −3.7  0.01  2.7  −0.07  −0.09  −0.24  Technician Level Experience  −0.11*  −3.8*  −0.08*  −4.9*  −0.11**  0.004  −0.03  First Responder Experience  −0.16**  −2.0  −0.02  −2.2  −0.09  −0.15  −0.17  Health Care Provider Experience  0.07  0.3  0.06  7.0  0.05  −0.20  −0.29  Experience Unspecified  −0.10  −2.1  −0.11  −4.0  −0.10  −0.13  0.04  Clinical Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Observation Experience  −0.09  −1.8  −0.05  −1.8  −0.10  −0.004  0.12  Volunteer Work  −0.06  −0.9  −0.03  −1.0  −0.07  −0.15*  −0.12  Work Unspecified  −0.11  −3.7  0.01  2.7  −0.07  −0.09  −0.24  Technician Level Experience  −0.11*  −3.8*  −0.08*  −4.9*  −0.11**  0.004  −0.03  First Responder Experience  −0.16**  −2.0  −0.02  −2.2  −0.09  −0.15  −0.17  Health Care Provider Experience  0.07  0.3  0.06  7.0  0.05  −0.20  −0.29  Experience Unspecified  −0.10  −2.1  −0.11  −4.0  −0.10  −0.13  0.04  * p < 0.05 and ** p < 0.01 while correcting for undergraduate GPA. Reported regression coefficients are unstandardized. View Large Self-reported research experience quality appeared to have no significant predictive value for success in medical school and GME (Table IV). Students who performed research, but did not produce a publication performed slightly better on USMLE Step 1 by 2.8 points (p = 0.012) compared to students without research experience; however, the effect size (0.17) was trivial. Similarly, although the students who performed research, but did not achieve a peer-reviewed publication, were slightly less professional than students who had no research experience, this effect size (0.16) also appeared to be trivial. There was no significant difference noted between students who were published before entering medical school in comparison to the students who did not report research activity. TABLE IV. Impact of Research Experience on Outcome Measures Reflected by Multiple Linear Regression Models Research Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Research Experience Without Publication(s)  0.03  2.8*  −0.002  0.3  0.002  −0.08  −0.12*  Research Experience With Publication(s)  0.10  0.6  0.05  1.6  0.06  0.03  0.06  Research Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Research Experience Without Publication(s)  0.03  2.8*  −0.002  0.3  0.002  −0.08  −0.12*  Research Experience With Publication(s)  0.10  0.6  0.05  1.6  0.06  0.03  0.06  * p < 0.05 and **p < 0.01 while correcting for undergraduate GPA. Reported regression coefficients are unstandardized. View Large TABLE IV. Impact of Research Experience on Outcome Measures Reflected by Multiple Linear Regression Models Research Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Research Experience Without Publication(s)  0.03  2.8*  −0.002  0.3  0.002  −0.08  −0.12*  Research Experience With Publication(s)  0.10  0.6  0.05  1.6  0.06  0.03  0.06  Research Experience  Regression Coefficients of Explanatory Variables  Preclinical Years GPA  USMLE Step 1  Clinical Years GPA  USMLE Step 2  Cumulative GPA  PGY-1 Expertise  PGY-1 Professionalism  No Experience  Referent  Referent  Referent  Referent  Referent  Referent  Referent  Research Experience Without Publication(s)  0.03  2.8*  −0.002  0.3  0.002  −0.08  −0.12*  Research Experience With Publication(s)  0.10  0.6  0.05  1.6  0.06  0.03  0.06  * p < 0.05 and **p < 0.01 while correcting for undergraduate GPA. Reported regression coefficients are unstandardized. View Large DISCUSSION After an initial review using school-specific criteria (often anchored by undergraduate GPA and performance on the MCAT), most medical school admission committees consider all aspects of an applicant's record in a holistic review process.2 Our study provides outcome-based data to provide guidance on the use of commonly reported clinical and research experience for selection to medical school. Unfortunately, these self-reported experiences explained little of the variance in medical school and initial GME performance. When controlling for prematriculation GPA, only technician-level students performed significantly differently than students with other or no clinical experience across multiple outcomes. These students performed worse, though the differences were small. Thus, our data suggests that medical school admission committees should not give “extra points for clinical experience” to students who report technician-level experience. However, this finding does not necessarily devalue the previous experiences of students in regard to the perspectives they bring to the patient care setting. It is important to note that this study does not evaluate this more holistic aspect of student experiences and the value that each student brings to medicine. In regard to research, students with any level of research experience performed essentially the same as students without research experience when research experience was separated into two categories determined by the presence of a publication. This result is slightly different from our previous study which found a small positive effect in early medical school education; however, the effect size found previously was small and likely not practically important from an admissions process standpoint. In the present study, we further categorized research experience, which likely accounts for the differences between the current findings and our previous work. Given that only one category of self-reported clinical experience helped predict medical school and GME outcomes when evaluating both clinical and research experience, it is useful to consider other avenues of inquiry. Clinical and research experiences are often components of the applicant's history, which help to determine acceptance into medical school. In 2008, a cohort study described this very phenomenon. In the study, researchers administered a survey to U.S. and Canadian medical school administrators and found that they ranked the presence of clinical experience to be “as important” in granting acceptance in to medical school as personal statements and MCAT scores on the biological section.2 Volunteer activities in a medical environment were only outranked by GPA (3.6 on scale of 1 = not important to 5 = extremely important), letters of recommendation (3.7), and the interview recommendation (4.5).2 These results demonstrate the degree to which some admissions committees favor these factors in granting acceptance. These experiences serve as surrogate marker for noncognitive traits where data on the predictive nature for success has been lacking. Other studies have discussed the creation of a standardized interview process to help evaluate noncognitive traits such as personal qualities7,–9 that are desired in a physician; however, other reviews have indicated that most studies that have evaluated the effectiveness of such approaches have found standard interviews to be less effective at predicting future medical school grades, licensing examinations and intern performance. Thus, some have argued that using noncognitive traits for admissions is an unreliable way to determine future success in practice.10 To address this issue, Eva et al have worked to develop the multiple mini-interview (MMI) instead of the traditional interviewing panels. Applicants selected using the MMI methods have demonstrated higher scores in Canadian national licensing examinations in those students accepted through this process.8 Our group has also evaluated our interview process in comparison to performance in GME training and have found that negative comments by the interviewer correlate to poor evaluations by program directors on professionalism; however, this effect was quite small.11 We have also examined the predictive value of the personal essay when scored using a journalist scoring system. This too did not provide any value in predicting success throughout medical training.12 Considering the studies discussed above, it seems that noncognitive trait evaluation is an ongoing area of research in medical education. Further study of noncognitive traits in relation to past experiences may help explain why students with technician-level experience in our study performed worse than those without experience. This finding may be more related to specific attitudes common in these students and their drive for obtaining a medical degree than to aptitude. Students who report technician-level clinical experience might be more likely to focus on the end goal of independent practice than on the process of becoming a physician. As such, these students may place less emphasis on the pathophysiology of disease, for example, and more on just getting through the material and day. Perhaps these students also have an elevated level of academic self-efficacy as a result of past successes in clinical environments.13 However, such high levels of confidence may be somewhat misplaced in relation to physician training. Although speculative, further research is clearly needed to explain the effect found in the current study. Our study has several limitations. First, MCAT scores were not systematically recorded in the LTCOS database during the time period of the study; however, we believe that previous undergraduate GPA serves as a good surrogate marker. Furthermore, only 1% of the students had the highest category of a health care provider–level experience, which limited our ability to make generalizations about that group. This also applies to our students who had research experience and a publication. Third, confirmation of these self-reported experiences was also not accomplished, and our database also did not quantify the amount of experience each student had (e.g., hours, days, weeks, months). Our database also did not indicate whether students performed this experience while also maintaining a full academic load in undergraduate education or conducted this experience before or after seeking their bachelor's degree. Nevertheless, we believe the type of experience serves as a reasonable surrogate marker for the time investment required to perform each activity. Thus, a student with shadowing experience may have invested less time and effort than a student who had previously been a health care provider since this type of experience requires, at minimum, an associate's degree, though some shadowing students may have had many hours of experience. Likewise, this assumption would apply to research experience. For instance, writing a peer-reviewed publication requires significant investment into a project and would likely demonstrate an increased level of involvement in the research activity performed. Training physicians is a long, effortful process for all those involved. Therefore, it is important to pick the right candidate who will have the best opportunity to be successful as a student and progress to a clinician who will continue to be invested in his/her ongoing education and development. Not doing so risks wasting societal resources, including both financial resources and time. Much like other medical schools, our university now requires a letter of recommendation from a clinician. This encourages students to seek authentic exposure to medicine before applying to medical school. In doing so, some potential applicants may decide against a career as a physician after having the chance to experience the trials and tribulations inherent to medicine, albeit in a limited scope. This experience may also serve to help them pick further specialty choice. It is unclear, however, whether such exposure actually improves the outcome of generating an independent, self-directed practicing physician. In conclusion, our study suggests that schools should consider not viewing self-reported, technician-level experience as a preferential characteristic when deciding admittance to medical school. REFERENCES 1. Ferguson E, James D, Madeley L Factors associated with success in medical school: systematic review of the literature. BMJ  2002; 324: 952– 7. Google Scholar CrossRef Search ADS PubMed  2. Monroe A, Quinn E, Samuelson W, Dunleavy DM, Dowd KW An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s? Acad Med  2013; 88: 672– 81. Google Scholar CrossRef Search ADS PubMed  3. Artino AR Jr, Gilliland WR, Waechter DM, Cruess D, Calloway M, Durning SJ Does self-reported clinical experience predict performance in medical school and internship? Med Educ  2012; 46: 172– 8. Google Scholar CrossRef Search ADS PubMed  4. Dong T, Durning SJ, Gilliland WR, et al.   Exploring the relationship between self-reported research experience and performance in medical school and internship. Mil Med  2012; 177: 11– 5. Google Scholar CrossRef Search ADS PubMed  5. Durning SJ, Pangaro LN, Lawrence LL, Waechter D, McManigle J, Jackson JL The feasibility, reliability, and validity of a program director's (supervisor's) evaluation form for medical school graduates. Acad Med  2005; 80: 964– 8. Google Scholar CrossRef Search ADS PubMed  6. Kazis LE, Anderson JJ, Meenan RF Effect sizes for interpreting changes in health status. Med Care  1989; 27: S178– 89. Google Scholar CrossRef Search ADS PubMed  7. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PM Assessing personal qualities in medical school admissions. Acad Med  2003; 78: 313– 21. Google Scholar CrossRef Search ADS PubMed  8. Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, Norman GR Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. JAMA  2012; 308: 2233– 40. Google Scholar CrossRef Search ADS PubMed  9. Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ  2009; 43: 767– 75. Google Scholar CrossRef Search ADS PubMed  10. Benbassat J, Baumal R Uncertainties in the selection of applicants for medical school. Adv Health Sci Edu Theory Pact  2007; 12: 509– 21. Google Scholar CrossRef Search ADS   11. Gilliland WR, Dong T, Artino AR Jr, et al.   Relationship between admissions committee review and student performance in medical school and internship. Mil Med  2012; 177: 21– 5. Google Scholar CrossRef Search ADS PubMed  12. Dong T, Kay A, Artino AR Jr, et al.   Application essays and future performance in medical school: are they related? Teach Learn Med  2013; 25: 55– 8. Google Scholar CrossRef Search ADS PubMed  13. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA  2006; 296: 1094– 102. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.

PhD, Steven J. Durning, MD,;PhD, Ting Dong,;(Ret.), Paul A. Hemmer, USAF MC;(Ret.), William R. Gilliland, MC USA;PhD, David F. Cruess,;PhD, John R. Boulet,;(Ret.), Louis N. Pangaro, MC USA

doi: 10.7205/MILMED-D-14-00569pmid: 25850122

ABSTRACT Purpose: To determine if there is an association between several commonly obtained premedical school and medical school measures and board certification performance. We specifically included measures from our institution for which we have predictive validity evidence into the internship year. We hypothesized that board certification would be most likely to be associated with clinical measures of performance during medical school, and with scores on standardized tests, whether before or during medical school. Methods: Achieving board certification in an American Board of Medical Specialties specialty was used as our outcome measure for a 7-year cohort of graduates (1995–2002). Age at matriculation, Medical College Admissions Test (MCAT) score, undergraduate college grade point average (GPA), undergraduate college science GPA, Uniformed Services University (USU) cumulative GPA, USU preclerkship GPA, USU clerkship year GPA, departmental competency committee evaluation, Internal Medicine (IM) clerkship clinical performance rating (points), IM total clerkship points, history of Student Promotion Committee review, and United States Medical Licensing Examination (USMLE) Step 1 score and USMLE Step 2 clinical knowledge score were associated with this outcome. Results: Ninety-three of 1,155 graduates were not certified, resulting in an average rate of board certification of 91.9% for the study cohort. Significant small correlations were found between board certification and IM clerkship points (r = 0.117), IM clerkship grade (r = 0.108), clerkship year GPA (r = 0.078), undergraduate college science GPA (r = 0.072), preclerkship GPA and medical school GPA (r = 0.068 for both), USMLE Step 1 (r = 0.066), undergraduate college total GPA (r = 0.062), and age at matriculation (r = −0.061). In comparing the two groups (board certified and not board certified cohorts), significant differences were seen for all included variables with the exception of MCAT and USMLE Step 2 clinical knowledge scores. All the variables put together could explain 4.1% of the variance of board certification by logistic regression. Conclusions: This investigation provides some additional validity evidence that measures collected for purposes of student evaluation before and during medical school are warranted. INTRODUCTION The most widely used measure to judge the quality of an individual physician is achieving specialty board certification. There are several findings in support of this rationale. Prior studies have demonstrated an association between board certification and clinical practice outcomes1,2 and between maintenance of certification examination scores and quality of care for Medicare beneficiaries.3 Board certification is also the final culminating certification in a typical physician's career, reflecting many years of education: medical school (4 years), residency and for some fellowship (3–9 years), and, in other cases, 2 years of clinical practice. As such, individuals who sit for the examination have successfully completed these several stages that include components of both written examinations and clinical performance measures. Finally, the general quality and rigor of the clinical vignette items is believed to be associated with practice.4 There is also a significant cost with not becoming board certified. Many acknowledge a physician shortage, at least in our current system of health care delivery, and not being successful with this final step can be at great cost to the individual with having spent years in medical school and subsequent training. Although an individual who is licensed but not board certified may practice medicine, these individuals currently are not required to participate in a maintenance of licensure process (beyond that required by their individual state medical board) similar to the maintenance of certification from the certifying boards; however, there is a recent effort to implement maintenance of licensure.5 From the societal view, it would be optimal to identify individuals who will not become board certified as early as possible in their education. If these individuals could be successfully identified, resources might be then spent to provide remediation and improvement in subsequent performance to include passing this examination, at least for some of these individuals. Our study's premise is that board certification is an appropriate long-term outcome for the system of selecting and training medical students. Beyond the use of in-training examination scores in residency, little is known regarding what predicts eventual board certification.6,–8 Few studies have addressed this question beyond performance on national examinations during medical school. This is likely because of several reasons. First, large cohorts are needed for analysis given that fortunately, only a small percentage of individuals who take the examination do not become certified. Second, there are challenges with following individual trainees longitudinally, and therefore single point in time large national examinations tend to be used for the purpose of board certification prediction. Third, many question the association, particularly between medical school measures and performance on the board certification examination given the amount of time and accrued knowledge and experience between medical school and board certification; in other words, from this view, board certification depends far more on residency training than premedical school or medical school measures. We explored the association between several commonly obtained premedical school and medical school measures and board certification performance. We specifically included measures from our institution or from other studies that have predictive validity evidence at least into the internship year. We hypothesized that clinical measures of performance would be most likely to be associated with board certification in addition to scores on national standardized tests as several investigations have shown that national standardized examinations predict future examinations.9,10 METHODS The study cohort was Uniformed Service University (USU) F. Edward Hébert School of Medicine students graduating between 1995 and 2002. The measures under investigation were board certification (certified or not), age at matriculation, Medical College Admissions Test (MCAT) score, undergraduate college grade point average (GPA), undergraduate college science GPA, USU cumulative GPA, USU preclerkship GPA, USU clerkship year GPA, departmental competency committee evaluation (Department of Medicine Education Committee [DOMEC], yes or no for review), Internal Medicine (IM) clerkship clinical points, IM total clerkship points, Student Promotion Committee (SPC) review (yes or no for review), and United States Medical Licensing Examination (USMLE) Step 1 score and USMLE Step 2 clinical knowledge (CK) score. Measures Board Certification Performance was divided dichotomously into having achieved certification or not, based on American Medical Association Masterfile (accessed September 2013 for USU graduates between years 1995 and 2002). Age at Matriculation The age of the students when they were accepted by USU was included, as questions regarding age at matriculation and medical school performance have been investigated in the past and age is a marker used by several admissions committees. Medical College Admissions Test The MCAT measure only included Classes of 1996 to 2002 because Class of 1995 was on a different version of MCAT. Undergraduate College GPA and Undergraduate College Science GPA We obtained the GPA measures from the admissions office. We included the cumulative GPA and the science/math GPA when they applied for USU. Medical School Cumulative GPA, Preclerkship GPA, and Clerkship Year GPA The GPA is a weighted average created by multiplying each course grade by the number of contact hours for the given course, summing the weighted grades across courses, and then dividing the sum by the total number of contact hours. Preclerkship GPA was calculated using course grades from the first 2 years of medical school. Clerkship year GPA contains no course grades. It is a composite of students' clerkship grades, which include clinical points, objective structured clinical examination scores, and National Board Medical Examiner (NBME) subject examination in IM. At the time of the study, clerkship year GPA referred to the third year of medical school where students completed their core clinical clerkship rotations, working in various ward and clinic settings. The medical school cumulative GPA was the average GPA of all 4 years. DOMEC Review If a student received one or more evaluative recommendations of less than passing from any teacher, or if a student failed the NBME shelf examination in medicine, the student was referred to the Department of Medicine Education Committee (DOMEC), an IM competency committee for grade adjudication. Thus DOMEC referral is a potential marker for a struggling student during the IM clerkship. IM Clerkship Clinical Points and Total Points During the student's 12-week IM clerkship (6 weeks inpatient and 6 weeks of ambulatory medicine), teachers recommend a final judgment of student performance based on the Reporter–Interpreter–Manager–Educator (RIME) framework,11 and include such recommendations on end of clerkship evaluation forms and comment at face-to-face evaluation sessions during the clerkship.12 These recommendations are converted to points for the purpose of grade calculations; in general, more points are allocated to teachers who work with the students for a longer period of time, and the second 6 weeks of the clerkship was weighted more heavily than the first 6 weeks (1.5 times more points available). Clinical points represent the summation of all points from the teachers during the 12-week clerkship. Clerkship “total points” represent the summation of clinical points and examination points, the latter points coming from the NBME subject examination in medicine, an in-house examination of analytic ability, and an in-house examination of interpreting laboratory values.13 We included these variables as the IM clerkship has also been found in prior studies to predict poor performance during internship.14,15 SPC Review Appearance before our SPC for any reason. Individuals are presented to this committee for concerns regarding academic performance. This variable has also been associated with poor performance in internship from a prior investigation at our institution.16 USMLE Step 1 Score and Step 2 CK Score The USMLE is a single-assessment program consisting of four separate examinations designed to assess an examinee's understanding of and ability to apply concepts and principles that are important in health, disease, and effective patient care. Students in this sample completed Step 1, which focuses on understanding of basic sciences, after their first 2 years of medical school. Students completed Step 2 CK during their fourth year of medical school. Step 2 CK is more clinically oriented compared with Step 1. Statistical Analysis We first reported the descriptive statistics of the measures in the board-certified and the not board-certified groups. Then, we examined the bivariate Pearson correlations between the measures. Last, we compared the differences between the two groups on the variables of interest using Student's t and χ2 tests. This study was approved by USU Institutional Review Board. RESULTS American Board of Medical Specialties certification data was available for 92% (1,155/1,255) of graduates during the study period. Table I shows the number and percentage of board certification vs. not board certification by graduating year. In all, 93 of 1,155 graduates were not certified, resulting in an average rate of board certification of 91.9% for the study cohort. Table II displays the results of the bivariate statistical comparisons between the two groups, and shows that the board-certified group was generally younger and showed better performance on the majority of our included measures (significant differences were seen for all included variables with the exception of MCAT and USMLE and Step 2 CK scores). Table III presents the bivariate Pearson correlation coefficients between the measures. This table reveals that board certification had weak to absent correlations with all our premedical school and medical school measures. Weak significant correlations were found between board certification and IM clerkship clinical points (r = 0.117), IM clerkship total points (r = 0.108), clerkship year GPA (r = 0.078), undergraduate college science GPA (r = 0.072), preclerkship GPA and medical school GPA (r = 0.068 for both), USMLE Step 1 (r = 0.066), undergraduate college total GPA (r = 0.062), and age at matriculation (r = −0.061). All the variables put together could explain 4.1% of the variance of board certification by logistic regression. As expected, the correlation between MCAT and Step 1 and Step 2 were significant and meaningful. TABLE I. Frequency of Board Certification by Graduating Year (1995–2002) Graduating Year  Not Board Certified (%)  Board Certified (%)  Total  1995  7 (4.8)    138 (95.2)    145  1996  4 (2.9)    135 (97.1)    139  1997  15 (10.4)    129 (89.6)    144  1998  16 (11.3)    125 (88.7)    141  1999  17 (11.4)    132 (88.6)    149  2000  6 (4.1)    139 (95.9)    145  2001  13 (9.4)    125 (90.6)    138  2002  15 (9.7)    139 (90.3)    154  1995–2002  93 (8.1)  1,062 (91.9)  1,155  Graduating Year  Not Board Certified (%)  Board Certified (%)  Total  1995  7 (4.8)    138 (95.2)    145  1996  4 (2.9)    135 (97.1)    139  1997  15 (10.4)    129 (89.6)    144  1998  16 (11.3)    125 (88.7)    141  1999  17 (11.4)    132 (88.6)    149  2000  6 (4.1)    139 (95.9)    145  2001  13 (9.4)    125 (90.6)    138  2002  15 (9.7)    139 (90.3)    154  1995–2002  93 (8.1)  1,062 (91.9)  1,155  View Large TABLE I. Frequency of Board Certification by Graduating Year (1995–2002) Graduating Year  Not Board Certified (%)  Board Certified (%)  Total  1995  7 (4.8)    138 (95.2)    145  1996  4 (2.9)    135 (97.1)    139  1997  15 (10.4)    129 (89.6)    144  1998  16 (11.3)    125 (88.7)    141  1999  17 (11.4)    132 (88.6)    149  2000  6 (4.1)    139 (95.9)    145  2001  13 (9.4)    125 (90.6)    138  2002  15 (9.7)    139 (90.3)    154  1995–2002  93 (8.1)  1,062 (91.9)  1,155  Graduating Year  Not Board Certified (%)  Board Certified (%)  Total  1995  7 (4.8)    138 (95.2)    145  1996  4 (2.9)    135 (97.1)    139  1997  15 (10.4)    129 (89.6)    144  1998  16 (11.3)    125 (88.7)    141  1999  17 (11.4)    132 (88.6)    149  2000  6 (4.1)    139 (95.9)    145  2001  13 (9.4)    125 (90.6)    138  2002  15 (9.7)    139 (90.3)    154  1995–2002  93 (8.1)  1,062 (91.9)  1,155  View Large TABLE II. Group (Board Certified vs. Not Board Certified) Comparison    Not Board Certified Mean (SD)  Board Certified Mean (SD)  t-test  Age at Matriculation  26.29 (3.75)  25.52 (3.36)  t (1,149) = 2.09*  MCAT  29.87 (3.31)  29.91 (3.22)  t (986) = −0.11  College GPA  3.39 (0.32)  3.46 (0.29)  t (1,139) = −2.08*  College Science GPA  3.36 (0.36)  3.44 (0.32)  t (1,139) = −2.45*  Cumulative Medical School GPA  3.12 (0.37)  3.21 (0.35)  t (1,099) = −2.27*  Preclinical (First 2 Years) GPA  2.93 (0.50)  3.05 (0.48)  t (1,141) = −2.31*  Initial Clerkship Year (3rd Year) GPA  3.16 (0.48)  3.28 (0.41)  t (1,140) = −2.64**  IM Clerkship Clinical Points  32.95 (18.82)  40.46 (17.12)  t (949) = −3.64**  IM Clerkship Total Grade  45.64 (22.69)  54.03 (20.81)  t (947) = −3.35**  Step 1  205.26 (17.34)  209.35 (16.83)  t (1134) = −2.23*  Step 2 CK  204.04 (20.61)  206.82 (18.84)  t (1135) = −1.34       Not Board Certified Number (Percentage)  Board Certified Number (Percentage)  Chi-Square Test    DOMEC  9 out of 46 (19.6)  70 out of 485 (14.4)  χ2 (1) = 0.87  SPC Appearance  18 out of 92 (19.6)  145 out of 1060 (13.7)  χ2 (1) = 2.41     Not Board Certified Mean (SD)  Board Certified Mean (SD)  t-test  Age at Matriculation  26.29 (3.75)  25.52 (3.36)  t (1,149) = 2.09*  MCAT  29.87 (3.31)  29.91 (3.22)  t (986) = −0.11  College GPA  3.39 (0.32)  3.46 (0.29)  t (1,139) = −2.08*  College Science GPA  3.36 (0.36)  3.44 (0.32)  t (1,139) = −2.45*  Cumulative Medical School GPA  3.12 (0.37)  3.21 (0.35)  t (1,099) = −2.27*  Preclinical (First 2 Years) GPA  2.93 (0.50)  3.05 (0.48)  t (1,141) = −2.31*  Initial Clerkship Year (3rd Year) GPA  3.16 (0.48)  3.28 (0.41)  t (1,140) = −2.64**  IM Clerkship Clinical Points  32.95 (18.82)  40.46 (17.12)  t (949) = −3.64**  IM Clerkship Total Grade  45.64 (22.69)  54.03 (20.81)  t (947) = −3.35**  Step 1  205.26 (17.34)  209.35 (16.83)  t (1134) = −2.23*  Step 2 CK  204.04 (20.61)  206.82 (18.84)  t (1135) = −1.34       Not Board Certified Number (Percentage)  Board Certified Number (Percentage)  Chi-Square Test    DOMEC  9 out of 46 (19.6)  70 out of 485 (14.4)  χ2 (1) = 0.87  SPC Appearance  18 out of 92 (19.6)  145 out of 1060 (13.7)  χ2 (1) = 2.41  * p < 0.05; ** p < 0.01. View Large TABLE II. Group (Board Certified vs. Not Board Certified) Comparison    Not Board Certified Mean (SD)  Board Certified Mean (SD)  t-test  Age at Matriculation  26.29 (3.75)  25.52 (3.36)  t (1,149) = 2.09*  MCAT  29.87 (3.31)  29.91 (3.22)  t (986) = −0.11  College GPA  3.39 (0.32)  3.46 (0.29)  t (1,139) = −2.08*  College Science GPA  3.36 (0.36)  3.44 (0.32)  t (1,139) = −2.45*  Cumulative Medical School GPA  3.12 (0.37)  3.21 (0.35)  t (1,099) = −2.27*  Preclinical (First 2 Years) GPA  2.93 (0.50)  3.05 (0.48)  t (1,141) = −2.31*  Initial Clerkship Year (3rd Year) GPA  3.16 (0.48)  3.28 (0.41)  t (1,140) = −2.64**  IM Clerkship Clinical Points  32.95 (18.82)  40.46 (17.12)  t (949) = −3.64**  IM Clerkship Total Grade  45.64 (22.69)  54.03 (20.81)  t (947) = −3.35**  Step 1  205.26 (17.34)  209.35 (16.83)  t (1134) = −2.23*  Step 2 CK  204.04 (20.61)  206.82 (18.84)  t (1135) = −1.34       Not Board Certified Number (Percentage)  Board Certified Number (Percentage)  Chi-Square Test    DOMEC  9 out of 46 (19.6)  70 out of 485 (14.4)  χ2 (1) = 0.87  SPC Appearance  18 out of 92 (19.6)  145 out of 1060 (13.7)  χ2 (1) = 2.41     Not Board Certified Mean (SD)  Board Certified Mean (SD)  t-test  Age at Matriculation  26.29 (3.75)  25.52 (3.36)  t (1,149) = 2.09*  MCAT  29.87 (3.31)  29.91 (3.22)  t (986) = −0.11  College GPA  3.39 (0.32)  3.46 (0.29)  t (1,139) = −2.08*  College Science GPA  3.36 (0.36)  3.44 (0.32)  t (1,139) = −2.45*  Cumulative Medical School GPA  3.12 (0.37)  3.21 (0.35)  t (1,099) = −2.27*  Preclinical (First 2 Years) GPA  2.93 (0.50)  3.05 (0.48)  t (1,141) = −2.31*  Initial Clerkship Year (3rd Year) GPA  3.16 (0.48)  3.28 (0.41)  t (1,140) = −2.64**  IM Clerkship Clinical Points  32.95 (18.82)  40.46 (17.12)  t (949) = −3.64**  IM Clerkship Total Grade  45.64 (22.69)  54.03 (20.81)  t (947) = −3.35**  Step 1  205.26 (17.34)  209.35 (16.83)  t (1134) = −2.23*  Step 2 CK  204.04 (20.61)  206.82 (18.84)  t (1135) = −1.34       Not Board Certified Number (Percentage)  Board Certified Number (Percentage)  Chi-Square Test    DOMEC  9 out of 46 (19.6)  70 out of 485 (14.4)  χ2 (1) = 0.87  SPC Appearance  18 out of 92 (19.6)  145 out of 1060 (13.7)  χ2 (1) = 2.41  * p < 0.05; ** p < 0.01. View Large TABLE III. Matrix of Pearson Correlations Between Board Certification and Assessment Measurements Measures  Mean (SD)  Board Certification  Age at Matriculation  MCAT  College GPA  College Science GPA  Cumulative Medical School GPA  Preclinical (First 2 years) GPA  Initial Clerkship Year (3rd Year) GPA  DOMEC Review  IM Clerkship Clinical Points  IM Clerkship Total Points  SPC Review  Step 1  Step 2 CK  Board Certification  —    −0.061*  0.003  0.062*  0.072*  0.068*  0.068*  0.078*  —  0.117**  0.108**  —  0.066*  0.040  Age at Matriculation  25.55 (3.41)      0.092**  −0.212**  −0.107**  0.043  0.060*  −0.010  0.040  −0.005  −0.021  0.041  0.021  −0.021  MCATa  29.82 (3.24)        0.010  0.033  0.203**  0.260**  0.084**  −0.08  0.044  0.111**  −0.142**  0.393**  0.275**  College GPA  3.45 (0.29)          0.870**  0.145**  0.177**  0.045  −0.119**  0.074*  0.108**  −0.024  0.183**  0.162**  College Science GPA  3.43 (0.32)            0.160**  0.197**  0.052  −0.121**  0.093**  0.123**  −0.032  0.206**  0.158**  Cumulative Medical School GPA  3.20 (0.35)              0.923**  0.776**  −0.314**  0.596**  0.697**  −0.343**  0.724**  0.645**  Preclinical (First 2 Years) GPA  3.04 (0.48)                0.515**  −0.238**  0.401**  0.507**  −0.361**  0.754**  0.602**  Initial Clerkship Year (3rd Year) GPA  3.27 (0.41)                  −0.431**  0.749**  0.800**  −0.321**  0.459**  0.512**  DOMEC  —b                    −0.456**  −0.501**  0.354**  −0.302**  −0.275**  IM Clerkship Clinical Points  39.86 (17.37)                      0.954**  −0.262**  0.340**  0.368**  IM Clerkship Total Grade  53.36 (21.07)                        −0.299**  0.485**  0.517**  SPC Appearance  —b                          −0.344**  −0.235**  Step 1  209.02 (16.90)                            0.723**  Step 2 CK  206.59 (19.00)                              Measures  Mean (SD)  Board Certification  Age at Matriculation  MCAT  College GPA  College Science GPA  Cumulative Medical School GPA  Preclinical (First 2 years) GPA  Initial Clerkship Year (3rd Year) GPA  DOMEC Review  IM Clerkship Clinical Points  IM Clerkship Total Points  SPC Review  Step 1  Step 2 CK  Board Certification  —    −0.061*  0.003  0.062*  0.072*  0.068*  0.068*  0.078*  —  0.117**  0.108**  —  0.066*  0.040  Age at Matriculation  25.55 (3.41)      0.092**  −0.212**  −0.107**  0.043  0.060*  −0.010  0.040  −0.005  −0.021  0.041  0.021  −0.021  MCATa  29.82 (3.24)        0.010  0.033  0.203**  0.260**  0.084**  −0.08  0.044  0.111**  −0.142**  0.393**  0.275**  College GPA  3.45 (0.29)          0.870**  0.145**  0.177**  0.045  −0.119**  0.074*  0.108**  −0.024  0.183**  0.162**  College Science GPA  3.43 (0.32)            0.160**  0.197**  0.052  −0.121**  0.093**  0.123**  −0.032  0.206**  0.158**  Cumulative Medical School GPA  3.20 (0.35)              0.923**  0.776**  −0.314**  0.596**  0.697**  −0.343**  0.724**  0.645**  Preclinical (First 2 Years) GPA  3.04 (0.48)                0.515**  −0.238**  0.401**  0.507**  −0.361**  0.754**  0.602**  Initial Clerkship Year (3rd Year) GPA  3.27 (0.41)                  −0.431**  0.749**  0.800**  −0.321**  0.459**  0.512**  DOMEC  —b                    −0.456**  −0.501**  0.354**  −0.302**  −0.275**  IM Clerkship Clinical Points  39.86 (17.37)                      0.954**  −0.262**  0.340**  0.368**  IM Clerkship Total Grade  53.36 (21.07)                        −0.299**  0.485**  0.517**  SPC Appearance  —b                          −0.344**  −0.235**  Step 1  209.02 (16.90)                            0.723**  Step 2 CK  206.59 (19.00)                              * p < 0.05; ** p < 0.01. a MCAT measure only included classes of 1996 to 2002 because class of 1995 was on a different version of MCAT. b SPC appearance and DOMEC were binomial variables. SPC had a frequency of 180 yes and 1,072 no. DOMEC had 84 yes, 481 no, and 690 missing records. View Large TABLE III. Matrix of Pearson Correlations Between Board Certification and Assessment Measurements Measures  Mean (SD)  Board Certification  Age at Matriculation  MCAT  College GPA  College Science GPA  Cumulative Medical School GPA  Preclinical (First 2 years) GPA  Initial Clerkship Year (3rd Year) GPA  DOMEC Review  IM Clerkship Clinical Points  IM Clerkship Total Points  SPC Review  Step 1  Step 2 CK  Board Certification  —    −0.061*  0.003  0.062*  0.072*  0.068*  0.068*  0.078*  —  0.117**  0.108**  —  0.066*  0.040  Age at Matriculation  25.55 (3.41)      0.092**  −0.212**  −0.107**  0.043  0.060*  −0.010  0.040  −0.005  −0.021  0.041  0.021  −0.021  MCATa  29.82 (3.24)        0.010  0.033  0.203**  0.260**  0.084**  −0.08  0.044  0.111**  −0.142**  0.393**  0.275**  College GPA  3.45 (0.29)          0.870**  0.145**  0.177**  0.045  −0.119**  0.074*  0.108**  −0.024  0.183**  0.162**  College Science GPA  3.43 (0.32)            0.160**  0.197**  0.052  −0.121**  0.093**  0.123**  −0.032  0.206**  0.158**  Cumulative Medical School GPA  3.20 (0.35)              0.923**  0.776**  −0.314**  0.596**  0.697**  −0.343**  0.724**  0.645**  Preclinical (First 2 Years) GPA  3.04 (0.48)                0.515**  −0.238**  0.401**  0.507**  −0.361**  0.754**  0.602**  Initial Clerkship Year (3rd Year) GPA  3.27 (0.41)                  −0.431**  0.749**  0.800**  −0.321**  0.459**  0.512**  DOMEC  —b                    −0.456**  −0.501**  0.354**  −0.302**  −0.275**  IM Clerkship Clinical Points  39.86 (17.37)                      0.954**  −0.262**  0.340**  0.368**  IM Clerkship Total Grade  53.36 (21.07)                        −0.299**  0.485**  0.517**  SPC Appearance  —b                          −0.344**  −0.235**  Step 1  209.02 (16.90)                            0.723**  Step 2 CK  206.59 (19.00)                              Measures  Mean (SD)  Board Certification  Age at Matriculation  MCAT  College GPA  College Science GPA  Cumulative Medical School GPA  Preclinical (First 2 years) GPA  Initial Clerkship Year (3rd Year) GPA  DOMEC Review  IM Clerkship Clinical Points  IM Clerkship Total Points  SPC Review  Step 1  Step 2 CK  Board Certification  —    −0.061*  0.003  0.062*  0.072*  0.068*  0.068*  0.078*  —  0.117**  0.108**  —  0.066*  0.040  Age at Matriculation  25.55 (3.41)      0.092**  −0.212**  −0.107**  0.043  0.060*  −0.010  0.040  −0.005  −0.021  0.041  0.021  −0.021  MCATa  29.82 (3.24)        0.010  0.033  0.203**  0.260**  0.084**  −0.08  0.044  0.111**  −0.142**  0.393**  0.275**  College GPA  3.45 (0.29)          0.870**  0.145**  0.177**  0.045  −0.119**  0.074*  0.108**  −0.024  0.183**  0.162**  College Science GPA  3.43 (0.32)            0.160**  0.197**  0.052  −0.121**  0.093**  0.123**  −0.032  0.206**  0.158**  Cumulative Medical School GPA  3.20 (0.35)              0.923**  0.776**  −0.314**  0.596**  0.697**  −0.343**  0.724**  0.645**  Preclinical (First 2 Years) GPA  3.04 (0.48)                0.515**  −0.238**  0.401**  0.507**  −0.361**  0.754**  0.602**  Initial Clerkship Year (3rd Year) GPA  3.27 (0.41)                  −0.431**  0.749**  0.800**  −0.321**  0.459**  0.512**  DOMEC  —b                    −0.456**  −0.501**  0.354**  −0.302**  −0.275**  IM Clerkship Clinical Points  39.86 (17.37)                      0.954**  −0.262**  0.340**  0.368**  IM Clerkship Total Grade  53.36 (21.07)                        −0.299**  0.485**  0.517**  SPC Appearance  —b                          −0.344**  −0.235**  Step 1  209.02 (16.90)                            0.723**  Step 2 CK  206.59 (19.00)                              * p < 0.05; ** p < 0.01. a MCAT measure only included classes of 1996 to 2002 because class of 1995 was on a different version of MCAT. b SPC appearance and DOMEC were binomial variables. SPC had a frequency of 180 yes and 1,072 no. DOMEC had 84 yes, 481 no, and 690 missing records. View Large DISCUSSION We explored the association between several premedical school and medical school measures and achieving board certification. Consistent with one of our hypotheses as well as prior work,14,15 IM clinical points and DOMEC presentation were associated with board certification, albeit weakly. Contrary to expectations, we did not find significant associations between USMLE Step 1 or Step 2 CK measures and board certification. Given the expected strong association between MCAT and Step 1 and Step 2 scores, we believe that the lack of association with board certification may be due to the length of time between taking these examinations and board certification as well as our dichotomous outcome variable (certified or not certified); prior studies at other institutions have shown an association between USMLE Step 1 and certification examination performance.8,17 It was also notable that undergraduate college GPA was weakly associated with board certification and that clinical year GPA added little to the explained variance from undergraduate college GPA. We were not surprised by the small sizes of the correlations, as there are many intervening steps between board certification and prematriculation as well as the fact that medical school performance measures are distant from board certification. Arguably, measures in closer proximity to board certification such as residency performance and program director evaluations would be expected to show much stronger associations. This being said, we did find some measures that were associated with board certification and, as such, identification of such measures may be of importance for academicians who view the process of medical education from the certification lens. With regard to the IM clerkship measures of performance, it is notable that these provided the strongest of the noted associations. We believe this is because of the rigor of the evaluation process during the clerkship—that we use a common frame of reference in the Reporter–Interpreter–Manager–Educator framework and use direct conversations with faculty to develop this shared mental model and facilitate the process of evaluation. The IM clerkship grading and grade review process has placed a premium on narrative assessment18 and has long used a competency committee approach for deliberation and decision making about at-risk students.19 The findings of this study provide additional evidence of the importance and value of this process, and with adoption of competency committee reviews in residency training, we would hypothesize that residency programs will enhance their identification of at-risk resident trainees.20 In addition, our work would suggest that undergraduate college GPA, and in particular science undergraduate college GPA is more important than the MCAT for predicting certification. If repeated in larger studies, we would postulate that this would be due to the concept that undergraduate college GPA is not only a measure of test performance in a single point in time but also may speak to dedication to a field over a long period of time. Also, several of our medical school measures from prior studies that have been shown to predict internship performance do have an association, albeit small, with board certification. This provides additional evidence that such medical school measures are important to collect and still provide additional justification for their use in making, at times, challenging decisions regarding student progress in our curriculum. There were several limitations of our study. First, this investigation was conducted in a single institution and may not be generalizable to other institutions. The fact that we included several measures for which we have collected validity evidence and our quite large sample size across the medical education continuum are strengths. Second, we used board certification as a binary measure as opposed to one's certification score. We suspect that the associations would be more robust with using a continuous outcome measure. Third, we did not include GME level data, such as in-training examination performance or residency performance evaluations because we did not have access to such data at the individual trainee level for this cohort. Finally, the reasons for not achieving board certification are unknown and in addition to not passing the examination, the graduate could still be in training (e.g., fellowship), decided to pursue another field, decided to remain as a General Medical Officer or decided for other reason(s) to not pursue board certification. This investigation provides some additional validity evidence that measures collected before and during medical school for purposes of student evaluation are warranted. REFERENCES 1. Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med  1989; 110: 719– 26. Google Scholar CrossRef Search ADS PubMed  2. Tamblyn R, Abrahamowicz M, Dauphinee D, et al.   Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA  2007; 298: 993– 1001. Google Scholar CrossRef Search ADS PubMed  3. Holmboe ES, Wang Y, Meehan TP, et al.   Associations between maintenance of certification examinations scores and quality of care for medicine beneficiaries. Arch Intern Med  2008; 168: 1396– 403. Google Scholar CrossRef Search ADS PubMed  4. Lipner RS, Lucey CR Putting the secure examination to the test. JAMA  2010; 304( 12): 1379– 80. Google Scholar CrossRef Search ADS PubMed  5. Maintenance of Licensure (MOL) A Framework for Medical License Renewal. Federation of State Medical Boards . Available at http://www.fsmb.org/fsmb-mol; accessed September 22, 2014. 6. McClintock JC, Gravlee GP Predicting success on the certification examinations of the American Board of Anesthesiology. Anesthesiology  2010; 112( 1): 212– 9. Google Scholar CrossRef Search ADS PubMed  7. Juul D, Sexson SB, Brooks BA, et al.   Relationship between performance on child and adolescent psychiatry in-training and certification examinations. J Grad Med Educ  2013; 5( 2): 262– 6. Google Scholar CrossRef Search ADS PubMed  8. de Virgilio C, Yaghoubian A, Kaji A, et al.   Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg  2010; 145( 9): 852– 6. Google Scholar CrossRef Search ADS PubMed  9. Sosenko J, Stekel KW, Soto R, Gelbard M NBME Examination Part I as a predictor of clinical and ABIM certifying examination performances. J Gen Intern Med  1993; 8: 86– 8. Google Scholar CrossRef Search ADS PubMed  10. Berner ES, Brooks CM, Erdmann JB Use of the USMLE to select residents. Acad Med  1993; 68: 753– 9. Google Scholar CrossRef Search ADS PubMed  11. Pangaro L A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med  1999; 74( 11): 1203– 7. Google Scholar CrossRef Search ADS PubMed  12. Hemmer PA, Pangaro L Using formal evaluation sessions for case-based faculty development during clinical clerkships. Acad Med  2000; 75( 12): 1216– 21. Google Scholar CrossRef Search ADS PubMed  13. Durning SJ, Pangaro LN, Denton GD, et al.   Intersite consistency as a measurement of programmatic evaluation in a medicine clerkship with multiple, geographically separated sites. Acad Med  2003; 78( 10 Suppl): S36– 8. Google Scholar CrossRef Search ADS PubMed  14. Lavin B, Pangaro L Internship ratings as a validity outcome measure for an evaluation system to identify inadequate clerkship performance. Acad Med  1998; 73( 9): 998– 1002. Google Scholar CrossRef Search ADS PubMed  15. Hemann BA, Durning SJ, Kelly WF, Dong T, Pangaro LN, Hemmer PA The association of students requiring remediation in the internal medicine clerkship with poor performance during internship. Mil Med  2015; 180( 4 Suppl): 47– 53. Google Scholar CrossRef Search ADS PubMed  16. Durning SJ, Cohen DL, Cruess D, McManigle JM, MacDonald R Does student promotions committee appearance predict below-average performance during internship? A seven-year study. Teach Learn Med  2008; 20( 3): 267– 72. Google Scholar CrossRef Search ADS PubMed  17. Armstrong A, Alvero R, Nielsen P, et al.   Do U.S. medical licensure examination step 1 scores correlate with council on resident education in obstetrics and gynecology in-training examination scores and American board of obstetrics and gynecology written examination performance? Mil Med  2007; 172( 6): 640– 3. Google Scholar CrossRef Search ADS PubMed  18. Hanson JL, Rosenberg AA, Lane JL Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States. Front Psychol  2013; 4: 668. Google Scholar CrossRef Search ADS PubMed  19. Gaglione MM, Moores L, Pangaro L, Hemmer PA Does group discussion of student clerkship performance at an education committee affect an individual committee member's decisions? Acad Med  2005; 80( 10 Suppl): S55– 8. Google Scholar CrossRef Search ADS PubMed  20. ACGME ACGME Common Program Requirements . Available at https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf; accessed September 21, 2014. Reprint & Copyright © Association of Military Surgeons of the U.S.

PhD, Ting Dong,;MC, Jeffrey S. LaRochelle, USAF;PhD, Steven J. Durning, MD,;USA, Aaron Saguil, MC;PhD, Kimberly Swygert,;USN, Anthony R. Artino, Jr., MSC

doi: 10.7205/MILMED-D-14-00565pmid: 25850123

ABSTRACT Background: The Essential Elements of Communication (EEC) were developed from the Kalamazoo consensus statement on physician–patient communication. The Uniformed Services University of the Health Sciences (USU) has adopted a longitudinal curriculum to use the EEC both as a learning tool during standardized patient encounters and as an evaluation tool culminating with the end of preclerkship objective-structured clinical examinations (OSCE). Medical educators have recently emphasized the importance of teaching communication skills, as evidenced by the United States Medical Licensing Examination testing both the integrated clinical encounter (ICE) and communication and interpersonal skills (CIS) within the Step 2 Clinical Skills exam (CS). Purpose: To determine the associations between students' EEC OSCE performance at the end of the preclerkship period with later communication skills assessment and evaluation outcomes in the context of a longitudinal curriculum spanning both undergraduate medical education and graduate medical education. Methods: Retrospective data from preclerkship (overall OSCE scores and EEC OSCE scores) and clerkship outcomes (internal medicine [IM] clinical points and average clerkship National Board of Medical Examiners [NBME] scores) were collected from 167 USU medical students from the class of 2011 and compared to individual scores on the CIS and ICE components of Step 2 CS, as well as to the communication skills component of the program directors' evaluation of trainees during their postgraduate year 1 (PGY-1) residency. In addition to bivariate Pearson correlation analysis, we conducted multiple linear regression analysis to examine the predictive power of the EEC score beyond the IM clerkship clinical points and the average NBME Subject Exams score on the outcome measures. Results: The EEC score was a significant predictor of the CIS score and the PGY-1 communication skills score. Beyond the average NBME Subject Exams score and the IM clerkship clinical points, the EEC score explained an additional 13% of the variance in the Step 2 CIS score and an additional 6% of the variance in the PGY-1 communication skills score. In addition, the EEC score was more closely associated with the CIS score than the ICE score. Conclusion: The use of a standardized approach with a communication tool like the EEC can help explain future performance in communication skills independent of other education outcomes. In the context of a longitudinal curriculum, this information may better inform medical educators on learners' communication capabilities and more accurately direct future remediation efforts. INTRODUCTION The importance of teaching, assessing, and improving the communication skills of physicians-in-training cannot be overemphasized—physicians must be competent communicators to effectively practice medicine.1 The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) have jointly identified communication and interpersonal skills (CIS) as one of the six general competencies for physicians.2,3 The Institute of Medicine also specifies communication skills as one of the six essential curricular domains for effective patient care.4 Postgraduate accredited training programs are required to demonstrate that they teach and evaluate trainees' communication skills. The importance of communication skills is also demonstrated in licensing examinations. The U.S. Medical Licensing Examination requires students to take a clinical skills examination with standardized patients as part of Step 2,5 the Step 2 Clinical Skills (CS) assessment; this has a separate subcomponent and passing standard for the evaluation of CIS. The development and administration of CIS assessment tools is challenging and resource intensive. It is difficult to assess communication skills through inauthentic means (such as a written test), as it requires in vivo demonstration.1 Competence in communication skills is not only about the presence of specific behaviors but also about the timing of effective verbal and nonverbal behaviors in the context of interactions with patients.6 In 1999, leaders and representatives from medical schools, residency programs, continuing medical education providers, and prominent medical educational organizations in North America gathered in Kalamazoo for the purpose of identifying and specifically articulating ways to facilitate the teaching and evaluation of physician–patient communication.7 After examining 5 models of physician–patient communication that had been used by the conference attendees, the group reached a consensus on a set of Essential Elements of Communication (EEC). From the standpoint of learning communication skills, many research studies have shown that these skills can be taught.8,9 Yedidia et al9 found that communications curricula significantly improved third-year medical students' “overall communications competence as well as their skills in relationship building, organization and time management, patient assessment, and negotiation and shared decision-making.” Aspegren8 also pointed out that those students with the lowest pretraining scores benefit the most from communication skills curriculum and that the best time to learn these skills in medical school is most likely during clinical clerkships. At the Uniformed Services University of the Health Sciences (USU), we have adopted a longitudinal curriculum using the EEC (see the  Appendix) as both a learning tool during standardized patient encounters and as an evaluation tool culminating with the end of preclerkship objective-structured clinical examination (OSCE). Students are provided with a copy of the EEC on their second day of medical school and immediately begin developing skills related to the first two domains (opening the discussion and building the relationship). Students build upon their communication skills through standardized patient interactions, real patient interviews, and small-group discussions, using the domains of the EEC as a guide. The EEC then becomes the primary assessment tool for communication skills during standardized and real patient encounters over the course of the preclerkship training period. The purpose of this study was to determine the associations between students' EEC OSCE performance during the preclerkship period with later communication skills assessment and evaluation outcomes in the context of a longitudinal curriculum spanning both undergraduate medical education (UME) and graduate medical education (GME). The outcomes were the CIS and integrated clinical encounter (ICE) scores of Step 2 CS, as well as the communication skills component of the program directors (PDs) evaluation of trainees' during their postgraduate year 1 (PGY-1) residency. If poor performance can be diagnosed as early as the end of the preclerkship period by EEC assessment, this would better inform medical educators on learner ability, and more accurately direct remediation efforts. The research hypotheses were (1) students' EEC score would explain a significant amount of variance in the Step 2 CS CIS score and the PGY-1 communication skills score, beyond the variance explained by other established clerkship performance measures, and (2) the associations between EEC score and Step 2 CS ICE score would be weaker compared with those between EEC score and CIS score since both EEC and CIS scores are measures of students' communication skills. METHODS Study Context and Participants This investigation was part of the larger Long-Term Career Outcome Study conducted at the F. Edward Hébert School of Medicine, USU. As the United States' only federal medical school, USU matriculates approximately 170 medical students annually and, at the time of this study, offered a traditional 4-year curriculum: 2 years of basic science courses followed by 2 years of clinical rotations (clerkships). The participants of the present study were students graduating in 2011 (N = 167; 58 were female [34.7%] and 109 were male [65.3%]). Measures EEC Score The EEC score on the OSCE consists of seven domains (open the discussion, build the relationship, gather information, understand the patient perspective, share information, reach agreement, and provide closure), in addition to an overall global rating. Each domain has a 5-point Likert-type scale associated with behavioral anchors. Standardized patients are trained on how to use the EEC assessment tool and complete their evaluations after each encounter with a student. The overall score on each OSCE station is converted into a percentage of available points, and the average across all OSCE stations is recorded as the final EEC score on the OSCE. According to a previously conducted but unpublished generalizability study at the USU, the second-year OSCE stations demonstrated a moderate generalizability coefficient (r = 0.52), with 18.1% and 3.7% of the overall variance explained by the OSCE station and rater, respectively. Overall, 40.8% of the total variance was explained by student ability. These generalizability values are slightly lower than the published reliabilities for the Step 2 CS components of CIS, data gathering, and patient note, but are in line with other school-level OSCE reliabilities.10 The EEC reliability estimate is higher, as a study by Joyce et al found the internal reliability coefficient (Cronbach's α) of standardized patient EEC scores to be 0.90.11 Average NBME Subject Exams Score The third-year curriculum consisted of the school's core clerkship rotations: family medicine, internal medicine (IM), general surgery, psychiatry, pediatrics, and obstetrics and gynecology. All core clerkship rotations use the relevant NBME Subject Exam, which is given near the end of the core rotation. The average NBME Subject Exam score was the un-weighted mean of the scores across the clerkships. Nationally, the mean scores (2009–2010) ranged from 73.1 (SD = 8.9) in general surgery through 78.2 (SD = 8.8) in psychiatry. IM Clerkship Clinical Points During the student's IM clerkship, teachers recommended grades for each student, and reported the number of clinics spent with each student. Teacher-recommended grades were weighted according to the number of clinics the teacher spent with the students and summarized into a measure called clinical points. CIS and ICE Scores of Step 2 CS We accessed the students' CIS and ICE scores of Step 2 through a collaboration with the NBME. The ICE subcomponent is a composite score of data gathering and data interpretation skills via both spoken and written communication. Performance on the ICE subcomponent is rated by the standardized patients for the history taking and physical examination portion (via checklist), and by trained physician raters for the patient note (via a 1–9 scale). The final ICE score is on the standardized score scale, scaled to have a mean of zero and SD of 1. The CIS subcomponent includes assessment of the patient-centered communication skills of fostering the relationship, gathering information, providing information, helping the patient make a decision, and supporting emotions. Examinees' performance of the CIS subcomponent is evaluated by the standardized patients. This component is scored as a combination of three 1 to 9 scales, with final CIS scores ranging from 3 to 27.12 PGY-1 Communication Skills Score We collect PGY-1 data annually from PDs. The items in our most recent survey were designed to parallel the six ACGME competencies. Each spring we identify the programs where our interns and residents are trained, and we mail the evaluation forms for each trainee to the respective GME PDs. The psychometric properties of this evaluation form were recently investigated, and the results indicated reasonable validity and reliability.13 The structure of the form suggested five factors or subscales—Medical Expertise, Military-unique Practice, Professionalism, System-based Practice, and Communication and Interpersonal Skills. We used the CIS subscale score, which was calculated as the average of the four items in this subscale, as an outcome in the present study (hereafter referred to as the “PGY-1 communication skills score”). Statistical Analyses First, we examined the descriptive statistics and the bivariate correlations of all the measures included. Next, we conducted multiple linear regression analyses to examine the predictive power of the EEC score beyond the IM clerkship clinical points and the average NBME Subject Exams score on the outcome measures. The USU's Institutional Review Board provided ethical approval for the present study. RESULTS Table I shows the descriptive statistics of the measures and Table II presents the bivariate Pearson correlations among the measures. The EEC score had small but statistically significant correlations with the IM clerkship clinical points (r = 0.24, p < 0.01) and the ICE scores (r = 0.17, p < 0.05), and it had moderate correlations with the CIS score (r = 0.42, p < 0.01) and the PGY-1 communication skills score (r = 0.31, p < 0.01). Both the average Subject Exams score and the IM clerkship clinical points were more strongly correlated with the ICE score (r = 0.29, p < 0.01; r = 0.37, p < 0.01) than the CIS score (r = 0.12, p = 0.16; r = 0.28, p < 0.01). The correlation between the ICE score and the CIS score approached a medium effect size (r = 0.30, p < 0.01). TABLE I. Descriptive Statistics of the Measures Measure  Mean  SD  Min  Max  EEC Score    0.66    0.09    0.38    0.84  Average NBME Subject Exams Score Across Clerkships  74.24    6.18  62.33  97.75  IM Clerkship Clinical Points  42.63  14.17  −5.30  69.50  CIS Score of Step 2 CS  20.09    1.01  17.00  22.44  ICE Score of Step 2 CS    0.34    0.74  −2.02    2.76  Communication Skills Component of PGY-1 Residency    3.70    0.74    2.00    5.00  Measure  Mean  SD  Min  Max  EEC Score    0.66    0.09    0.38    0.84  Average NBME Subject Exams Score Across Clerkships  74.24    6.18  62.33  97.75  IM Clerkship Clinical Points  42.63  14.17  −5.30  69.50  CIS Score of Step 2 CS  20.09    1.01  17.00  22.44  ICE Score of Step 2 CS    0.34    0.74  −2.02    2.76  Communication Skills Component of PGY-1 Residency    3.70    0.74    2.00    5.00  View Large TABLE I. Descriptive Statistics of the Measures Measure  Mean  SD  Min  Max  EEC Score    0.66    0.09    0.38    0.84  Average NBME Subject Exams Score Across Clerkships  74.24    6.18  62.33  97.75  IM Clerkship Clinical Points  42.63  14.17  −5.30  69.50  CIS Score of Step 2 CS  20.09    1.01  17.00  22.44  ICE Score of Step 2 CS    0.34    0.74  −2.02    2.76  Communication Skills Component of PGY-1 Residency    3.70    0.74    2.00    5.00  Measure  Mean  SD  Min  Max  EEC Score    0.66    0.09    0.38    0.84  Average NBME Subject Exams Score Across Clerkships  74.24    6.18  62.33  97.75  IM Clerkship Clinical Points  42.63  14.17  −5.30  69.50  CIS Score of Step 2 CS  20.09    1.01  17.00  22.44  ICE Score of Step 2 CS    0.34    0.74  −2.02    2.76  Communication Skills Component of PGY-1 Residency    3.70    0.74    2.00    5.00  View Large TABLE II. Bivariate Pearson Correlations Between the Measures Measure  EEC  Average NBME  Clinical Points  CIS  ICE  PGY-1 Communication  EEC Scorea    0.02  0.24**  0.42**  0.17*  0.31**  Average NBME Subject Exams Score Across Clerkships      0.50**  0.12  0.29**  0.004  IM Clerkship Clinical Points        0.28**  0.37**  0.25**  CIS Score of Step 2 CS          0.30**  0.22*  ICE Score of Step 2 CS            0.21*  Communication Skills Component of PGY-1 Residency              Measure  EEC  Average NBME  Clinical Points  CIS  ICE  PGY-1 Communication  EEC Scorea    0.02  0.24**  0.42**  0.17*  0.31**  Average NBME Subject Exams Score Across Clerkships      0.50**  0.12  0.29**  0.004  IM Clerkship Clinical Points        0.28**  0.37**  0.25**  CIS Score of Step 2 CS          0.30**  0.22*  ICE Score of Step 2 CS            0.21*  Communication Skills Component of PGY-1 Residency              a According to a previously conducted but unpublished generalizability study at USU, the second-year OSCE stations demonstrated a moderate generalizability coefficient (r = 0.52). The EEC reliability estimate is higher, as a study by Joyce et al found the internal reliability coefficient (Cronbach's α) of standardized patient EEC scores to be 0.90.11 * p < 0.05; ** p < 0.01. View Large TABLE II. Bivariate Pearson Correlations Between the Measures Measure  EEC  Average NBME  Clinical Points  CIS  ICE  PGY-1 Communication  EEC Scorea    0.02  0.24**  0.42**  0.17*  0.31**  Average NBME Subject Exams Score Across Clerkships      0.50**  0.12  0.29**  0.004  IM Clerkship Clinical Points        0.28**  0.37**  0.25**  CIS Score of Step 2 CS          0.30**  0.22*  ICE Score of Step 2 CS            0.21*  Communication Skills Component of PGY-1 Residency              Measure  EEC  Average NBME  Clinical Points  CIS  ICE  PGY-1 Communication  EEC Scorea    0.02  0.24**  0.42**  0.17*  0.31**  Average NBME Subject Exams Score Across Clerkships      0.50**  0.12  0.29**  0.004  IM Clerkship Clinical Points        0.28**  0.37**  0.25**  CIS Score of Step 2 CS          0.30**  0.22*  ICE Score of Step 2 CS            0.21*  Communication Skills Component of PGY-1 Residency              a According to a previously conducted but unpublished generalizability study at USU, the second-year OSCE stations demonstrated a moderate generalizability coefficient (r = 0.52). The EEC reliability estimate is higher, as a study by Joyce et al found the internal reliability coefficient (Cronbach's α) of standardized patient EEC scores to be 0.90.11 * p < 0.05; ** p < 0.01. View Large The multiple linear regression modeling results for the 3 outcome measures are shown in Table III. The EEC score was a significant predictor of the CIS score and the PGY-1 communication skills score. Beyond the average NBME Subject Exams score and the IM clerkship clinical points, the EEC score explained 13% of the additional variance in the CIS score and 6% additional variance in the PGY-1 communication skills score. For the ICE score, 16% of the variance was accounted for by the average NBME Subject Exams score and the IM clerkship clinical points, while the EEC score explained only a modicum of additional variance (R2 change = 0.01). However, if we remove the EEC component score from the overall OSCE score, the remainder of the OSCE accounts for an additional 7% of the variance in the ICE score. TABLE III. Multiple Linear Regression Models of the Outcomes Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  R2 Change  CIS  ICE  PGY-1  CIS  ICE  PGY-1  CIS  ICE  PGY-1  Average NBME Subject Exams Score  0.004  0.02  −0.02  0.02  0.16  −0.14  0.08  0.16  0.08  IM Clerkship Clinical Points  0.01*  0.01**  0.01*  0.18  0.27  0.25        EEC score  3.87**  0.65  1.85**  0.37  0.09  0.25  0.13  0.01  0.06  Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  R2 Change  CIS  ICE  PGY-1  CIS  ICE  PGY-1  CIS  ICE  PGY-1  Average NBME Subject Exams Score  0.004  0.02  −0.02  0.02  0.16  −0.14  0.08  0.16  0.08  IM Clerkship Clinical Points  0.01*  0.01**  0.01*  0.18  0.27  0.25        EEC score  3.87**  0.65  1.85**  0.37  0.09  0.25  0.13  0.01  0.06  * p < 0.05; ** p < 0.01. View Large TABLE III. Multiple Linear Regression Models of the Outcomes Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  R2 Change  CIS  ICE  PGY-1  CIS  ICE  PGY-1  CIS  ICE  PGY-1  Average NBME Subject Exams Score  0.004  0.02  −0.02  0.02  0.16  −0.14  0.08  0.16  0.08  IM Clerkship Clinical Points  0.01*  0.01**  0.01*  0.18  0.27  0.25        EEC score  3.87**  0.65  1.85**  0.37  0.09  0.25  0.13  0.01  0.06  Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  R2 Change  CIS  ICE  PGY-1  CIS  ICE  PGY-1  CIS  ICE  PGY-1  Average NBME Subject Exams Score  0.004  0.02  −0.02  0.02  0.16  −0.14  0.08  0.16  0.08  IM Clerkship Clinical Points  0.01*  0.01**  0.01*  0.18  0.27  0.25        EEC score  3.87**  0.65  1.85**  0.37  0.09  0.25  0.13  0.01  0.06  * p < 0.05; ** p < 0.01. View Large DISCUSSION The aim of this study was to investigate the strength of the association between students' EEC OSCE performance at the end of the preclerkship period and later communication skills assessment and evaluation outcomes at both the UME and GME level. The results demonstrate that the EEC score is a strong predictor of the CIS score on Step 2 CS and a good predictor of the PGY-1 communication skills score. These findings provide fairly robust validity evidence for USU's EEC evaluation method. The use of a standardized approach with a communication tool such as the EEC can help inform future performance in communication skills independent of other education outcomes. In the context of a longitudinal curriculum, this information may better inform medical educators on learners' communication capabilities and more accurately direct future remediation efforts (there was approximately one year between EEC assessment and Step 2 CS Exam and another 18 months to PGY-1 PDs evaluation). As our results suggest, for the ICE score, the EEC was a poor predictor. However, the average NBME Subject Exams score, the IM clerkship clinical points, and the overall OSCE score without the EEC component score were significantly associated with this component. This finding indicates that clinical knowledge as measured by the Subject Exams and preceptor observations are better predictors of the ICE performance. To some extent, what EEC tested was different from the rest of the OSCE and this provides another piece of construct validity evidence for our EEC assessment. There are several important limitations of the present, single-institution study. In particular, since the reliabilities of the EEC, OSCE, IM clinical points, and PGY-1 factors do not reach the levels adequate for high-stakes decisions, the correlations between measures may have been weakened, thereby impacting our ability to accurately calculate the adjusted correlation coefficients. A larger sample size consisting of multiple classes of students would also increase the generalizability of the findings. In conclusion, our EEC assessment appears to be a good predictor of students' later performance on communication skills evaluation of Step 2 CS and the first year of residency. As such, this tool could be used as a sign of poor performance of communication skills as early as the start of the third-year clerkship where specific interventions can be effectively applied and tested before graduation from medical school. Future studies should focus on the longitudinal development within individual domains on the EEC, and the impact this may have future performance. APPENDIX REFERENCES 1. Schirmer JM, Mauksch L, Lang F, et al.   Assessing communication competence: a review of current tools. Fam Med  2005; 37: 184– 92. Google Scholar PubMed  2. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S General competencies and accreditation in graduate medical education. Health Aff (Milwood)  2002; 21: 103– 11. Google Scholar CrossRef Search ADS   3. Horowitz SD Evaluation of clinical competencies: basic certification, subspecialty certification, and recertification. Am J Phys Med Rehabil  2000; 79: 478– 80. Google Scholar CrossRef Search ADS PubMed  4. Institute of Medicine Improving Medical Education: Enhancing the Behavioral and Social Science Content of Medical School Curricula . Edited by Cuff PA, Vanselow Neal Washington, DC, National Academic Press, 2004. 5. Brown RF, Bylund CL Communication skills training: describing a new conceptual model. Acad Med  2008; 83: 37– 44. Google Scholar CrossRef Search ADS PubMed  6. Candib LM When the doctor takes off her clothes: reflections on being seen by patients in the exercise setting. Families, Systems, and Health  1999; 17: 349– 63. Google Scholar CrossRef Search ADS   7. Makoul G Essential elements of communication in medical encounters: the Kalamazoo consensus statement. Acad Med  2001; 76: 390– 3. Google Scholar CrossRef Search ADS PubMed  8. Aspegren K BEME guide no. 2: teaching and learning communications skills in medicine: a review with quality grading of articles. Med Teach  1999; 21: 563– 70. Google Scholar CrossRef Search ADS PubMed  9. Yedidia MJ, Gillespie CC, Kachur E, et al.   Effect of communications training on medical student performance. JAMA  2003; 290: 1157– 65. Google Scholar CrossRef Search ADS PubMed  10. Harik P, Clauser BE, Grabovsky I, Margolis MJ, Dillon GF, Boulet JR Relationships among subcomponents of the USMLE step 2 clinical skills examination, the step 1, and the step 2 clinical knowledge examinations. Acad Med  2006; 81( 10): S21– 4. Google Scholar CrossRef Search ADS PubMed  11. Joyce BL, Steenbergh T, Scher E Use of the kalamazoo essential elements communication checklist (adapted) in an institutional interpersonal and communication skills curriculum. J Grad Med Educ  2010; 2: 165– 9. Google Scholar CrossRef Search ADS PubMed  12. USMLE Step 2 CS. Official website of USMLE . Available at http://www.usmle.org/step-2-cs/; accessed July 14, 2014. 13. Dong T, Durning SJ, Gilliland W, Swygert K, Artino AR Development and initial validation of a program director's evaluation form for medical school graduates. Mil Med  2015; 180( 4 Suppl): 97– 103. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.

USN, Henry L. Phillips, IV, MSC;PhD, Ting Dong,;PhD, Steven J. Durning, MD,;USN, Anthony R. Artino, Jr., MSC

doi: 10.7205/MILMED-D-14-00557pmid: 25850124

ABSTRACT Recent research in medical education suggests that students' motivational beliefs, such as their beliefs about the importance of a task, and their emotions are meaningful predictors of learning and performance. The primary purpose of this study was to develop a self-report measure of “task importance” and “anxiety” in relation to several medical education competencies and to collect validity evidence for the new measures. The secondary purpose was to evaluate differences in these measures by year of medical school. Exploratory factor analysis of scores from 368 medical school students suggested two task importance factors and three anxiety factors. The task importance and anxiety subscales were weakly related to each other and exhibited consistently negative and positive correlations, respectively, with three self-efficacy subscales. The task importance subscales were positively related to “metacognition,” whereas “interpersonal skills anxiety” and “health knowledge anxiety” were positively related to “procrastination.” All three anxiety factors were positively related to “avoidance of help seeking,” whereas “interpersonal skills and professionalism importance” was negatively related to help avoidance behaviors. Finally, comparisons across the 4 years of medical school indicated that some aspects of task importance and anxiety varied significantly. Overall, findings from this study provide validity evidence for the psychometric quality of these scales, which capture task importance and anxiety in medical students. Limitations and implications for medical education research are discussed. INTRODUCTION In recent years, more attention has been given to the influence of affective factors, such as motivational beliefs and emotions, on academic outcomes in educational research.1,2 Findings from a host of contemporary education studies suggest that student affect explains considerable variance in academic outcomes, above and beyond the effects of cognitive factors such as aptitude and prior achievement.3,–5 Based, in part, on his own empirical work, Pekrun6 proposed a theoretical model of affect to help explain these findings. Pekrun's control-value theory of achievement emotions4 depicts a reciprocal relationship between the learning environment, personal factors (such as cognition, motivation, and emotion), and academic outcomes. Pekrun proposed that the learning environment influences students' appraisals of the “control” they have over their learning environment, as well as the “value” they attach to tasks and activities in that environment. This value appraisal dictates the subjective importance attached to the achievement of academic goals. A perceived lack of control, combined with a low degree of task importance, can result in maladaptive emotions, avoidance of help seeking, and procrastination, among other negative outcomes, all of which can contribute to low academic achievement. Conversely, a high degree of control combined with high levels of task importance can lead to adaptive emotions, higher levels of achievement motivation, and greater self-efficacy. Other researchers have examined motivational beliefs, such as task value and self-efficacy, as predictors of discrete achievement emotions, such as enjoyment and anxiety.7 These achievement emotions then influence various academic outcomes, such as course grades, continuing motivation to learn,8,9 and the use of adaptive learning strategies (such as metacognition10) or maladaptive learning strategies (such as procrastination11 and avoidance of help seeking12). Within medical education, several scholars have recently called for more research into the role of motivational beliefs and emotions in medical training.5,9,13 To date, however, empirical studies to directly examine theoretical models of motivation and emotion, such as Pekrun's control-value theory, have been scarce in medical education. In one recent study, Artino et al2 explored the complex interplay between the learning environment, personal factors, and academic outcomes using a sample of second-year medical students in a year-long clinical reasoning course. The authors investigated two types of motivational beliefs. The first was task value, which has been defined as the degree to which individuals find a task interesting, important, and useful.14 The second motivational belief was self-efficacy, which has been defined as students' judgments of their capabilities to successfully perform specific academic tasks.15 Results from this work revealed that task value and self-efficacy beliefs accounted for considerable variance in students' achievement outcomes. The primary purpose of this study was to develop a quantitative self-report measure of task value beliefs (in particular, “task importance,” a specific type of task value) and “anxiety,” an essential negative emotion, in relation to the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME). Because we also sought to compare task importance and anxiety at different phases of medical education, we chose to develop initial survey items based on these six core competencies, which students are expected to achieve by the end of residency training. Our target population was medical students at various stages of undergraduate medical education at one institution. Once developed, we gathered two types of validity evidence for the survey results: (1) evidence based on internal structure and (2) evidence based on relations to other variables relevant to the control-value theory framework.4 A priori to any evaluations of dimensionality, we posited the following hypotheses based on the theoretical underpinnings of control-value theory: (H1) Task importance factors will be positively correlated with two adaptive factors (self-efficacy and metacognition), and negatively correlated with two maladaptive factors (procrastination and avoidance of help seeking). (H2) Anxiety factors will be negatively correlated with two adaptive factors (self-efficacy and metacognition), and positively correlated with two maladaptive factors (procrastination and avoidance of help seeking). The secondary purpose of the current study was to explore potential differences in task importance and anxiety from year 1 of medical school to year 4. Based on the findings of Artino et al16 in their development of several self-efficacy scales, we had the following hypotheses: (H3) Task importance will be higher among more senior students. (H4) Anxiety will be lower among more senior students. As medical students progress through the curriculum, they develop greater efficacy (e.g., Artino et al16 found seniority related to higher scores on “patient care self-efficacy” and “evidence-based medicine self-efficacy”), and in turn these students are expected to increase the degree of importance they attach to the ACGME competencies, upon which scale items were based. By the same token, progress through the medical school curriculum and development as a practitioner are expected to result in less anxiety as students' confidence in attaining the ACGME competencies increases. Similarly, as students develop greater skill and a deeper understanding of their roles as caregivers, it is expected that they will attach greater importance to, and perceive greater value in, the goals of their academic programs. More senior students are expected to better understand why their assigned academic and practitioner goals are structured as they are, and are more likely to agree with those goals as experience broadens their perspectives. METHODS Study Context This study was conducted at the F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences (USU) under the auspices of the Long-Term Career Outcome Study. At the time of the study, USU matriculated roughly 170 medical students per year and offered a traditional 4-year curriculum, structured as 2 years of basic sciences coursework followed by 2 years of clinical rotations (clerkships). Item Development and Content Validation Initial survey items were written to align with the content of the six core competencies defined by the ACGME17: (1) patient care, (2) medical and population health knowledge, (3) interpersonal and communication skills, (4) practice-based learning and improvement (i.e., understanding practice and ways to improve practice, such as through quality assurance), (5) professionalism, and (6) systems-based practice (i.e., understanding issues of access and use of resources in health care17). Since one of our goals was to evaluate medical students' motivational beliefs and emotions as they related to their medical knowledge and skills at different time points in their medical school education, we developed items targeting the ACGME competencies, which students are expected to achieve by the end of residency training. It is worth noting that at the time of this study, USU's learning objectives for all medical students were structured around the six ACGME core competencies. After developing a library of draft items, we recruited five senior physician-educators, each with more than 10 years of experience in medical education, to provide content validity evaluations. Using a standardized content validation form,18 we individually asked each physician-educator to indicate the degree to which the items addressed the targeted constructs. In addition, each physician-educator was asked to provide editorial recommendations for item improvement or elimination. The result of this effort was a list of 19 behavioral items related to the six ACGME core competencies. Next, the 19 items were organized into a survey instrument, which asked respondents to rate their perceptions of self-efficacy, task importance, and anxiety as they related to each of the 19 ACGME-framed items (for a total of 57 items). Details of validation of the self-efficacy subscale results, using a smaller subsample, are provided in the work of Artino et al.16 Lists of the task importance and anxiety items are presented in Tables I and II. TABLE I. Results for the Task Importance Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; λ = 0; N = 368) Item  Factors  C  1  2  3  4  5  6  IMP-1 Apply knowledge of normal function to each of the major organ systems?  0.55  0.31 (0.54)  −0.04 (−0.03)  0.46 (0.66)  −0.10 (−0.12)  −0.12 (−0.15)  −0.17 (−0.28)  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.77  0.79 (0.84)  −0.04 (0.03)  0.06 (0.47)  −0.03 (−0.04)  −0.03 (−0.02)  −0.23 (−0.33)  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.59  0.62 (0.73)  0.03 (0.07)  0.21 (0.52)  −0.04 (−0.05)  −0.02 (−0.05)  −0.07 (−0.18)  IMP-4 Use effective listening skills when interacting with a patient?  0.77  −0.08 (0.35)  0.01 (−0.02)  0.89 (0.87)  0.00 (−0.10)  −0.03 (−0.19)  −0.09 (−0.21)  IMP-5 Demonstrate caring when counseling a patient?  0.7  −0.10 (0.32)  0.03 (−0.01)  0.89 (0.83)  0.05 (−0.08)  0.03 (−0.14)  0.05 (−0.07)  IMP-6 Accurately gather essential information from a patient?  0.77  0.11 (0.48)  −0.02 (−0.04)  0.74 (0.83)  −0.01 (−0.11)  0.00 (−0.14)  −0.27 (−0.39)  IMP-7 Perform a thorough physical exam?  0.54  0.18 (0.47)  0.00 (−0.05)  0.57 (0.69)  −0.07 (−0.16)  0.03 (−0.05)  −0.20 (−0.29)  IMP-8 Develop an appropriate differential diagnosis?  0.77  0.64 (0.75)  −0.09 (−0.03)  0.14 (0.50)  0.03 (−0.03)  0.02 (−0.02)  −0.43 (−0.52)  IMP-9 Generate a patient-specific treatment plan?  0.84  0.87 (0.86)  −0.10 (−0.01)  −0.07 (0.38)  0.00 (−0.03)  0.00 (0.03)  −0.30 (−0.39)  IMP-10 Use information technology to support patient-care decisions?  0.65  0.76 (0.80)  −0.06 (0.03)  0.05 (0.40)  0.11 (0.05)  0.07 (0.04)  −0.10 (−0.19)  IMP-11 Work effectively with other health care professionals to provide high-quality patient care?  0.6  0.50 (0.69)  0.04 (0.06)  0.37 (0.62)  −0.04 (−0.07)  −0.01 (−0.07)  −0.10 (−0.21)  IMP-12 Improve clinical practice using a systematic approach?  0.66  0.74 (0.80)  0.07 (0.12)  0.10 (0.46)  −0.01 (0.00)  0.01 (−0.02)  −0.06 (−0.16)  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  0.72  0.81 (0.85)  0.01 (0.08)  0.09 (0.45)  0.03 (0.00)  0.02 (0.02)  0.06 (−0.05)  IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.67  0.79 (0.80)  0.06 (0.11)  0.06 (0.40)  0.01 (−0.01)  0.05 (0.05)  0.15 (0.05)  IMP-15 Demonstrate sensitivity to patients' cultural differences?  0.5  0.09 (0.37)  0.02 (0.00)  0.66 (0.66)  0.02 (−0.07)  −0.01 (−0.10)  0.24 (0.14)  IMP-16 Balance professional responsibilities with personal responsibilities?  0.35  0.05 (0.29)  −0.05 (−0.08)  0.56 (0.57)  −0.05 (−0.14)  −0.03 (−0.06)  0.14 (0.05)  IMP-17 Discuss methods of controlling health care costs?  0.82  0.90 (0.83)  0.06 (0.16)  −0.08 (0.29)  0.05 (0.07)  −0.01 (0.02)  0.34 (0.24)  IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?  0.79  0.92 (0.86)  0.02 (0.12)  −0.07 (0.33)  0.02 (0.04)  −0.03 (0.01)  0.22 (0.11)  IMP-19 Apply high-quality health care in deployed military environments?  0.59  0.78 (0.76)  0.04 (0.11)  −0.03 (0.35)  −0.06 (−0.01)  −0.07 (−0.03)  0.01 (−0.09)  Item  Factors  C  1  2  3  4  5  6  IMP-1 Apply knowledge of normal function to each of the major organ systems?  0.55  0.31 (0.54)  −0.04 (−0.03)  0.46 (0.66)  −0.10 (−0.12)  −0.12 (−0.15)  −0.17 (−0.28)  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.77  0.79 (0.84)  −0.04 (0.03)  0.06 (0.47)  −0.03 (−0.04)  −0.03 (−0.02)  −0.23 (−0.33)  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.59  0.62 (0.73)  0.03 (0.07)  0.21 (0.52)  −0.04 (−0.05)  −0.02 (−0.05)  −0.07 (−0.18)  IMP-4 Use effective listening skills when interacting with a patient?  0.77  −0.08 (0.35)  0.01 (−0.02)  0.89 (0.87)  0.00 (−0.10)  −0.03 (−0.19)  −0.09 (−0.21)  IMP-5 Demonstrate caring when counseling a patient?  0.7  −0.10 (0.32)  0.03 (−0.01)  0.89 (0.83)  0.05 (−0.08)  0.03 (−0.14)  0.05 (−0.07)  IMP-6 Accurately gather essential information from a patient?  0.77  0.11 (0.48)  −0.02 (−0.04)  0.74 (0.83)  −0.01 (−0.11)  0.00 (−0.14)  −0.27 (−0.39)  IMP-7 Perform a thorough physical exam?  0.54  0.18 (0.47)  0.00 (−0.05)  0.57 (0.69)  −0.07 (−0.16)  0.03 (−0.05)  −0.20 (−0.29)  IMP-8 Develop an appropriate differential diagnosis?  0.77  0.64 (0.75)  −0.09 (−0.03)  0.14 (0.50)  0.03 (−0.03)  0.02 (−0.02)  −0.43 (−0.52)  IMP-9 Generate a patient-specific treatment plan?  0.84  0.87 (0.86)  −0.10 (−0.01)  −0.07 (0.38)  0.00 (−0.03)  0.00 (0.03)  −0.30 (−0.39)  IMP-10 Use information technology to support patient-care decisions?  0.65  0.76 (0.80)  −0.06 (0.03)  0.05 (0.40)  0.11 (0.05)  0.07 (0.04)  −0.10 (−0.19)  IMP-11 Work effectively with other health care professionals to provide high-quality patient care?  0.6  0.50 (0.69)  0.04 (0.06)  0.37 (0.62)  −0.04 (−0.07)  −0.01 (−0.07)  −0.10 (−0.21)  IMP-12 Improve clinical practice using a systematic approach?  0.66  0.74 (0.80)  0.07 (0.12)  0.10 (0.46)  −0.01 (0.00)  0.01 (−0.02)  −0.06 (−0.16)  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  0.72  0.81 (0.85)  0.01 (0.08)  0.09 (0.45)  0.03 (0.00)  0.02 (0.02)  0.06 (−0.05)  IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.67  0.79 (0.80)  0.06 (0.11)  0.06 (0.40)  0.01 (−0.01)  0.05 (0.05)  0.15 (0.05)  IMP-15 Demonstrate sensitivity to patients' cultural differences?  0.5  0.09 (0.37)  0.02 (0.00)  0.66 (0.66)  0.02 (−0.07)  −0.01 (−0.10)  0.24 (0.14)  IMP-16 Balance professional responsibilities with personal responsibilities?  0.35  0.05 (0.29)  −0.05 (−0.08)  0.56 (0.57)  −0.05 (−0.14)  −0.03 (−0.06)  0.14 (0.05)  IMP-17 Discuss methods of controlling health care costs?  0.82  0.90 (0.83)  0.06 (0.16)  −0.08 (0.29)  0.05 (0.07)  −0.01 (0.02)  0.34 (0.24)  IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?  0.79  0.92 (0.86)  0.02 (0.12)  −0.07 (0.33)  0.02 (0.04)  −0.03 (0.01)  0.22 (0.11)  IMP-19 Apply high-quality health care in deployed military environments?  0.59  0.78 (0.76)  0.04 (0.11)  −0.03 (0.35)  −0.06 (−0.01)  −0.07 (−0.03)  0.01 (−0.09)  C, Communality; Imp, Task Importance. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) ≥0.30 on only one factor, and communalities ≥0.40. Results presented above reflect loadings and communalities for Task Importance items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously. View Large TABLE I. Results for the Task Importance Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; λ = 0; N = 368) Item  Factors  C  1  2  3  4  5  6  IMP-1 Apply knowledge of normal function to each of the major organ systems?  0.55  0.31 (0.54)  −0.04 (−0.03)  0.46 (0.66)  −0.10 (−0.12)  −0.12 (−0.15)  −0.17 (−0.28)  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.77  0.79 (0.84)  −0.04 (0.03)  0.06 (0.47)  −0.03 (−0.04)  −0.03 (−0.02)  −0.23 (−0.33)  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.59  0.62 (0.73)  0.03 (0.07)  0.21 (0.52)  −0.04 (−0.05)  −0.02 (−0.05)  −0.07 (−0.18)  IMP-4 Use effective listening skills when interacting with a patient?  0.77  −0.08 (0.35)  0.01 (−0.02)  0.89 (0.87)  0.00 (−0.10)  −0.03 (−0.19)  −0.09 (−0.21)  IMP-5 Demonstrate caring when counseling a patient?  0.7  −0.10 (0.32)  0.03 (−0.01)  0.89 (0.83)  0.05 (−0.08)  0.03 (−0.14)  0.05 (−0.07)  IMP-6 Accurately gather essential information from a patient?  0.77  0.11 (0.48)  −0.02 (−0.04)  0.74 (0.83)  −0.01 (−0.11)  0.00 (−0.14)  −0.27 (−0.39)  IMP-7 Perform a thorough physical exam?  0.54  0.18 (0.47)  0.00 (−0.05)  0.57 (0.69)  −0.07 (−0.16)  0.03 (−0.05)  −0.20 (−0.29)  IMP-8 Develop an appropriate differential diagnosis?  0.77  0.64 (0.75)  −0.09 (−0.03)  0.14 (0.50)  0.03 (−0.03)  0.02 (−0.02)  −0.43 (−0.52)  IMP-9 Generate a patient-specific treatment plan?  0.84  0.87 (0.86)  −0.10 (−0.01)  −0.07 (0.38)  0.00 (−0.03)  0.00 (0.03)  −0.30 (−0.39)  IMP-10 Use information technology to support patient-care decisions?  0.65  0.76 (0.80)  −0.06 (0.03)  0.05 (0.40)  0.11 (0.05)  0.07 (0.04)  −0.10 (−0.19)  IMP-11 Work effectively with other health care professionals to provide high-quality patient care?  0.6  0.50 (0.69)  0.04 (0.06)  0.37 (0.62)  −0.04 (−0.07)  −0.01 (−0.07)  −0.10 (−0.21)  IMP-12 Improve clinical practice using a systematic approach?  0.66  0.74 (0.80)  0.07 (0.12)  0.10 (0.46)  −0.01 (0.00)  0.01 (−0.02)  −0.06 (−0.16)  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  0.72  0.81 (0.85)  0.01 (0.08)  0.09 (0.45)  0.03 (0.00)  0.02 (0.02)  0.06 (−0.05)  IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.67  0.79 (0.80)  0.06 (0.11)  0.06 (0.40)  0.01 (−0.01)  0.05 (0.05)  0.15 (0.05)  IMP-15 Demonstrate sensitivity to patients' cultural differences?  0.5  0.09 (0.37)  0.02 (0.00)  0.66 (0.66)  0.02 (−0.07)  −0.01 (−0.10)  0.24 (0.14)  IMP-16 Balance professional responsibilities with personal responsibilities?  0.35  0.05 (0.29)  −0.05 (−0.08)  0.56 (0.57)  −0.05 (−0.14)  −0.03 (−0.06)  0.14 (0.05)  IMP-17 Discuss methods of controlling health care costs?  0.82  0.90 (0.83)  0.06 (0.16)  −0.08 (0.29)  0.05 (0.07)  −0.01 (0.02)  0.34 (0.24)  IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?  0.79  0.92 (0.86)  0.02 (0.12)  −0.07 (0.33)  0.02 (0.04)  −0.03 (0.01)  0.22 (0.11)  IMP-19 Apply high-quality health care in deployed military environments?  0.59  0.78 (0.76)  0.04 (0.11)  −0.03 (0.35)  −0.06 (−0.01)  −0.07 (−0.03)  0.01 (−0.09)  Item  Factors  C  1  2  3  4  5  6  IMP-1 Apply knowledge of normal function to each of the major organ systems?  0.55  0.31 (0.54)  −0.04 (−0.03)  0.46 (0.66)  −0.10 (−0.12)  −0.12 (−0.15)  −0.17 (−0.28)  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.77  0.79 (0.84)  −0.04 (0.03)  0.06 (0.47)  −0.03 (−0.04)  −0.03 (−0.02)  −0.23 (−0.33)  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.59  0.62 (0.73)  0.03 (0.07)  0.21 (0.52)  −0.04 (−0.05)  −0.02 (−0.05)  −0.07 (−0.18)  IMP-4 Use effective listening skills when interacting with a patient?  0.77  −0.08 (0.35)  0.01 (−0.02)  0.89 (0.87)  0.00 (−0.10)  −0.03 (−0.19)  −0.09 (−0.21)  IMP-5 Demonstrate caring when counseling a patient?  0.7  −0.10 (0.32)  0.03 (−0.01)  0.89 (0.83)  0.05 (−0.08)  0.03 (−0.14)  0.05 (−0.07)  IMP-6 Accurately gather essential information from a patient?  0.77  0.11 (0.48)  −0.02 (−0.04)  0.74 (0.83)  −0.01 (−0.11)  0.00 (−0.14)  −0.27 (−0.39)  IMP-7 Perform a thorough physical exam?  0.54  0.18 (0.47)  0.00 (−0.05)  0.57 (0.69)  −0.07 (−0.16)  0.03 (−0.05)  −0.20 (−0.29)  IMP-8 Develop an appropriate differential diagnosis?  0.77  0.64 (0.75)  −0.09 (−0.03)  0.14 (0.50)  0.03 (−0.03)  0.02 (−0.02)  −0.43 (−0.52)  IMP-9 Generate a patient-specific treatment plan?  0.84  0.87 (0.86)  −0.10 (−0.01)  −0.07 (0.38)  0.00 (−0.03)  0.00 (0.03)  −0.30 (−0.39)  IMP-10 Use information technology to support patient-care decisions?  0.65  0.76 (0.80)  −0.06 (0.03)  0.05 (0.40)  0.11 (0.05)  0.07 (0.04)  −0.10 (−0.19)  IMP-11 Work effectively with other health care professionals to provide high-quality patient care?  0.6  0.50 (0.69)  0.04 (0.06)  0.37 (0.62)  −0.04 (−0.07)  −0.01 (−0.07)  −0.10 (−0.21)  IMP-12 Improve clinical practice using a systematic approach?  0.66  0.74 (0.80)  0.07 (0.12)  0.10 (0.46)  −0.01 (0.00)  0.01 (−0.02)  −0.06 (−0.16)  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  0.72  0.81 (0.85)  0.01 (0.08)  0.09 (0.45)  0.03 (0.00)  0.02 (0.02)  0.06 (−0.05)  IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.67  0.79 (0.80)  0.06 (0.11)  0.06 (0.40)  0.01 (−0.01)  0.05 (0.05)  0.15 (0.05)  IMP-15 Demonstrate sensitivity to patients' cultural differences?  0.5  0.09 (0.37)  0.02 (0.00)  0.66 (0.66)  0.02 (−0.07)  −0.01 (−0.10)  0.24 (0.14)  IMP-16 Balance professional responsibilities with personal responsibilities?  0.35  0.05 (0.29)  −0.05 (−0.08)  0.56 (0.57)  −0.05 (−0.14)  −0.03 (−0.06)  0.14 (0.05)  IMP-17 Discuss methods of controlling health care costs?  0.82  0.90 (0.83)  0.06 (0.16)  −0.08 (0.29)  0.05 (0.07)  −0.01 (0.02)  0.34 (0.24)  IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?  0.79  0.92 (0.86)  0.02 (0.12)  −0.07 (0.33)  0.02 (0.04)  −0.03 (0.01)  0.22 (0.11)  IMP-19 Apply high-quality health care in deployed military environments?  0.59  0.78 (0.76)  0.04 (0.11)  −0.03 (0.35)  −0.06 (−0.01)  −0.07 (−0.03)  0.01 (−0.09)  C, Communality; Imp, Task Importance. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) ≥0.30 on only one factor, and communalities ≥0.40. Results presented above reflect loadings and communalities for Task Importance items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously. View Large TABLE II. Results for the Anxiety Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; delta = 0; N = 368) Item  Factors  C  1  2  3  4  5  6  ANX-1 Applying knowledge of normal function to each of the major organ systems?  0.51  −0.02 (−0.05)  0.06 (0.37)  −0.03 (0.02)  0.18 (0.46)  −0.59 (−0.69)  0.00 (−0.08)  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.55  0.01 (0.02)  0.08 (0.35)  0.05 (0.16)  0.01 (0.34)  −0.69 (−0.73)  0.02 (−0.08)  ANX-3 Applying knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.48  0.01 (0.02)  v0.31 (0.53)  0.00 (0.04)  0.09 (0.41)  −0.46 (−0.62)  0.02 (−0.05)  ANX-4 Using effective listening skills when interacting with a patient?  0.81  0.01 (0.01)  0.10 (0.44)  −0.01 (−0.15)  0.88 (0.89)  0.07 (−0.35)  0.03 (0.01)  ANX-5 Demonstrating caring when counseling a patient?  0.79  0.04 (0.04)  0.11 (0.42)  −0.01 (−0.15)  0.90 (0.88)  0.15 (−0.28)  0.01 (−0.01)  ANX-6 Accurately gathering essential information from a patient?  0.70  0.01 (−0.02)  −0.11 (0.30)  −0.01 (−0.06)  0.71 (0.79)  −0.31 (−0.56)  0.07 (0.01)  ANX-7 Performing a thorough physical exam?  0.50  −0.01 (−0.03)  −0.15 (0.22)  0.00 (0.00)  0.51 (0.62)  −0.39 (−0.55)  0.01 (−0.05)  ANX-8 Developing an appropriate differential diagnosis?  0.76  0.02 (−0.01)  0.00 (0.35)  0.00 (0.14)  −0.01 (0.37)  −0.88 (−0.87)  0.04 (−0.07)  ANX-9 Generating a patient-specific treatment plan?  0.67  0.02 (0.06)  0.18 (0.44)  0.06 (0.21)  −0.11 (0.28)  −0.76 (−0.80)  −0.04 (−0.15)  ANX-10 Using information technology to support patient-care decisions?  0.52  −0.09 (−0.03)  0.59 (0.68)  0.02 (−0.01)  0.02 (0.36)  −0.23 (−0.48)  0.00 (−0.03)  ANX-11 Working effectively with other health care professionals to provide high-quality patient care?  0.50  −0.06 (−0.02)  0.39 (0.58)  −0.01 (−0.07)  0.37 (0.58)  −0.09 (−0.42)  −0.14 (−0.16)  ANX-12 Improving clinical practice using a systematic approach?  0.64  −0.07 (0.01)  0.62 (0.75)  0.03 (−0.02)  0.19 (0.50)  −0.14 (−0.49)  −0.08 (−0.11)  ANX-13 Evaluating evidence from scientific studies relevant to your patients' health problems?  0.66  0.03 (0.10)  0.63 (0.72)  −0.05 (−0.01)  0.07 (0.41)  −0.13 (−0.45)  −0.33 (−0.36)  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.61  0.08 (0.16)  0.62 (0.69)  −0.03 (0.05)  0.01 (0.34)  −0.15 (−0.43)  −0.30 (−0.33)  ANX-15 Demonstrating sensitivity to patients' cultural differences?  0.61  0.10 (0.08)  0.28 (0.53)  −0.10 (−0.16)  0.63 (0.73)  0.07 (−0.30)  −0.05 (−0.07)  ANX-16 Balancing professional responsibilities with personal responsibilities?  0.31  −0.09 (−0.06)  −0.02 (0.23)  0.07 (0.04)  0.34 (0.45)  −0.29 (−0.46)  −0.11 (−0.16)  ANX-17 Discussing methods of controlling health care costs?  0.70  0.05 (0.10)  0.86 (0.81)  −0.02 (−0.07)  0.00 (0.30)  0.13 (−0.19)  0.13 (0.13)  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?  0.70  0.03 (0.08)  0.79 (0.81)  0.00 (−0.05)  0.05 (0.37)  0.01 (−0.31)  0.18 (0.16)  ANX-19 Applying high-quality health care in deployed military environments?  0.37  0.04 (0.08)  0.48 (0.57)  0.02 (0.03)  0.04 (0.32)  −0.19 (−0.40)  0.06 (0.02)  Item  Factors  C  1  2  3  4  5  6  ANX-1 Applying knowledge of normal function to each of the major organ systems?  0.51  −0.02 (−0.05)  0.06 (0.37)  −0.03 (0.02)  0.18 (0.46)  −0.59 (−0.69)  0.00 (−0.08)  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.55  0.01 (0.02)  0.08 (0.35)  0.05 (0.16)  0.01 (0.34)  −0.69 (−0.73)  0.02 (−0.08)  ANX-3 Applying knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.48  0.01 (0.02)  v0.31 (0.53)  0.00 (0.04)  0.09 (0.41)  −0.46 (−0.62)  0.02 (−0.05)  ANX-4 Using effective listening skills when interacting with a patient?  0.81  0.01 (0.01)  0.10 (0.44)  −0.01 (−0.15)  0.88 (0.89)  0.07 (−0.35)  0.03 (0.01)  ANX-5 Demonstrating caring when counseling a patient?  0.79  0.04 (0.04)  0.11 (0.42)  −0.01 (−0.15)  0.90 (0.88)  0.15 (−0.28)  0.01 (−0.01)  ANX-6 Accurately gathering essential information from a patient?  0.70  0.01 (−0.02)  −0.11 (0.30)  −0.01 (−0.06)  0.71 (0.79)  −0.31 (−0.56)  0.07 (0.01)  ANX-7 Performing a thorough physical exam?  0.50  −0.01 (−0.03)  −0.15 (0.22)  0.00 (0.00)  0.51 (0.62)  −0.39 (−0.55)  0.01 (−0.05)  ANX-8 Developing an appropriate differential diagnosis?  0.76  0.02 (−0.01)  0.00 (0.35)  0.00 (0.14)  −0.01 (0.37)  −0.88 (−0.87)  0.04 (−0.07)  ANX-9 Generating a patient-specific treatment plan?  0.67  0.02 (0.06)  0.18 (0.44)  0.06 (0.21)  −0.11 (0.28)  −0.76 (−0.80)  −0.04 (−0.15)  ANX-10 Using information technology to support patient-care decisions?  0.52  −0.09 (−0.03)  0.59 (0.68)  0.02 (−0.01)  0.02 (0.36)  −0.23 (−0.48)  0.00 (−0.03)  ANX-11 Working effectively with other health care professionals to provide high-quality patient care?  0.50  −0.06 (−0.02)  0.39 (0.58)  −0.01 (−0.07)  0.37 (0.58)  −0.09 (−0.42)  −0.14 (−0.16)  ANX-12 Improving clinical practice using a systematic approach?  0.64  −0.07 (0.01)  0.62 (0.75)  0.03 (−0.02)  0.19 (0.50)  −0.14 (−0.49)  −0.08 (−0.11)  ANX-13 Evaluating evidence from scientific studies relevant to your patients' health problems?  0.66  0.03 (0.10)  0.63 (0.72)  −0.05 (−0.01)  0.07 (0.41)  −0.13 (−0.45)  −0.33 (−0.36)  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.61  0.08 (0.16)  0.62 (0.69)  −0.03 (0.05)  0.01 (0.34)  −0.15 (−0.43)  −0.30 (−0.33)  ANX-15 Demonstrating sensitivity to patients' cultural differences?  0.61  0.10 (0.08)  0.28 (0.53)  −0.10 (−0.16)  0.63 (0.73)  0.07 (−0.30)  −0.05 (−0.07)  ANX-16 Balancing professional responsibilities with personal responsibilities?  0.31  −0.09 (−0.06)  −0.02 (0.23)  0.07 (0.04)  0.34 (0.45)  −0.29 (−0.46)  −0.11 (−0.16)  ANX-17 Discussing methods of controlling health care costs?  0.70  0.05 (0.10)  0.86 (0.81)  −0.02 (−0.07)  0.00 (0.30)  0.13 (−0.19)  0.13 (0.13)  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?  0.70  0.03 (0.08)  0.79 (0.81)  0.00 (−0.05)  0.05 (0.37)  0.01 (−0.31)  0.18 (0.16)  ANX-19 Applying high-quality health care in deployed military environments?  0.37  0.04 (0.08)  0.48 (0.57)  0.02 (0.03)  0.04 (0.32)  −0.19 (−0.40)  0.06 (0.02)  Anx, Anxiety; C, Communality. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) ≥0.30 on only one factor, and communalities ≥0.40. Results presented above reflect loadings and communalities for Anxiety items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously. Results continued from Table I. View Large TABLE II. Results for the Anxiety Items from the Six-Factor Principal Axis Factor Analysis Solution with Oblique Rotation (Oblimin; delta = 0; N = 368) Item  Factors  C  1  2  3  4  5  6  ANX-1 Applying knowledge of normal function to each of the major organ systems?  0.51  −0.02 (−0.05)  0.06 (0.37)  −0.03 (0.02)  0.18 (0.46)  −0.59 (−0.69)  0.00 (−0.08)  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.55  0.01 (0.02)  0.08 (0.35)  0.05 (0.16)  0.01 (0.34)  −0.69 (−0.73)  0.02 (−0.08)  ANX-3 Applying knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.48  0.01 (0.02)  v0.31 (0.53)  0.00 (0.04)  0.09 (0.41)  −0.46 (−0.62)  0.02 (−0.05)  ANX-4 Using effective listening skills when interacting with a patient?  0.81  0.01 (0.01)  0.10 (0.44)  −0.01 (−0.15)  0.88 (0.89)  0.07 (−0.35)  0.03 (0.01)  ANX-5 Demonstrating caring when counseling a patient?  0.79  0.04 (0.04)  0.11 (0.42)  −0.01 (−0.15)  0.90 (0.88)  0.15 (−0.28)  0.01 (−0.01)  ANX-6 Accurately gathering essential information from a patient?  0.70  0.01 (−0.02)  −0.11 (0.30)  −0.01 (−0.06)  0.71 (0.79)  −0.31 (−0.56)  0.07 (0.01)  ANX-7 Performing a thorough physical exam?  0.50  −0.01 (−0.03)  −0.15 (0.22)  0.00 (0.00)  0.51 (0.62)  −0.39 (−0.55)  0.01 (−0.05)  ANX-8 Developing an appropriate differential diagnosis?  0.76  0.02 (−0.01)  0.00 (0.35)  0.00 (0.14)  −0.01 (0.37)  −0.88 (−0.87)  0.04 (−0.07)  ANX-9 Generating a patient-specific treatment plan?  0.67  0.02 (0.06)  0.18 (0.44)  0.06 (0.21)  −0.11 (0.28)  −0.76 (−0.80)  −0.04 (−0.15)  ANX-10 Using information technology to support patient-care decisions?  0.52  −0.09 (−0.03)  0.59 (0.68)  0.02 (−0.01)  0.02 (0.36)  −0.23 (−0.48)  0.00 (−0.03)  ANX-11 Working effectively with other health care professionals to provide high-quality patient care?  0.50  −0.06 (−0.02)  0.39 (0.58)  −0.01 (−0.07)  0.37 (0.58)  −0.09 (−0.42)  −0.14 (−0.16)  ANX-12 Improving clinical practice using a systematic approach?  0.64  −0.07 (0.01)  0.62 (0.75)  0.03 (−0.02)  0.19 (0.50)  −0.14 (−0.49)  −0.08 (−0.11)  ANX-13 Evaluating evidence from scientific studies relevant to your patients' health problems?  0.66  0.03 (0.10)  0.63 (0.72)  −0.05 (−0.01)  0.07 (0.41)  −0.13 (−0.45)  −0.33 (−0.36)  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.61  0.08 (0.16)  0.62 (0.69)  −0.03 (0.05)  0.01 (0.34)  −0.15 (−0.43)  −0.30 (−0.33)  ANX-15 Demonstrating sensitivity to patients' cultural differences?  0.61  0.10 (0.08)  0.28 (0.53)  −0.10 (−0.16)  0.63 (0.73)  0.07 (−0.30)  −0.05 (−0.07)  ANX-16 Balancing professional responsibilities with personal responsibilities?  0.31  −0.09 (−0.06)  −0.02 (0.23)  0.07 (0.04)  0.34 (0.45)  −0.29 (−0.46)  −0.11 (−0.16)  ANX-17 Discussing methods of controlling health care costs?  0.70  0.05 (0.10)  0.86 (0.81)  −0.02 (−0.07)  0.00 (0.30)  0.13 (−0.19)  0.13 (0.13)  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?  0.70  0.03 (0.08)  0.79 (0.81)  0.00 (−0.05)  0.05 (0.37)  0.01 (−0.31)  0.18 (0.16)  ANX-19 Applying high-quality health care in deployed military environments?  0.37  0.04 (0.08)  0.48 (0.57)  0.02 (0.03)  0.04 (0.32)  −0.19 (−0.40)  0.06 (0.02)  Item  Factors  C  1  2  3  4  5  6  ANX-1 Applying knowledge of normal function to each of the major organ systems?  0.51  −0.02 (−0.05)  0.06 (0.37)  −0.03 (0.02)  0.18 (0.46)  −0.59 (−0.69)  0.00 (−0.08)  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  0.55  0.01 (0.02)  0.08 (0.35)  0.05 (0.16)  0.01 (0.34)  −0.69 (−0.73)  0.02 (−0.08)  ANX-3 Applying knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  0.48  0.01 (0.02)  v0.31 (0.53)  0.00 (0.04)  0.09 (0.41)  −0.46 (−0.62)  0.02 (−0.05)  ANX-4 Using effective listening skills when interacting with a patient?  0.81  0.01 (0.01)  0.10 (0.44)  −0.01 (−0.15)  0.88 (0.89)  0.07 (−0.35)  0.03 (0.01)  ANX-5 Demonstrating caring when counseling a patient?  0.79  0.04 (0.04)  0.11 (0.42)  −0.01 (−0.15)  0.90 (0.88)  0.15 (−0.28)  0.01 (−0.01)  ANX-6 Accurately gathering essential information from a patient?  0.70  0.01 (−0.02)  −0.11 (0.30)  −0.01 (−0.06)  0.71 (0.79)  −0.31 (−0.56)  0.07 (0.01)  ANX-7 Performing a thorough physical exam?  0.50  −0.01 (−0.03)  −0.15 (0.22)  0.00 (0.00)  0.51 (0.62)  −0.39 (−0.55)  0.01 (−0.05)  ANX-8 Developing an appropriate differential diagnosis?  0.76  0.02 (−0.01)  0.00 (0.35)  0.00 (0.14)  −0.01 (0.37)  −0.88 (−0.87)  0.04 (−0.07)  ANX-9 Generating a patient-specific treatment plan?  0.67  0.02 (0.06)  0.18 (0.44)  0.06 (0.21)  −0.11 (0.28)  −0.76 (−0.80)  −0.04 (−0.15)  ANX-10 Using information technology to support patient-care decisions?  0.52  −0.09 (−0.03)  0.59 (0.68)  0.02 (−0.01)  0.02 (0.36)  −0.23 (−0.48)  0.00 (−0.03)  ANX-11 Working effectively with other health care professionals to provide high-quality patient care?  0.50  −0.06 (−0.02)  0.39 (0.58)  −0.01 (−0.07)  0.37 (0.58)  −0.09 (−0.42)  −0.14 (−0.16)  ANX-12 Improving clinical practice using a systematic approach?  0.64  −0.07 (0.01)  0.62 (0.75)  0.03 (−0.02)  0.19 (0.50)  −0.14 (−0.49)  −0.08 (−0.11)  ANX-13 Evaluating evidence from scientific studies relevant to your patients' health problems?  0.66  0.03 (0.10)  0.63 (0.72)  −0.05 (−0.01)  0.07 (0.41)  −0.13 (−0.45)  −0.33 (−0.36)  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  0.61  0.08 (0.16)  0.62 (0.69)  −0.03 (0.05)  0.01 (0.34)  −0.15 (−0.43)  −0.30 (−0.33)  ANX-15 Demonstrating sensitivity to patients' cultural differences?  0.61  0.10 (0.08)  0.28 (0.53)  −0.10 (−0.16)  0.63 (0.73)  0.07 (−0.30)  −0.05 (−0.07)  ANX-16 Balancing professional responsibilities with personal responsibilities?  0.31  −0.09 (−0.06)  −0.02 (0.23)  0.07 (0.04)  0.34 (0.45)  −0.29 (−0.46)  −0.11 (−0.16)  ANX-17 Discussing methods of controlling health care costs?  0.70  0.05 (0.10)  0.86 (0.81)  −0.02 (−0.07)  0.00 (0.30)  0.13 (−0.19)  0.13 (0.13)  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?  0.70  0.03 (0.08)  0.79 (0.81)  0.00 (−0.05)  0.05 (0.37)  0.01 (−0.31)  0.18 (0.16)  ANX-19 Applying high-quality health care in deployed military environments?  0.37  0.04 (0.08)  0.48 (0.57)  0.02 (0.03)  0.04 (0.32)  −0.19 (−0.40)  0.06 (0.02)  Anx, Anxiety; C, Communality. Pattern coefficients are presented first, followed by structure coefficients in parentheses. Entries in bold indicate pattern coefficients (absolute values) >0.50 on at least one factor, pattern coefficients (absolute values) ≥0.30 on only one factor, and communalities ≥0.40. Results presented above reflect loadings and communalities for Anxiety items only from the 6 factor solution examining Anxiety and Task Importance items simultaneously. Results continued from Table I. View Large Participants and Procedures Participants were recruited from the population of USU medical school students during the 2010–2011 and 2011–2012 academic years (N = 864). In May 2011 and May 2012, all students were contacted by e-mail and invited to complete the online survey of their perceptions of self-efficacy, task importance, and anxiety. Two follow-up e-mail reminders were sent out over the next 4 weeks, at which point the response period was closed. Participation in the survey was voluntary, and ethical approval was obtained from the USU Institutional Review Board. Analyses Before analysis, we screened the data for accuracy and missing values and evaluated all item scores for distribution normality. To evaluate the representativeness of our sample, we compared the respondents and the nonrespondents on undergraduate grade point average, Medical College Admissions Test scores, and age using multivariate analyses of variance (MANOVAs), as well as on gender using a χ2 test of independence. Next, we conducted an exploratory factor analyses on the task importance and anxiety items to evaluate their dimensional structure. These results were used to define subscales representing dimensions of task importance and anxiety related to the ACGME competencies. Internal consistency estimates and descriptive statistics were computed for all measures, and correlations were computed to assess relations to other variables. Finally, we conducted MANOVA tests to evaluate differences by year group for the task importance and anxiety subscales. We followed the MANOVAs with Bonferroni multiple comparisons of differences across year groups for each subscale. All analyses were completed using SPSS 21.0 (IBM Corporation, 2012, New York). Measures Avoidance of Help Seeking Avoidance of help seeking was assessed using a five-item scale developed by Pajares et al.19 These items assessed the extent to which students avoided asking for help even when they need it. Higher scores indicate greater help avoidance, and all items employed a 5-point, Likert-type response scale. Metacognition Metacognition was assessed using an eight-item scale from the Motivated Strategies for Learning Questionnaire.20 These items assessed the frequency with which students used metacognitive control strategies, including planning, goal setting, comprehension monitoring, and performance regulation to evaluate their progress as learners. Higher scores on this scale generally indicate a greater awareness of one's own thinking. All items employed a 5-point, Likert-type response scale. Procrastination Procrastination was captured using a four-item scale derived by Wolters.11 These items assessed the extent to which students disengage academically or tend to delay getting started on academic work. Higher scores indicate a greater degree of procrastination. All items employed a 5-point, Likert-type response scale. Self-Efficacy Self-efficacy was assessed using three subscales derived by Artino et al,16 based on variations of the 19-item ACGME competency items used in the present study. For all three scales, higher scores indicated greater self-efficacy, and all items employed 5-point Likert-type response scales. (a) Patient Care Self-Efficacy: This eight-item scale captured the extent to which medical students believe they are capable of providing adequate patient care. (b) Interpersonal Skills Self-Efficacy: This three-item scale captured the extent to which medical students believe they are capable of interacting and communicating effectively with patients. (c) Evidence-Based Medicine Self-Efficacy: This three-item scale assessed the degree to which students believe they are capable of practicing evidence-based medicine. RESULTS Survey responses were obtained from 368 students (43%) out of a total pool of 864 students enrolled during the 2010–2011 and 2011–2012 academic years. Of these, 131 (36%) were enrolled in medical school year 1 (MS-1), 97 (26.6%) in MS-2, 73 (19.8%) in MS-3, and 67 (18.2%) in MS-4. The sample included 102 (28%) female respondents, which is representative of the overall USU student population. Results of the comparisons of respondents versus nonrespondents revealed the following statistically significant differences: For MS-1 students, the respondents were, on average, slightly older than the nonrespondents [F (1) = 15.10, p < 0.001]. For MS-2 students, the respondents were, on average, slightly older than the nonrespondents [F (1) = 7.69, p < 0.05], had lower undergraduate grade point averages [F (1) = 11.18, p < 0.05], and had a higher proportion of males [χ2 (2) = 16.61, p < 0.05]. For MS-3 and MS-4 students, there were no differences between the two groups on any of the four measures. We interpreted these results as evidence that the sample of 368 students was sufficiently similar to the overall student population to justify proceeding with the study. Evidence Based on Internal Structure: Exploratory Factor Analysis Following data screening, we performed a principal axis factor analysis with oblique rotation (delta = 0), on the set of 38 task importance and anxiety items. The correlation matrix was found to be factorable; the Kaiser–Meyer–Olkin Measure of Sampling Adequacy was 0.90, which is considered extremely good.21 Bartlett's test of sphericity (χ2 = 11136.15, df = 703, p < 0.01) indicated that the correlation matrix was not an identity matrix, and all measures of sampling adequacy were deemed sufficient (i.e., >0.60). The scree plot and Eigenvalue criteria indicated a six-factor solution, with all factors accounting for 63.2% of the total variance in the items. Evaluation of the communalities extracted revealed that two anxiety items (ANX-16 and ANX-19) and one task importance item (IMP-16) had communalities <0.40 (Tables I and II), indicating that the extracted factors account for a large proportion of the remaining 35 items' common variance. These three items were removed from further analyses. Per the procedures outlined in the literature,16,21 we used several additional rules to further evaluate the interpretability of the items to be retained for the identified factors: (a) factors needed to contain at least three items; (b) the absolute value of all factor pattern coefficients needed to be >0.50 on at least one factor; and (c) items with factor pattern coefficients (absolute value) ≥0.30 on more than one factor were dropped. The factor pattern and structure coefficients from these analyses are displayed in Tables I and II. The first factor extracted (Eigenvalue = 10.13; variance = 25.83%) included nine task importance items (IMP-2, IMP-3, IMP-9, IMP-10, IMP-12, IMP-13, IMP-14, IMP-18, and IMP-19). Several other task importance items loaded on this factor as well, but none uniquely: IMP-1 and IMP-11 both also loaded on Factor 3, and IMP-8 and IMP-17 both also loaded on Factor 6. These four items were dropped from further analysis. Factor 1's correlations with other factors were 0.09 with Factor 2, 0.47 with Factor 3, −0.01 with Factor 4, 0.02 with Factor 5, and −0.12 with Factor 6. The second factor extracted (Eigenvalue =8.27; variance = 20.76%) included five anxiety items (ANX-10, ANX-12, ANX-14, ANX-17, and ANX-18). Item ANX-3 loaded on Factor 2, but also on Factor 5; item ANX-11 loaded here, but also on Factor 4; and item ANX-13 loaded on this factor as well as on Factor 6. All three of these items were excluded from further analysis. Factor 2 correlated at −0.05 with Factor 3, 0.41 with Factor 4, −0.40 with Factor 5, and −0.02 with Factor 6. The third factor extracted (Eigenvalue = 3.07; variance = 7.23%) included five uniquely loading task importance items (IMP-4, IMP-5, IMP-6, IMP-7, and IMP-15). The third factor correlated with Factor 4 at −0.14, Factor 5 at −0.16, and Factor 6 at −0.14. The fourth factor extracted (Eigenvalue = 1.39; variance = 4.19%) was defined by three uniquely loading anxiety items (ANX-4, ANX-5, and ANX-15). Items ANX-6 and ANX-7 loaded on Factor 4 as well as Factor 5, and were excluded from further analysis. Factor 4 exhibited correlations of −0.43 with Factor 5 and −0.03 with Factor 6. The fifth factor extracted (Eigenvalue = 1.39; variance = 2.79%) included four uniquely loading anxiety items (ANX-1, ANX-2, ANX-8, and ANX-9). Factor 5's correlation with Factor 6 was 0.13. The sixth factor extracted (Eigenvalue = 1.26; variance = 2.46%) included no uniquely loading items and was not considered further in analyses. Based on these results, we named each of the retained factors as follows: (a) Factor 1 was labeled “systems-based practice importance,” (b) Factor 2 was labeled “systems-based practice anxiety,” (c) Factor 3 was labeled “interpersonal skills and professionalism importance,” (d) Factor 4 was labeled “interpersonal skills anxiety,” and (e) Factor 5 was labeled “health knowledge anxiety.” Lists of the items in the task importance and anxiety subscales are provided in the  Appendix, in Tables AI and AII, respectively. Reliabilities for the subscales were all above the 0.75 threshold defined by McCoach et al.22 Further, all inter-item correlations were <0.70, indicating little item redundancy with each scale. Following internal consistency analyses, scale scores were computed for all five subscales as the mean score for all items comprising a given subscale. Evidence Based on Relations to Other Variables: Descriptive Statistics and Correlations Table III reports the overall means, standard deviations, and correlations among the task importance and anxiety subscales, the three self-efficacy subscales defined by Artino et al,16 and the learning strategies of metacognition, procrastination and avoidance of help seeking. Table III also displays internal consistency reliability values for all subscales included in this study, with values ranging from 0.78 to 0.92. As this table illustrates, the correlation between the two task importance subscales was moderate and statistically significant (r = 0.51, p < 0.01), as were the correlations among the three anxiety subscales (r = 0.53, r = 51, and r = 0.37; all p < 0.01) and the correlations among the three self-efficacy subscales (r = 0.39, r = 0.68, and r = 0.44; all p < 0.01). The statistically significant correlations between the task importance and anxiety subscales were limited to small correlations of “interpersonal skills and professionalism importance” with “interpersonal skills anxiety” (r = −0.15, p < 0.01) and “health knowledge anxiety” (r = −0.16, p < 0.01). Finally, the correlations between the newly developed subscales and the other variables assessed in this study were largely consistent with our first two hypotheses. There were a few notable exceptions (e.g., “procrastination” was uncorrelated with either task importance factor; it was also uncorrelated with “systems-based practice anxiety”). TABLE III. Correlations, Descriptive Statistics, and Internal Consistency Reliabilities       M (SD)  1  2  3  4  5  6  7  8  9  10  11    1  Systems-Based Practice Anxiety  2.08 (0.79)  (0.86)                        2  Int Skills Anxiety  1.73 (0.79)  0.53  (0.89)                      3  Health Knowledge Anxiety  2.83 (0.88)  0.51  0.37  (0.86)                    4  SystemsBased Practice Imp  3.50 (0.99)  0.08  0.05  0.03  (0.84)                  5  Int Skills and Professionalism Imp  4.32 (0.66)  −0.01  −0.15  −0.16  0.51  (0.88)                6  Patient Care SE  2.98 (0.86)  −0.12  −0.10  −0.34  0.50  0.25  (0.92)              7  Int Skills SE  4.12 (0.66)  −0.11  −0.32  −0.04  0.15  0.32  0.39  (0.78)            8  Evidence-Based Medicine SE  3.02 (0.87)  −0.20  −0.13  −0.24  0.35  0.16  0.68  0.44  (0.79)          9  Metacognition  3.31 (0.65)  0.01  −0.03  −0.04  0.19  0.25  0.16  0.25  0.24  (0.81)      10  Procrastination  2.6 (1.00)  0.06  0.17  0.15  −0.06  −0.08  −0.14  −0.04  −0.08  −0.17  (0.91)    11  Avoidance of Help Seeking  2.02 (0.84)  0.29  0.31  0.33  −0.02  −0.10  −0.09  −0.11  −0.05  −0.09  0.39  (0.83)        M (SD)  1  2  3  4  5  6  7  8  9  10  11    1  Systems-Based Practice Anxiety  2.08 (0.79)  (0.86)                        2  Int Skills Anxiety  1.73 (0.79)  0.53  (0.89)                      3  Health Knowledge Anxiety  2.83 (0.88)  0.51  0.37  (0.86)                    4  SystemsBased Practice Imp  3.50 (0.99)  0.08  0.05  0.03  (0.84)                  5  Int Skills and Professionalism Imp  4.32 (0.66)  −0.01  −0.15  −0.16  0.51  (0.88)                6  Patient Care SE  2.98 (0.86)  −0.12  −0.10  −0.34  0.50  0.25  (0.92)              7  Int Skills SE  4.12 (0.66)  −0.11  −0.32  −0.04  0.15  0.32  0.39  (0.78)            8  Evidence-Based Medicine SE  3.02 (0.87)  −0.20  −0.13  −0.24  0.35  0.16  0.68  0.44  (0.79)          9  Metacognition  3.31 (0.65)  0.01  −0.03  −0.04  0.19  0.25  0.16  0.25  0.24  (0.81)      10  Procrastination  2.6 (1.00)  0.06  0.17  0.15  −0.06  −0.08  −0.14  −0.04  −0.08  −0.17  (0.91)    11  Avoidance of Help Seeking  2.02 (0.84)  0.29  0.31  0.33  −0.02  −0.10  −0.09  −0.11  −0.05  −0.09  0.39  (0.83)  Imp, Task importance; Int, interpersonal; M, mean; SD, standard deviation; SE, self-efficacy. All pairwise n = 368. Correlations above ∣0.10∣ are significant at p < 0.05; correlations above ∣0.14∣ are significant at p < 0.01. Numbers along the diagonal in parentheses are coefficient alpha values. View Large TABLE III. Correlations, Descriptive Statistics, and Internal Consistency Reliabilities       M (SD)  1  2  3  4  5  6  7  8  9  10  11    1  Systems-Based Practice Anxiety  2.08 (0.79)  (0.86)                        2  Int Skills Anxiety  1.73 (0.79)  0.53  (0.89)                      3  Health Knowledge Anxiety  2.83 (0.88)  0.51  0.37  (0.86)                    4  SystemsBased Practice Imp  3.50 (0.99)  0.08  0.05  0.03  (0.84)                  5  Int Skills and Professionalism Imp  4.32 (0.66)  −0.01  −0.15  −0.16  0.51  (0.88)                6  Patient Care SE  2.98 (0.86)  −0.12  −0.10  −0.34  0.50  0.25  (0.92)              7  Int Skills SE  4.12 (0.66)  −0.11  −0.32  −0.04  0.15  0.32  0.39  (0.78)            8  Evidence-Based Medicine SE  3.02 (0.87)  −0.20  −0.13  −0.24  0.35  0.16  0.68  0.44  (0.79)          9  Metacognition  3.31 (0.65)  0.01  −0.03  −0.04  0.19  0.25  0.16  0.25  0.24  (0.81)      10  Procrastination  2.6 (1.00)  0.06  0.17  0.15  −0.06  −0.08  −0.14  −0.04  −0.08  −0.17  (0.91)    11  Avoidance of Help Seeking  2.02 (0.84)  0.29  0.31  0.33  −0.02  −0.10  −0.09  −0.11  −0.05  −0.09  0.39  (0.83)        M (SD)  1  2  3  4  5  6  7  8  9  10  11    1  Systems-Based Practice Anxiety  2.08 (0.79)  (0.86)                        2  Int Skills Anxiety  1.73 (0.79)  0.53  (0.89)                      3  Health Knowledge Anxiety  2.83 (0.88)  0.51  0.37  (0.86)                    4  SystemsBased Practice Imp  3.50 (0.99)  0.08  0.05  0.03  (0.84)                  5  Int Skills and Professionalism Imp  4.32 (0.66)  −0.01  −0.15  −0.16  0.51  (0.88)                6  Patient Care SE  2.98 (0.86)  −0.12  −0.10  −0.34  0.50  0.25  (0.92)              7  Int Skills SE  4.12 (0.66)  −0.11  −0.32  −0.04  0.15  0.32  0.39  (0.78)            8  Evidence-Based Medicine SE  3.02 (0.87)  −0.20  −0.13  −0.24  0.35  0.16  0.68  0.44  (0.79)          9  Metacognition  3.31 (0.65)  0.01  −0.03  −0.04  0.19  0.25  0.16  0.25  0.24  (0.81)      10  Procrastination  2.6 (1.00)  0.06  0.17  0.15  −0.06  −0.08  −0.14  −0.04  −0.08  −0.17  (0.91)    11  Avoidance of Help Seeking  2.02 (0.84)  0.29  0.31  0.33  −0.02  −0.10  −0.09  −0.11  −0.05  −0.09  0.39  (0.83)  Imp, Task importance; Int, interpersonal; M, mean; SD, standard deviation; SE, self-efficacy. All pairwise n = 368. Correlations above ∣0.10∣ are significant at p < 0.05; correlations above ∣0.14∣ are significant at p < 0.01. Numbers along the diagonal in parentheses are coefficient alpha values. View Large Mean Differences Across Years in Medical School Next, we performed a MANOVA to evaluate differences across the two task importance subscales by year group, treating respondents from each year of medical school, MS-1 to MS-4, as a separate group. Table IV presents mean scores for the sample partitioned by the respondent's year of medical school at the time of survey completion, and Figure 1 depicts these means for task importance plotted across year of medical school. Results of this test were significant [F(6,728) = 9.52, p < 0.01; λ = 0.86]. Between-subject tests indicated differences across year groups for both “systems-based practice importance” [F(3,365) = 15.82, p < 0.001] and “interpersonal skills and professionalism importance” [F(3,365) = 3.23, p < 0.05]. Results of Bonferroni multiple comparisons indicated significant differences on “systems-based practice importance” for all year group pairs except MS-3 and MS-4, with a trend of increasing scores across year groups on this scale. MS-3 respondents had higher scores than MS-1 respondents on “interpersonal skills and professionalism importance.” These results were interpreted as providing good support for our third hypothesis. TABLE IV. Subscale Score Means and Standard Deviations for the N = 368 Respondents by Year Group    MS-1  MS-2  MS-3  MS-4  p  n = 131  n = 97  n = 73  n = 67  Systems-Based Practice Anxiety  2.15 (0.84)  2.11 (0.77)  2.03 (0.80)  1.97 (0.71)    Interpersonal skills Anxiety  1.77 (0.79)  1.85 (0.86)  1.58 (0.74)  1.61 (0.71)    Health Knowledge Anxiety  2.98 (0.88)  3.04 (0.80)  2.65 (0.85)  2.43 (0.89)  <0.001  Systems-Based Practice Importance  3.08 (1.13)  3.46 (0.92)  3.89 (0.75)  3.95 (0.63)  <0.001  Interpersonal Skills and Professionalism Importance  4.19 (0.74)  4.37 (0.66)  4.47 (0.59)  4.36 (0.51)  <0.05     MS-1  MS-2  MS-3  MS-4  p  n = 131  n = 97  n = 73  n = 67  Systems-Based Practice Anxiety  2.15 (0.84)  2.11 (0.77)  2.03 (0.80)  1.97 (0.71)    Interpersonal skills Anxiety  1.77 (0.79)  1.85 (0.86)  1.58 (0.74)  1.61 (0.71)    Health Knowledge Anxiety  2.98 (0.88)  3.04 (0.80)  2.65 (0.85)  2.43 (0.89)  <0.001  Systems-Based Practice Importance  3.08 (1.13)  3.46 (0.92)  3.89 (0.75)  3.95 (0.63)  <0.001  Interpersonal Skills and Professionalism Importance  4.19 (0.74)  4.37 (0.66)  4.47 (0.59)  4.36 (0.51)  <0.05  MS, Medical school year. View Large TABLE IV. Subscale Score Means and Standard Deviations for the N = 368 Respondents by Year Group    MS-1  MS-2  MS-3  MS-4  p  n = 131  n = 97  n = 73  n = 67  Systems-Based Practice Anxiety  2.15 (0.84)  2.11 (0.77)  2.03 (0.80)  1.97 (0.71)    Interpersonal skills Anxiety  1.77 (0.79)  1.85 (0.86)  1.58 (0.74)  1.61 (0.71)    Health Knowledge Anxiety  2.98 (0.88)  3.04 (0.80)  2.65 (0.85)  2.43 (0.89)  <0.001  Systems-Based Practice Importance  3.08 (1.13)  3.46 (0.92)  3.89 (0.75)  3.95 (0.63)  <0.001  Interpersonal Skills and Professionalism Importance  4.19 (0.74)  4.37 (0.66)  4.47 (0.59)  4.36 (0.51)  <0.05     MS-1  MS-2  MS-3  MS-4  p  n = 131  n = 97  n = 73  n = 67  Systems-Based Practice Anxiety  2.15 (0.84)  2.11 (0.77)  2.03 (0.80)  1.97 (0.71)    Interpersonal skills Anxiety  1.77 (0.79)  1.85 (0.86)  1.58 (0.74)  1.61 (0.71)    Health Knowledge Anxiety  2.98 (0.88)  3.04 (0.80)  2.65 (0.85)  2.43 (0.89)  <0.001  Systems-Based Practice Importance  3.08 (1.13)  3.46 (0.92)  3.89 (0.75)  3.95 (0.63)  <0.001  Interpersonal Skills and Professionalism Importance  4.19 (0.74)  4.37 (0.66)  4.47 (0.59)  4.36 (0.51)  <0.05  MS, Medical school year. View Large FIGURE 1. View largeDownload slide Differences in task importance across year groups. FIGURE 1. View largeDownload slide Differences in task importance across year groups. Next, we conducted a second MANOVA to examine the effect of medical school year group on the three anxiety subscale. Results of this test were also significant [F(9,884) = 3.644, p < 0.01; λ = 0.92]. Between-subjects tests indicated year group differences for “health knowledge anxiety” [F(3,365) = 6.84, p < 0.001], but not for “systems-based practice anxiety or interpersonal skills anxiety.” Means for anxiety factors by year group are depicted in Figure 2. Results of Bonferroni multiple comparisons indicated lower scores on “health knowledge anxiety” for MS-4 respondents than for MS-1 and MS-2 respondents, and lower scores for MS-3 than MS-2 respondents. No other differences were found, indicating mixed support for our third hypothesis. FIGURE 2. View largeDownload slide Differences in anxiety scores across year groups. FIGURE 2. View largeDownload slide Differences in anxiety scores across year groups. DISCUSSION The primary purpose of this investigation was to develop a self-report measure of task importance and anxiety in relation to the six ACGME core competencies and to collect validity evidence for the new measures. The secondary purpose was to examine potential differences among the constructs of task importance and anxiety across cohorts enrolled at different points in medical school. Below, we provide a summary of our most important findings, and we provide interpretations and potential implications. Evidence Based on Internal Structure: Factor Analysis Our results suggest that task importance, as it relates to the ACGME competencies in undergraduate medical education, are composed of two factors: “systems-based practice importance” and “interpersonal skills and professionalism importance.” On the other hand, student anxiety, as it relates to these ACGME competencies, is represented by three distinct factors: “systems-based practice anxiety, interpersonal skills anxiety,” and “health knowledge anxiety.” All five of the subscales so derived have well-represented factor structures and good internal consistency, indicating that the primary goal of this study—derivation of psychometrically sound measures of task importance and anxiety—was achieved. In their previous work, Artino et al16 reported three self-efficacy subscales (“patient care self-efficacy, interpersonal skills self-efficacy,” and “evidence-based medicine self-efficacy”). Given that these self-efficacy subscales were derived from the same base set of 19 ACGME competencies as the task importance and anxiety subscales, it comes as no surprise that there are notable similarities in their factorial structures. However, there were also notable differences. Although interpersonal skills are distinct enough to be represented by distinct self-efficacy and anxiety subscales, the task importance subscales capturing this content area also reflected content related to professionalism. Systems-based practice was a concept represented by distinct importance and anxiety subscales, but differences were observed between the items describing these factors and those describing the self-efficacy subscales, the nearest approximation among the self-efficacy subscales being “evidence-based medicine self-efficacy.” Differences were also noted between the ACGME content describing “patient care self-efficacy” and that representing “health knowledge anxiety,” which dealt less with patient interaction. Evidence Based on Relations to Other Variables: Task Importance and Anxiety Factors In general, subscales reflecting task importance, anxiety, and self-efficacy all correlated with the other subscales, as predicted. It is notable that “systems-based practice importance” correlated with none of the three anxiety subscales, including “systems-based practice anxiety,” which is consistent in broad strokes with the findings of Artino et al.2 This suggests that task importance, relative to adopting systems-based practice, does not relate to the level of anxiety students experience with regard to their ability to implement systems-based practice, nor does it relate to their ability to interact effectively with patients or to their knowledge base regarding medicine and patient care. It is worth noting, however, that both “interpersonal skills anxiety” and “health knowledge anxiety” were negatively correlated with “interpersonal skills and professionalism importance,” which may suggest that students experiencing higher interpersonal skills anxiety and more anxiety related to their level of basic health knowledge may tend to attach less importance to this aspect of their performance as medical practitioners in a form of self-serving bias23 allowing them to rationalize their anxieties as less potentially damaging to their academic performance and career prospects. If borne out in future research, this finding could be very telling, and would have implications for teachers and curriculum designers, suggesting that early interventions to address these specific types of anxiety among students might help them recognize the importance of interpersonal skills and professionalism. It is also possible that interventions designed to convey the importance of interpersonal skills and professionalism may help remediate these specific types of anxiety in medical students, which could potentially improve learning outcomes and affect in a variety of ways.4 Evidence Based on Relations to Other Variables: Task Importance and Learning Strategies Consistent with our first hypothesis, the correlations of the task importance subscales with the self-efficacy subscales were all positive, suggesting that students with high degrees of self-efficacy tend to attach greater value to systems-based practice and interpersonal skills and professionalism. Conversely, it is also possible that students who attach higher levels of importance to these two factors tend to develop greater self-efficacy regarding patient care, interpersonal skills, and evidence-based medicine. It is also noteworthy that “avoidance of help seeking” exhibited negative relationships with both “interpersonal skills and professionalism importance” and “interpersonal skills self-efficacy.” This makes sense, since the act of seeking help is an inherently interpersonal activity, so it is unsurprising that students who attach low value to interpersonal skills or who have little confidence in their interpersonal skills are less likely to seek help for academic problems.24 The hypothesized relationships of “systems-based practice importance” and “interpersonal skills and professionalism importance” with “procrastination” were not observed. That said, “procrastination” was negatively related to “patient care self-efficacy,” suggesting that this type of efficacy is related, to a small degree, to a student's propensity to delay task accomplishment. The relationships of “systems-based practice importance” and “interpersonal skills and professionalism importance” with “metacognition” and self-efficacy factors were all positive and consistent with our hypothesis. Students who attach higher value to the practice of medicine according to systemic structure and to dealing effectively and professionally with peers and patients appear to have more awareness of their own cognitive processes, their own study strategies, and to have more confidence in their abilities, all of which is consistent with control-value theory.4 Evidence Based on Relations to Other Variables: Anxiety and Learning Strategies Not surprisingly, the correlations between anxiety subscales and self-efficacy subscales were primarily negative. The strongest relationships observed were between “interpersonal skills self-efficacy” and “interpersonal skills anxiety,” and between “patient care self-efficacy” and “health knowledge anxiety.” Medical students experiencing anxiety regarding their knowledge of health care appear to experience low self-efficacy regarding their ability to provide patient care or practice evidence-based medicine, though it is not possible to infer the directionality of this relationship from these analyses. Contrary to our second hypothesis, “metacognition” exhibited no relationship with any of the anxiety factors, suggesting that awareness of cognition and use of metacognitive control strategies does not mitigate any of the specific anxiety factors examined here. This is surprising, and although this study offers no visibility into relationships of metacognition with direct learning outcomes, its un-relatedness to any anxiety factors or to “avoidance of help seeking”24 in this sample suggests that either metacognition is a less effective strategy than previous research20 has suggested, or that something unusual may have happened in this sample. The measure used exhibited reasonable variance, has a plausible mean, and had excellent internal consistency, none of which would support an alternative explanation that respondents misunderstood the scale items. The sample did not appear to be unrepresentative of the target population, and the response rate, whereas somewhat low, was not alarmingly so. “Metacognition's” negative relationship with “procrastination” was unsurprising. Procrastination was related to “interpersonal skills anxiety” and “health knowledge anxiety,” but not to “systems-based practice anxiety,” which could indicate that the latter type of anxiety, which is related to effective utilization of resources, is less driven by a student's proclivity for “procrastination” than are the other two anxiety factors. In contrast, “health knowledge anxiety,” which deals with a capability developed over time through sustained effort, could easily be influenced by the degree of “procrastination” a student exhibits. Not surprisingly, “avoidance of help seeking” was positively correlated with all three anxiety factors. It is difficult to determine whether anxiety or avoidance is a proximal cause of the other; a cyclical relationship between the two seems possible, in which increasing anxiety leads to greater help avoidance, whereas greater avoidance simultaneously feeds higher anxiety levels. Task Importance and Anxiety Differences Across Medical School We found notable differences across year groups for the task importance and anxiety subscales. Medical students appear to have very different perceptions of task importance and anxiety, and these perceptions appear to differ in meaningful ways depending on the duration of medical students' education at the time of reporting. Support for our third hypothesis regarding changes across medical school was mixed, in that “systems-based practice anxiety” and “interpersonal skills anxiety” levels both remained constant across year groups, indicating that students experienced some baseline anxiety in these two dimensions that did not diminish over the course of undergraduate medical education. Consistent with this hypothesis, however, “health knowledge anxiety” was markedly higher for first and second-year undergraduate medical students than for more senior students. Students gain considerable health knowledge during their first 2 years of medical school and, as they begin their clinical rotations in year 3, it seems they are less anxious about their health knowledge and potentially more concerned about their upcoming clinical rotations and interactions with patients. This potential decrease in “health knowledge anxiety” across medical school years is consistent with the increases in “patient care self-efficacy” and “evidence-based medicine self-efficacy” reported by Artino et al.16 Consistent with our fourth hypothesis, more senior students tended to attach greater task importance to systems-based practice than did junior students, as evidenced by a rather strong upward trend. A similar difference was observed in “interpersonal skills and professionalism importance” in that third-year students attached greater importance to this competency area than did first-year students. Implications for Future Research The simple finding that both task importance and anxiety can be conceptualized as multidimensional constructs has potential implications for future research. Previous models treating these as unidimensional constructs should be re-examined in light of the present findings. Although relationships with other constructs were consistent across task importance and anxiety subscales in some cases, in others they were not. “Systems-based practice importance” and “systems-based practice anxiety” both tended to exhibit fewer significant relationships with other variables assessed than did other task importance and anxiety subscales. Beyond this finding, additional work is required to further evaluate the validity of the measures developed herein before any major conclusions can be drawn about theoretical implications. A logical next step would be an empirical test of these measures' relations to other variables in a path analysis or structural equation model in a control-value theory framework. Such work could shed light on the conditions under which different dimensions of task importance and anxiety in undergraduate medical education may exhibit different relationships with other elements of the control-value theory framework. In addition, the dimensional structure identified here should be evaluated in different contexts and settings, and for different populations. Limitations There are several limitations to this study, including the relatively small sample size, limited response rate, and our reliance on a single-institution sample. The cross-sectional year group differences reported cannot be interpreted as evidence of development or change within individuals, as the cohorts providing these data may differ in important ways not captured in these data. The observed patterns of hypothesized differences across years of medical school year could be partially due to the attrition of students experiencing academic difficulties. Our exclusive reliance on self-report data must also be acknowledged as a possible source of social desirability or other response bias, although available psychometric evidence does not indicate any such problems in the current data. To generate more generalizable results, this study should be replicated across educational settings, and if sufficient data become available, analyses of differences across year groups should be evaluated longitudinally. Another critical requirement is the need to evaluate the relationships of these scales with a broader range of constructs, including objective performance measures, to demonstrate even stronger validity evidence. CONCLUSIONS This study represents an important follow-on to the work reported by Artino et al,16 the purpose of which was development and initial validation of a medical skills self-efficacy measure for use in evaluating undergraduate medical students. The present work describes how, using the same set of baseline ACGME competency descriptors, assessments of task importance and anxiety were developed and initially validated. Importantly, this study illustrates that by linking students' motivational beliefs and emotional experiences to specific core competencies, medical educators are able to assess if and how these beliefs and emotions evolve (or regress) across the medical education continuum. Understanding how students' beliefs and emotions might change during medical school is important considering the close links that have recently been observed between these so-called “noncognitive factors” and students' self-regulated learning and performance in undergraduate medical education and beyond.2,5,9,15 Thus, by providing preliminary validity evidence for a set of assessment results of task importance and anxiety, we have given medical educators another useful tool for assessing these factors and making inferences about the relationships between medical student affect and other educationally relevant outcomes. Furthermore, because the quality of research in medical education hinges, in part, on the quality of the measurement tools employed, examining newly developed survey instruments and collecting validity evidence for their interpretation and intended use further informs the field. ACKNOWLEDGMENTS This research was supported by an intramural research grant from the Dean's Education Endowment, F. Edward Hébert School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland. APPENDIX Tables AI and AII include scale-specific item lists for the two task importance subscales and three anxiety subscales derived in this study. The following section provides limited background on the development of the three self-efficacy subscales generated from factor analysis of the same set of 19 ACGME competency items described in Artino et al16 using a more limited dataset. TABLE AI. Task Importance Items Classified Into the Two Task Importance Subscales The following items address the importance you place on medical knowledge and skills. For each item, select the response that best reflects the level of importance you place on each knowledge or skill.  At this point in your medical training, how important is it that you can…  Systems-Based Practice Importance  Interpersonal Skills and Professionalism Importance  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  IMP-4 Use effective listening skills when interacting with a patient?  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  IMP-5 Demonstrate caring when counseling a patient?  IMP-9 Generate a patient-specific treatment plan?  IMP-6 Accurately gather essential information from a patient?  IMP-10 Use information technology to support patient-care decisions?  IMP-7 Perform a thorough physical exam?  IMP-12 Improve clinical practice using a systematic approach?  IMP-15 Demonstrate sensitivity to patients' cultural differences?  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?    IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?    IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?    IMP-19 Apply high-quality health care in deployed military environments?    The following items address the importance you place on medical knowledge and skills. For each item, select the response that best reflects the level of importance you place on each knowledge or skill.  At this point in your medical training, how important is it that you can…  Systems-Based Practice Importance  Interpersonal Skills and Professionalism Importance  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  IMP-4 Use effective listening skills when interacting with a patient?  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  IMP-5 Demonstrate caring when counseling a patient?  IMP-9 Generate a patient-specific treatment plan?  IMP-6 Accurately gather essential information from a patient?  IMP-10 Use information technology to support patient-care decisions?  IMP-7 Perform a thorough physical exam?  IMP-12 Improve clinical practice using a systematic approach?  IMP-15 Demonstrate sensitivity to patients' cultural differences?  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?    IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?    IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?    IMP-19 Apply high-quality health care in deployed military environments?    View Large TABLE AI. Task Importance Items Classified Into the Two Task Importance Subscales The following items address the importance you place on medical knowledge and skills. For each item, select the response that best reflects the level of importance you place on each knowledge or skill.  At this point in your medical training, how important is it that you can…  Systems-Based Practice Importance  Interpersonal Skills and Professionalism Importance  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  IMP-4 Use effective listening skills when interacting with a patient?  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  IMP-5 Demonstrate caring when counseling a patient?  IMP-9 Generate a patient-specific treatment plan?  IMP-6 Accurately gather essential information from a patient?  IMP-10 Use information technology to support patient-care decisions?  IMP-7 Perform a thorough physical exam?  IMP-12 Improve clinical practice using a systematic approach?  IMP-15 Demonstrate sensitivity to patients' cultural differences?  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?    IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?    IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?    IMP-19 Apply high-quality health care in deployed military environments?    The following items address the importance you place on medical knowledge and skills. For each item, select the response that best reflects the level of importance you place on each knowledge or skill.  At this point in your medical training, how important is it that you can…  Systems-Based Practice Importance  Interpersonal Skills and Professionalism Importance  IMP-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  IMP-4 Use effective listening skills when interacting with a patient?  IMP-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  IMP-5 Demonstrate caring when counseling a patient?  IMP-9 Generate a patient-specific treatment plan?  IMP-6 Accurately gather essential information from a patient?  IMP-10 Use information technology to support patient-care decisions?  IMP-7 Perform a thorough physical exam?  IMP-12 Improve clinical practice using a systematic approach?  IMP-15 Demonstrate sensitivity to patients' cultural differences?  IMP-13 Evaluate evidence from scientific studies relevant to your patients' health problems?    IMP-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?    IMP-18 Practice cost-effective health care delivery that does not compromise quality of care?    IMP-19 Apply high-quality health care in deployed military environments?    View Large TABLE AII. Anxiety Items Classified Into the Three Anxiety Subscales The following items address your anxiety in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of anxiety.  At this point in your medical training, how much anxiety do you feel about…  Systems-Based Practice Anxiety  Interpersonal skills Anxiety  Health Knowledge Anxiety  ANX-10 Using information technology to support patient-care decisions?  ANX-4 Using effective listening skills when interacting with a patient?  ANX-1 Applying knowledge of normal function to each of the major organ systems?  ANX-12 Improving clinical practice using a systematic approach?  ANX-5 Demonstrating caring when counseling a patient?  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  ANX-15 Demonstrating sensitivity to patients' cultural differences?  ANX-8 Developing an appropriate differential diagnosis?  ANX-17 Discussing methods of controlling health care costs?    ANX-9 Generating a patient-specific treatment plan?  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?      The following items address your anxiety in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of anxiety.  At this point in your medical training, how much anxiety do you feel about…  Systems-Based Practice Anxiety  Interpersonal skills Anxiety  Health Knowledge Anxiety  ANX-10 Using information technology to support patient-care decisions?  ANX-4 Using effective listening skills when interacting with a patient?  ANX-1 Applying knowledge of normal function to each of the major organ systems?  ANX-12 Improving clinical practice using a systematic approach?  ANX-5 Demonstrating caring when counseling a patient?  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  ANX-15 Demonstrating sensitivity to patients' cultural differences?  ANX-8 Developing an appropriate differential diagnosis?  ANX-17 Discussing methods of controlling health care costs?    ANX-9 Generating a patient-specific treatment plan?  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?      View Large TABLE AII. Anxiety Items Classified Into the Three Anxiety Subscales The following items address your anxiety in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of anxiety.  At this point in your medical training, how much anxiety do you feel about…  Systems-Based Practice Anxiety  Interpersonal skills Anxiety  Health Knowledge Anxiety  ANX-10 Using information technology to support patient-care decisions?  ANX-4 Using effective listening skills when interacting with a patient?  ANX-1 Applying knowledge of normal function to each of the major organ systems?  ANX-12 Improving clinical practice using a systematic approach?  ANX-5 Demonstrating caring when counseling a patient?  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  ANX-15 Demonstrating sensitivity to patients' cultural differences?  ANX-8 Developing an appropriate differential diagnosis?  ANX-17 Discussing methods of controlling health care costs?    ANX-9 Generating a patient-specific treatment plan?  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?      The following items address your anxiety in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of anxiety.  At this point in your medical training, how much anxiety do you feel about…  Systems-Based Practice Anxiety  Interpersonal skills Anxiety  Health Knowledge Anxiety  ANX-10 Using information technology to support patient-care decisions?  ANX-4 Using effective listening skills when interacting with a patient?  ANX-1 Applying knowledge of normal function to each of the major organ systems?  ANX-12 Improving clinical practice using a systematic approach?  ANX-5 Demonstrating caring when counseling a patient?  ANX-2 Effectively managing the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  ANX-14 Staying abreast of relevant scientific advances by reading peer-reviewed medical journals?  ANX-15 Demonstrating sensitivity to patients' cultural differences?  ANX-8 Developing an appropriate differential diagnosis?  ANX-17 Discussing methods of controlling health care costs?    ANX-9 Generating a patient-specific treatment plan?  ANX-18 Practicing cost-effective health care delivery that does not compromise quality of care?      View Large Similar to our task importance and anxiety scales, the items comprising the three self-efficacy subscales were related to the ACGME's six core competencies in that the items were designed to measure students' medical skills self-efficacy in relation to the attainment of the core competencies. All items employed a 5-point, Likert-type response scale: “not at all confident, slightly confident, moderately confident, quite confident, and extremely confident.” Results from Artino et al19 suggested a three-factor model of medical skills self-efficacy (Table AIII). The resulting survey was composed of three subscales measuring “patient care self-efficacy, interpersonal skills self-efficacy, and evidence-based medicine self-efficacy.” “Patient care self-efficacy” is an 8-item scale (items SE-1, SE-2, SE-3, SE-8, SE-9, SE-10, SE-12, and SE-19) capturing the degree to which medical students believe they are capable of providing adequate patient care. “Interpersonal skills self-efficacy” is a 3-item scale (items SE-4, SE-5, and SE-15) capturing the degree to which medical students believe they are capable of interacting and communicating effectively with patients. “Evidence-based medicine self-efficacy” is a 3-item scale (items SE-13, SE-14, and SE-18) assessing the degree to which students believe they are capable of practicing evidence-based medicine. TABLE AIII. Items from the Three Self-Efficacy Subscales Developed by Artino et al19 The following items address your confidence in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of confidence.  At this point in your medical training, how confident are you that you can…  Patient care self-efficacy  Interpersonal skills self-efficacy  Evidence-based medicine self-efficacy  SE-1 Apply knowledge of normal function to each of the major organ systems?  SE-4 Use effective listening skills when interacting with a patient?  SE-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  SE-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  SE-5 Demonstrate caring when counseling a patient?  SE-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  SE-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  SE-15 Demonstrate sensitivity to patients' cultural differences?  SE-18 Practice cost-effective health care delivery that does not compromise quality of care?  SE-8 Develop an appropriate differential diagnosis?      SE-9 Generate a patient-specific treatment plan?      SE-10 Use information technology to support patient-care decisions?      SE-12 Improve clinical practice using a systematic approach?      SE-19 Apply high-quality health care in deployed military environments?      The following items address your confidence in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of confidence.  At this point in your medical training, how confident are you that you can…  Patient care self-efficacy  Interpersonal skills self-efficacy  Evidence-based medicine self-efficacy  SE-1 Apply knowledge of normal function to each of the major organ systems?  SE-4 Use effective listening skills when interacting with a patient?  SE-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  SE-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  SE-5 Demonstrate caring when counseling a patient?  SE-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  SE-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  SE-15 Demonstrate sensitivity to patients' cultural differences?  SE-18 Practice cost-effective health care delivery that does not compromise quality of care?  SE-8 Develop an appropriate differential diagnosis?      SE-9 Generate a patient-specific treatment plan?      SE-10 Use information technology to support patient-care decisions?      SE-12 Improve clinical practice using a systematic approach?      SE-19 Apply high-quality health care in deployed military environments?      View Large TABLE AIII. Items from the Three Self-Efficacy Subscales Developed by Artino et al19 The following items address your confidence in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of confidence.  At this point in your medical training, how confident are you that you can…  Patient care self-efficacy  Interpersonal skills self-efficacy  Evidence-based medicine self-efficacy  SE-1 Apply knowledge of normal function to each of the major organ systems?  SE-4 Use effective listening skills when interacting with a patient?  SE-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  SE-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  SE-5 Demonstrate caring when counseling a patient?  SE-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  SE-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  SE-15 Demonstrate sensitivity to patients' cultural differences?  SE-18 Practice cost-effective health care delivery that does not compromise quality of care?  SE-8 Develop an appropriate differential diagnosis?      SE-9 Generate a patient-specific treatment plan?      SE-10 Use information technology to support patient-care decisions?      SE-12 Improve clinical practice using a systematic approach?      SE-19 Apply high-quality health care in deployed military environments?      The following items address your confidence in relation to your medical knowledge and skills. For each item, select the response that best reflects your level of confidence.  At this point in your medical training, how confident are you that you can…  Patient care self-efficacy  Interpersonal skills self-efficacy  Evidence-based medicine self-efficacy  SE-1 Apply knowledge of normal function to each of the major organ systems?  SE-4 Use effective listening skills when interacting with a patient?  SE-13 Evaluate evidence from scientific studies relevant to your patients' health problems?  SE-2 Effectively manage the uncertainty associated with patient care, such as when the patient has multiple treatment options, each with its own risks and benefits?  SE-5 Demonstrate caring when counseling a patient?  SE-14 Stay abreast of relevant scientific advances by reading peer-reviewed medical journals?  SE-3 Apply knowledge of epidemiology of common diseases, such as heart disease, to reduce disease incidence?  SE-15 Demonstrate sensitivity to patients' cultural differences?  SE-18 Practice cost-effective health care delivery that does not compromise quality of care?  SE-8 Develop an appropriate differential diagnosis?      SE-9 Generate a patient-specific treatment plan?      SE-10 Use information technology to support patient-care decisions?      SE-12 Improve clinical practice using a systematic approach?      SE-19 Apply high-quality health care in deployed military environments?      View Large REFERENCES 1. Pekrun R, Elliot AJ, Maier MA Achievement goals and discrete achievement emotions: a theoretical model and prospective test. J Educ Psychol  2006; 98( 3): 583– 97. Google Scholar CrossRef Search ADS   2. Artino AR, La Rochelle JS, Durning SJ Second-year medical students' motivational beliefs, emotions, and achievement. Med Educ  2010; 44( 12): 1203– 12. Google Scholar CrossRef Search ADS PubMed  3. Pintrich PR A motivational science perspective on the role of student motivation in learning and teaching contexts. J Educ Psychol  2003; 95( 4): 667– 86. Google Scholar CrossRef Search ADS   4. Pekrun R The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ Psychol Rev  2006; 18( 4): 315– 41. Google Scholar CrossRef Search ADS   5. McConnell MM, Eva KW The role of emotion in the learning and transfer of clinical skills and knowledge. Acad Med  2012; 87( 10): 1316– 22. Google Scholar CrossRef Search ADS PubMed  6. Pekrun R, Goetz T, Titz W, Perry RP Academic emotions in students' self-regulated learning and achievement: a program of qualitative and quantitative research. Educ Psychol  2002; 37( 2): 91– 105. Google Scholar CrossRef Search ADS   7. Bandura A Social Foundations of Thought and Action: A Social Cognitive Theory . Englewood Cliffs, NJ, Prentice-Hall, 1986. 8. Pekrun R, Elliot AJ, Maier MA Achievement goals and achievement emotions: testing a model of their joint relations with academic performance. J Educ Psychol  2009; 101( 1): 115– 35. Google Scholar CrossRef Search ADS   9. Artino AR, Holmboe ES, Durning SJ Control-value theory: using achievement emotions to improve understanding of motivation, learning, and performance in medical education: AMEE Guide No. 64. Med Teach  2012; 34( 3): e148– 60. Google Scholar CrossRef Search ADS PubMed  10. Kaplan S, Berman MG Directed attention as a common resource for executive functioning and self-regulation. Perspect Psychol Sci  2010; 5( 1): 43– 57. Google Scholar CrossRef Search ADS PubMed  11. Wolters CA Understanding procrastination from a self-regulated learning perspective. J Educ Psychol  2003; 95( 1): 179– 187. Google Scholar CrossRef Search ADS   12. Karabenick SA Seeking help in large college classes: a person-centered approach. Contemp Educ Psychol  2003; 28( 1): 37– 58. Google Scholar CrossRef Search ADS   13. Kusurkar RA, Croiset G, Mann KV, Custers E, Ten Cate O Have motivation theories guided the development and reform of medical education curricula? A review of the literature. Acad Med  2012; 87( 6): 735– 43. Google Scholar CrossRef Search ADS PubMed  14. Eccles JS, Wigfield A Motivational beliefs, values, and goals. Annu Rev Psychol  2002; 53: 109– 32. Google Scholar CrossRef Search ADS PubMed  15. Artino AR Academic self-efficacy: from educational theory to instructional practice. Perspect Med Educ  2012; 1( 2): 76– 85. Google Scholar CrossRef Search ADS PubMed  16. Artino AR, Dong T, DeZee KJ, et al.   Development and initial validation of a survey to assess students' self-efficacy in medical school. Mil Med  2012; 177( 9 Suppl): 31– 7. Google Scholar CrossRef Search ADS PubMed  17. ACGME Introduction to Competency-Based Education: Facilitator's Guide . Chicago, IL, 2006. Available at http://www.dconnect.acgme.org/outcome/e-learn/21M1_FacManual.pdf; accessed May 1, 2014. 18. McKenzie JF, Wood ML, Kotecki JE, Clark JK, Brey RA Establishing content validity: using qualitative and quantitative steps. In: Health Promotion & Education Research Methods: Using the Five Chapter Thesis/Dissertation Model , Ed 2, pp 311– 318. Edited by Cottrell R, McKenzie JF Sudbury, MA, Johns and Bartlett Publishers, 2011. 19. Pajares F, Cheong YF, Oberman P Psychometric analysis of computer science help-seeking scales. Educ Psychol Meas  2004; 64( 3): 496– 513. Google Scholar CrossRef Search ADS   20. Pintrich PR, Smith DAF, Garcia T, Mckeachie WJ Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educ Psychol Meas  1993; 53( 3): 801– 13. Google Scholar CrossRef Search ADS   21. Pett MA, Beck SL, Guo J-W, et al.   Confirmatory factor analysis of the pain care quality surveys (PainCQ©). Health Serv Res  2013; 48( 3): 1018– 38. Google Scholar CrossRef Search ADS PubMed  22. McCoach DB, Gable RK, Madura JP Instrument Development in the Affective Domain: School and Corporate Applications , Ed 3. New York, Springer, 2013. Google Scholar CrossRef Search ADS   23. Shepperd J, Malone W, Sweeny K Exploring causes of the self-serving bias. Soc Personal Psychol Compass  2008; 2: 895– 908. Google Scholar CrossRef Search ADS   24. Karabenick SA Perceived achievement goal structure and college student help seeking. J Educ Psychol  2004; 96( 3): 569– 81. Google Scholar CrossRef Search ADS   Footnotes 1 This study was approved by the Institutional Review Board of the Uniformed Services University of the Health Sciences, Bethesda, Maryland. Reprint & Copyright © Association of Military Surgeons of the U.S.

MC, Jeffrey S. LaRochelle, USAF;PhD, Ting Dong,;PhD, Steven J. Durning, MD,

doi: 10.7205/MILMED-D-14-00566pmid: 25850125

ABSTRACT Purpose: Many medical schools across the United States are undergoing curriculum reform designed, in part, to integrate basic sciences and clinical skills. Evidence has suggested that preclerkship courses in clinical skills and clinical reasoning are predictive of student performance on the clerkship. We hypothesized that a combination of outcome measures from preclerkship clinical skills and clinical reasoning courses (Objective Structured Clinical Examination scores, preceptor evaluations, National Board of Medical Examiners subject examination scores, and small group participation grades) would be correlated to performance in internship (program director [PD] evaluation form at end of first postgraduate year). Methods: Outcome measures from preclerkship clinical skills and clinical reasoning courses and PD evaluation forms from 514 medical students graduating between 2009 and 2011 were analyzed in a multiple linear regression model. Results: Preclerkship clinical skills and clinical reasoning outcome measures were significant contributors to the linear regression model and were able to explain 13.9% of the variance in expertise and 7.6% of the variance in professionalism as measured by the PD evaluation form. Conclusion: Clinical skills and clinical reasoning courses during the preclerkship period explained a significant amount of performance at the graduate medical education level. Our data suggest that these courses provide valuable information regarding student abilities in internship. Early recognition of struggling students may provide an opportunity to break a cycle of poor performance that can persist into residency training. INTRODUCTION Many medical schools around the United States are undergoing curriculum reform efforts. Faculty have championed curriculum reform for several reasons to include integrating basic sciences with clinical skills teaching and providing early exposure to clinical encounters.1 This is also consistent with student desires for both increased patient contact during the preclerkship period of medical school and for opportunities to experience early clinical exposure over didactic methods of teaching.2,3 Furthermore, students also desire more advanced clinical skills and clinical reasoning training in preparation for their clerkship experience, whereas faculty tend to focus the preclerkship curriculum on the basic components of communication skills, patient interviewing, physical examination skills, and clinical reasoning.1,4 Despite this focus on foundational skills, a significant number of clerkship directors do not feel students are adequately prepared for the core clerkships, and that even more emphasis should be placed on basic components of clinical skills during the preclerkship time period.5 Identification of learners at risk for poor clinical skills performance in the preclerkship period could help initiate early remediation efforts in an educational setting more suited for this endeavor.6,7 In fact, previous studies have suggested that students who perform poorly on an observed clinical examination in the preclerkship time frame are often at risk for poor performances on this same type of examination at the end of their medical school training.8 Although individual preclerkship components assessing communication skills, medical knowledge, professionalism, etc. have been shown to predict future performance on specific outcomes, these individual components often fall short of assessing the overall impact of the preclerkship teaching on foundational clinical skills.9 However, a combination of preclerkship assessments in the context of an integrated, longitudinal curriculum could provide a more accurate understanding of a learner's clinical skills and clinical reasoning throughout their undergraduate medical education and even into their graduate medical education (GME) training.10 In fact, others have suggested the development of a common vision with regards to foundational clinical skills teaching emphasizing a longitudinal approach extending into the clerkship time frame.11,12 At Uniformed Services University (USU), the Introduction to Clinical Medicine (ICM) and Introduction to Clinical Reasoning (ICR) provided a unique opportunity to examine how combined preclerkship assessments may explain the variance on future student performance. Data from previous work demonstrated preclerkship outcome measures explained 22% of variance in average National Board of Medical Examiners (NBME) clerkship subject examinations and 20.2% of the variance in clerkship final grades (LaRochelle et al, unpublished data). Our current study was designed to extend the time frame into the GME period, and examine the extent that combined pre-clerkship course evaluations from ICM and ICR are associated with students' clinical performance in the first year of residency training (PGY-1) as measured by our previously validated program director (PD) evaluation form.13 We hypothesized that student performance in ICM and ICR would explain some of the variance in their performance at the PGY-1 level independent of baseline academic abilities. METHODS Study Context and Participants This investigation was part of the larger Long-Term Career Outcome Study conducted at the F. Edward Hébert School of Medicine, USU, and was granted ethical approval through the USU Institutional Review Board. As the United States' only federal medical school, USU matriculates approximately 170 medical students annually and, at the time of this study, offered a traditional 4-year curriculum: 2 years of basic science courses followed by 2 years of clinical rotations (clerkships). Both the ICM and ICR courses run throughout the entire second year of medical school. The ICM course is a case-based curriculum that uses standardized patient encounters with direct faculty observation to teach the basic clinical skills of history taking and physical examination. The ICR course is also a case-based curriculum that uses a combination of didactics and small-group teaching to deliver instruction on a broad variety of clinical reasoning techniques.14 The participants of this study were students graduating in 2009 through 2011 (N = 514; 143 were female [27.8%] and 371 were male [72.2%]). Measures and Statistical Analysis ICM Course Performance Measures of student performance in ICM course included a preceptor evaluation, an Objective Structured Clinical Examination (OSCE), and the NBME subject examination on the Introduction to Clinical Diagnosis (ICD). The preceptor evaluations are based on direct observation of basic clinical skills (history taking, physical examination, oral case presentation, and written notes) over six sessions at the National Capital Area Simulation Center using standardized patient encounters. The OSCE and NBME are integrated assessments with ICR and are described below. ICR Course Performance The students' performance in ICR course was measured on faculty-derived examination points, which consists of two multiple-choice examinations and one cumulative essay examination. Additionally, students receive points based on their small group discussions, where students are graded based on their level of participation in each of over 30 small group sessions that deal with common symptoms, findings, and syndromes in medicine using a case-based approach. Integrated ICM and ICR Assessments The OSCE is a six-station examination evaluating basic clinical examination skills (history, physical examination, oral case presentation, and medical knowledge) and clinical reasoning across multiple content domains to include neurology, geriatrics, gastroenterology, endocrinology, and anemia given at the end of the preclerkship period. The NBME subject examination is also a shared assessment between the ICM and ICR course and is given at the end of the preclerkship period. PGY-1 Medical Expertise and Professionalism Score We collect PGY-1 data annually from PDs. Each spring, we identify the programs where our interns and residents are trained, and we mail the evaluation forms for each trainee to the respective GME PDs. Interns who received a score of 3 or less (on a 5-point Likert scale) on patient care, medical expertise, or professionalism were identified as being at risk in that particular domain. The PGY-1 survey for class of 2009 consisted of both the old form and the new form, whereas classes of 2010 and 2011 just used the new form. Factor analysis of the old form revealed two factors: Expertise and Professionalism. Factor analysis of the new form had five extracted factors, which also included Expertise and Professionalism, and only the new survey was designed parallel to the Accreditation Council for Graduate Medical Education competencies. For purposes of this study, we analyzed the medical expertise and professionalism, as these two factors overlapped on both the old and new forms. The psychometric properties of this evaluation form were recently investigated, and the results indicated reasonable validity and reliability.14 Statistical Analysis We first reported the descriptive statistics of all the outcome measures, clerkship counseling, and PGY-1 survey data. A multiple linear regression analysis was accomplished to examine the students' ICM and ICR course assessments ability to predict future performance at the PGY-1 level in the domains of medical expertise and professionalism. First-year grade point average from medical school was included in the analysis as a control variable for student academic ability. Additionally, χ2 testing was accomplished to determine any significant associations among students counseled during their internal medicine clerkship and subsequent low performance at the PGY-1 level in medical expertise and professionalism. All the statistical analyses were conducted using SPSS 22.0. RESULTS Descriptive statistics for each of the outcome measures and dependent variables are included in Table I. Including all of the ICM and ICR preclerkship outcome measures in the multiple linear regression model of students' performance, these two pre-clerkship courses explained 13.9% of the variance of the PGY-1 PD survey scores for medical expertise (Table II). The ICD NBME subject examination score (β = 0.192, p < 0.009) and the OSCE score (β = 0.134, p = 0.013) were significant predictors of the dependent variable. Both the ICR examination scores (β = 0.266) and ICR small group grades (β = 0.238) approached statistical significance within the model with p values of 0.07 and 0.09, respectively (Table II). TABLE I. Descriptive Statistics of Outcome Measures Measure  Mean  SD  Minimum  Maximum  ICM/ICR NBME  83.74  8.61  60  100  ICM Preceptor  87.58  6.90  31  100  ICM OSCE  79.29  5.93  61  95.19  ICR Examination Points  66.66  10.92  47.90  98.00  Z score of ICR Small Group Points  —  —  −5.13  2.55  PGY-1 Expertise  3.71  0.75  1.73  5.00  PGY-1 Professionalism  3.93  0.86  1.00  5.00  Measure  Mean  SD  Minimum  Maximum  ICM/ICR NBME  83.74  8.61  60  100  ICM Preceptor  87.58  6.90  31  100  ICM OSCE  79.29  5.93  61  95.19  ICR Examination Points  66.66  10.92  47.90  98.00  Z score of ICR Small Group Points  —  —  −5.13  2.55  PGY-1 Expertise  3.71  0.75  1.73  5.00  PGY-1 Professionalism  3.93  0.86  1.00  5.00  View Large TABLE I. Descriptive Statistics of Outcome Measures Measure  Mean  SD  Minimum  Maximum  ICM/ICR NBME  83.74  8.61  60  100  ICM Preceptor  87.58  6.90  31  100  ICM OSCE  79.29  5.93  61  95.19  ICR Examination Points  66.66  10.92  47.90  98.00  Z score of ICR Small Group Points  —  —  −5.13  2.55  PGY-1 Expertise  3.71  0.75  1.73  5.00  PGY-1 Professionalism  3.93  0.86  1.00  5.00  Measure  Mean  SD  Minimum  Maximum  ICM/ICR NBME  83.74  8.61  60  100  ICM Preceptor  87.58  6.90  31  100  ICM OSCE  79.29  5.93  61  95.19  ICR Examination Points  66.66  10.92  47.90  98.00  Z score of ICR Small Group Points  —  —  −5.13  2.55  PGY-1 Expertise  3.71  0.75  1.73  5.00  PGY-1 Professionalism  3.93  0.86  1.00  5.00  View Large TABLE II. Regression Models of ICM and ICR Course Performances to Predict Variance in PGY-1 PD Survey Scores for Medical Expertise Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.017  0.192  0.009  0.139  ICM Preceptor  0.000  0.003  0.945  ICM OSCE  0.017  0.034  0.013  ICR Small Group Points  0.034  0.238  0.090  ICR Examination Points  0.018  0.266  0.070  Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.017  0.192  0.009  0.139  ICM Preceptor  0.000  0.003  0.945  ICM OSCE  0.017  0.034  0.013  ICR Small Group Points  0.034  0.238  0.090  ICR Examination Points  0.018  0.266  0.070  View Large TABLE II. Regression Models of ICM and ICR Course Performances to Predict Variance in PGY-1 PD Survey Scores for Medical Expertise Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.017  0.192  0.009  0.139  ICM Preceptor  0.000  0.003  0.945  ICM OSCE  0.017  0.034  0.013  ICR Small Group Points  0.034  0.238  0.090  ICR Examination Points  0.018  0.266  0.070  Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.017  0.192  0.009  0.139  ICM Preceptor  0.000  0.003  0.945  ICM OSCE  0.017  0.034  0.013  ICR Small Group Points  0.034  0.238  0.090  ICR Examination Points  0.018  0.266  0.070  View Large Including all of the ICM and ICR preclerkship outcome measures in the multiple linear regression model of students' performance, these two preclerkship courses explained 7.6% of the variance of the PGY-1 PD survey scores for medical expertise (Table III). The OSCE score (β = 0.124, p < 0.026) was the only significant predictor of the dependent variable in this model. TABLE III. Regression Models of ICM and ICR Course Performances to Predict Variance in PGY-1 PD Survey Scores for Professionalism Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.010  0.098  0.194  0.076  ICM Preceptor  0.003  0.025  0.626  ICM OSCE  0.018  0.124  0.026  ICR Small Group Points  0.034  0.215  0.139  ICR Examination Points  0.018  0.231  0.128  Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.010  0.098  0.194  0.076  ICM Preceptor  0.003  0.025  0.626  ICM OSCE  0.018  0.124  0.026  ICR Small Group Points  0.034  0.215  0.139  ICR Examination Points  0.018  0.231  0.128  View Large TABLE III. Regression Models of ICM and ICR Course Performances to Predict Variance in PGY-1 PD Survey Scores for Professionalism Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.010  0.098  0.194  0.076  ICM Preceptor  0.003  0.025  0.626  ICM OSCE  0.018  0.124  0.026  ICR Small Group Points  0.034  0.215  0.139  ICR Examination Points  0.018  0.231  0.128  Explanatory Variables  Unstandardized Regression Coefficient  Standardized Regression Coefficient  p  R2 Change  ICD NBME  0.010  0.098  0.194  0.076  ICM Preceptor  0.003  0.025  0.626  ICM OSCE  0.018  0.124  0.026  ICR Small Group Points  0.034  0.215  0.139  ICR Examination Points  0.018  0.231  0.128  View Large DISCUSSION Our study suggests that a combination of preclerkship clinical skills and clinical reasoning outcome measures can explain a small, but significant amount of the variance in internship ratings, accounting for variance in both evaluation form factors, namely medical expertise and professionalism performance. This is consistent with, and adds to, our previous findings that these same preclerkship outcome measures explain the variance in overall clerkship grades and average NBME examination performance (LaRochelle et al, unpublished data). Taken together, there is evidence regarding the importance of identifying students who struggle with clinical skills and clinical reasoning early in the preclerkship period as these issues often persist into the clerkship period and beyond into residency training. There are also longitudinal markers in the preclerkship time period (NBME subject examination and OSCE) that predicted poor performance during residency training. As our study suggests, the identification of struggling students in the preclerkship time frame can be accomplished through the integration of several key clinical skills and clinical reasoning outcome measures, and we believe that assessing reasoning and clinical skill performance not be relegated to any single assessment. In addition, our findings suggest that these assessments should optimally be coordinated, and occur longitudinally. Since clerkships have traditionally been a poor setting for basic skills training and medical students are being observed by faculty educators less frequently on the clerkships, remediation should first occur in the preclerkship period in the context of skills improvement with an emphasis on mastering an agreed upon set of foundational clinical skills and clinical reasoning abilities.11,15 Future studies should address the impact of prescribed remediation efforts on future student performance to break the cycle of poor performance. This study has several limitations. First, this is a single institution, and the findings may not necessarily be generalizable to other medical schools. However, this study does include 3 years of graduating medical students and has the additional benefit of tracking clinical skills and clinical reasoning performance into their PGY-1 training using a form with previously collected reliability and validity evidence. Second, the reliability of some of the preclerkship outcome measures is not well defined and may impact the overall regression model. However, the two most significant contributors to the regression model were the NBME subject examination and the OSCE, which are both reliable and valid measures of student performance. In conclusion, our study provides evidence regarding the importance of preclerkship courses in clinical skills and clinical reasoning, and further emphasizes the need for early identification of struggling learners in the context of a longitudinal curriculum in clinical skills and clinical reasoning. REFERENCES 1. Wenrich M, Jackson MB, Scherpbier AJ, Wolfhagen IH, Ramsey PG, Goldstein EA Ready or not? Expectations of faculty and medical students for clinical skills preparation for clerkships. Med Educ Online  2010; 15. 2. Prince KJ, Boshuizen HP, van der Vleuten CP, Scherpbier AJ Students' opinions about their preparation for clinical practice. Med Educ  2005; 39( 7): 704– 12. Google Scholar CrossRef Search ADS PubMed  3. Whipple ME, Barlow CB, Smith S, Goldstein EA Early introduction of clinical skills improves medical student comfort at the start of third-year clerkships. Acad Med  2006; 81( 10 Suppl): S40– 3. Google Scholar CrossRef Search ADS PubMed  4. Alexander EK Perspective: moving students beyond an organ-based approach when teaching medical interviewing and physical examination skills. Acad Med  2008; 83( 10): 906– 9. Google Scholar CrossRef Search ADS PubMed  5. Windish DM, Paulman PM, Goroll AH, Bass EB Do clerkship directors think medical students are prepared for the clerkship years? Acad Med  2004; 79( 1): 56– 61. Google Scholar CrossRef Search ADS PubMed  6. Hauer KE, Teherani A, Irby DM, Kerr KM, O'Sullivan PS Approaches to medical student remediation after a comprehensive clinical skills examination. Med Educ  2008; 42( 1): 104– 12. Google Scholar CrossRef Search ADS PubMed  7. Davis MH, Harden RM Planning and implementing an undergraduate medical curriculum: the lessons learned. Med Teach  2003; 25( 6): 596– 608. Google Scholar CrossRef Search ADS PubMed  8. Klamen DL, Borgia PT Can students' scores on preclerkship clinical performance examinations predict that they will fail a senior clinical performance examination? Acad Med  2011; 86( 4): 516– 20. Google Scholar CrossRef Search ADS PubMed  9. Jackson MB, Keen M, Wenrich MD, Schaad DC, Robins L, Goldstein EA Impact of a pre-clinical clinical skills curriculum on student performance in third-year clerkships. J Gen Intern Med  2009; 24( 8): 929– 33. Google Scholar CrossRef Search ADS PubMed  10. Omori DM, Wong RY, Aontonelli MA, Hemmer PA Introduction to clinical medicine: a time for consensus and integration. Am J Med  2006; 118: 189– 94. Google Scholar CrossRef Search ADS   11. Remmen R, Derese A, Schepbier A, et al.   Can medical schools rely on clerkships to train students in basic clinical skills? Med Educ  1999; 33( 8): 600– 5. Google Scholar CrossRef Search ADS PubMed  12. Corbett EC Jr, Elnicki DM, Conaway MR When should students learn essential physical examination skills? Views of internal medicine clerkship directors in North America. Acad Med  2008; 83( 1): 96– 9. Google Scholar CrossRef Search ADS PubMed  13. Dong T, Durning SJ, Gilliland W, Swygert K, Artino AR Development and initial validation of a program director's evaluation form for medical school graduates. Mil Med  2015; 180( 4 Suppl): 97– 103. Google Scholar CrossRef Search ADS PubMed  14. LaRochelle JS, Durning SJ, Pangaro LN, Artino AR, van der Vleuten CP, Schuwirth L The Effect of increasing authenticity of instructional format on clinical reasoning performance in a pre-clerkship medical student course: a prospective randomized crossover trial. Med Educ  2011; 45: 807– 17. Google Scholar CrossRef Search ADS PubMed  15. Howley LD, Wilson WG Direct observation of students during clerkship rotations: a multiyear descriptive study. Acad Med  2004; 79( 3): 276– 80. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.

USA, Brian A. Hemann, MC;PhD, Steven J. Durning, MD,;USA, William F. Kelly, MC;PhD, Ting Dong,;(Ret.), Louis N. Pangaro, MC USA;(Ret.), Paul A. Hemmer, USAF MC

doi: 10.7205/MILMED-D-14-00567pmid: 25850126

ABSTRACT Purpose: To determine whether the Uniformed Services University (USU) system of workplace performance assessment for students in the internal medicine clerkship at the USU continues to be a sensitive predictor of subsequent poor performance during internship, when compared with assessments in other USU third year clerkships. Method: Utilizing Program Director survey results from 2007 through 2011 and U.S. Medical Licensing Examination (USMLE) Step 3 examination results as the outcomes of interest, we compared performance during internship for students who had less than passing performance in the internal medicine clerkship and required remediation, against students whose performance in the internal medicine clerkship was successful. We further analyzed internship ratings for students who received less than passing grades during the same time period on other third year clerkships such as general surgery, pediatrics, obstetrics and gynecology, family medicine, and psychiatry to evaluate whether poor performance on other individual clerkships were associated with future poor performance at the internship level. Results for this recent cohort of graduates were compared with previously published findings. Results: The overall survey response rate for this 5 year cohort was 81% (689/853). Students who received a less than passing grade in the internal medicine clerkship and required further remediation were 4.5 times more likely to be given poor ratings in the domain of medical expertise and 18.7 times more likely to demonstrate poor professionalism during internship. Further, students requiring internal medicine remediation were 8.5 times more likely to fail USMLE Step 3. No other individual clerkship showed any statistically significant associations with performance at the intern level. On the other hand, 40% of students who successfully remediated and did graduate were not identified during internship as having poor performance. Conclusions: Unsuccessful clinical performance which requires remediation in the third year internal medicine clerkship at Uniformed Services University of the Health Sciences continues to be strongly associated with poor performance at the internship level. No significant associations existed between any of the other clerkships and poor performance during internship and Step 3 failure. The strength of this association with the internal medicine clerkship is most likely because of an increased level of sensitivity in detecting poor performance. INTRODUCTION Predicting performance within medical school and into subsequent residency training is of significant interest to clerkship and program directors. Several prior studies have revealed associations among student performance measures at the clerkship level with high-performance during residency.1,–3 Other studies have explored performance measures during medical school that predict poor performance in residency.4,–6 In 1998, Lavin et al described a significant association of poor third year internal medicine clerkship performance with aggregate ratings on program director surveys of interns.4 In that study, a questionnaire (PGY-1 Survey) was sent to internship directors. It was found that students who required remediation of performance following their third year internal medicine clerkship, a judgment that was reached by a competency committee,7 were 12.9 times more likely to have low internship performance ratings and 9.4 times more likely to receive unfavorable comments from their internship director when compared to students who completed 4th year internal medicine rotations that were not remediation experiences. The assessment methods used in the Medicine clerkship were also found to be more predictive of future performance than for all clerkships combined in other departments using more conventional methods.4 The present study is a follow-up study to the original work published by Lavin et al. and we sought to determine if our methods for classifying students' performance in the third year internal medicine clerkship continues to have predictive validity, and specifically whether poor clerkship performance is associated with future poor performance at the internship level. We felt that a newer and expanded study would add to the literature in several ways. First, in the interim since the first analysis, other clerkships at our institution implemented a competency committee review of student performance as well as National Board of Medical Examiners (NBME) subject examinations as a part of their evaluation process. Second, we also sought to include an additional measure required for licensure, U.S. Medical Licensing Examination (USMLE) Step 3 performance. Third, we have changed from a 12-week inpatient to a 6 week ambulatory plus 6 week inpatient ward clerkship, which could impact the prior associations. Fourth, since the original analysis, the PGY-1 Survey has been shown to demonstrate additional reliability and validity evidence and has undergone further revision since the first study.8,9 Factor analysis of this instrument has shown that the specific items on the program director evaluation form load onto the domains of expertise and professionalism. In this current study, we employed a retrospective cohort design and we hypothesized that we would see a strong association of low performance ratings on the internal medicine clerkship with poor performance at the PGY-1 level including USMLE Step 3 performance. Furthermore, we now expected to see similar relationships between poor performance in other specialties' third year clerkships and poor performance at the PGY-1 level. METHOD Study Context and Participants This study was part of the larger Long-Term Career Outcome Study conducted at the F. Edward Hébert School of Medicine, Uniformed Services University (USU), and has been approved by the USU Institutional Review Board. As the United States' federal academic health center, USU matriculates approximately 170 medical students annually and, at the time of this study, offered a traditional 4-year curriculum: 2 years of basic science courses followed by 2 years of clinical rotations (clerkships). The cohort of the present study consisted of USU students graduating between 2007 and 2011 (N = 853). Measures Classification of Students as Needing Remediation (or of Unsuccessful Clerkship Performance) Internal Medicine Clerkship Performance Core clerkship medical students undergo a rigorous evaluation process during their 12 week internal medicine (IM) clerkship at USU. At seven military treatment facilities throughout the continental United States, the students are evaluated using a three-level system. At the level of student expectations, we communicate with learners and teachers using the RIME (Reporter, Interpreter, Manager, Educator) framework.10 In order to receive a “pass” in the clerkship, students must be consistent and reliable Reporters and demonstrate some progress toward Interpreter. Secondly, in order to calibrate teacher observations of students, formal evaluation sessions are used every 3 weeks, during which the site director meets with all clinical teachers to discuss a student's performance, followed by feedback to the student.11 Our teaching faculty make recommendations regarding at what level the student is currently performing. Any student who receives a less than passing recommendation from any teacher, fails the NBME subject examination (defined during the study period as an NBME score of 64 or lower), or has concerning comments from any teacher who had meaningful clinical interaction with the student (regardless of whether or not the teacher recommended an overall less than passing evaluation), is referred to the Department of Medicine Education Committee (DOMEC). The DOMEC provides the third tier in the system, and the entire process has demonstrated consistency across clerkship directors and clerkship sites over time.12 DOMEC membership includes all clerkship site directors, as well as stakeholders from preclinical courses, clerkship, advanced clerkships, residency program, and hospital leadership. The DOMEC carefully reviews the entirety of the record of performance for each student referred and recommends a final grade for the student to the clerkship director and department chair.12 Students judged to have less than passing performance are required to complete additional clinical work at either the third and/or fourth year student level, repeat (and pass) written examinations, or both, in order to remediate their performance. During the study period, we used a traditional letter grade system, and less than passing grades could include Incomplete, C (-), D, or F (Fail). The grade of C(-) was an intra-departmental designation that allowed us to require additional clinical work during the fourth year (as there was no required internal medicine clerkship or sub-internship in the fourth year) but did not result in the student being presented to our Student Promotions Committee; this grade designation would indicate concern about a student's performance but not sufficient to warrant a grade with certain consequences for the student. For the purpose of the current study, we classified internal medicine students into two groups: those students, who received a less than passing grade and were required to return to the department of medicine to address areas of concern, were labeled as “remediators,” and students who received passing grades and required no remediation. For purposes of this study students at the clerkship level were not subdivided into causes of failure. Medicine clerkship learning objectives have remained constant since the prior study,4 but there has been some evolution of requirements and administration. Six direct observations of student history or physical examination skills had to be documented. More recently, passing (NBME score ≥65) but poor performance (<10th national percentile) on the Medicine NBME subject examination also prompted DOMEC review. Finally, one of our low stakes multiple choice question end-of-clerkship examination was replaced with a script concordance examination. However, no changes were made to the process of descriptive evaluation, use of the RIME framework, or use of evaluation sessions across sites. Performance on the Other Five Clerkships Students rotated in five other required clerkships—family medicine (6 weeks), surgery (12 weeks), psychiatry (6 weeks), pediatrics (6 weeks), and obstetrics and gynecology (6 weeks). We created a dichotomous variable to indicate whether a student received a passing or less than passing grade (C-, D, F) as a final grade for each clerkship. During the time of the study, the Pediatrics clerkship had adopted a variation on RIME known as PRIME (Professionalism, Reporter, Interpreter, Manager, and Educator),13 and Family Medicine, and Obstetrics and Gynecology had a committee process for reviewing student performance. None of the clerkships during the study period were using a real-time discussion of student performance such as the evaluation sessions used in the medicine clerkship. Outcome Measures PGY-1 Program Director Evaluation (PGY-1 PD) We collect PGY-1 PD performance data of our medical school graduates annually from program directors who oversee the training of military medical trainees. Each spring, we identified military treatment facilities (and some nonmilitary training programs) where our interns and residents are trained, and mailed evaluation forms for each trainee to the respective Graduate Medical Education (GME) directors. We asked GME directors to distribute the forms to their program directors (PDs) and then return the completed forms in envelopes or via emailed attachment. Forms we received were entered into a database. It should be noted that we updated the PGY-1 evaluation form during the 2008 to 2009 academic year. Consequently, for the cohort under investigation, interns from the graduating Class of 2007 were all evaluated using the old form, interns from Classes of 2008 and 2009 were evaluated using either the old form or the new form, and interns from Classes of 2010 and 2011 were all evaluated using the new form. Both the old and new PGY-1 PD evaluation forms were validated in previous studies.8,9 In the old form, 13 items were loaded on a factor labeled “Expertise” and 10 items on the other factor labeled “Professionalism”. The new form was based on the old one and took into account regrouping of the questions and given the introduction of the newly constructed core competencies defined by the Accreditation Council for Graduate Medical Education. In this form, 45 items converged to five factors—“Expertise” (17 items), “Military-unique Practice” (11 items), “Professionalism” (7 items), “System-based Practice” (6 items), and “Communication and Interpersonal Skills” (4 items). For the present study, we used the expertise and professionalism factors from both forms. All items in both forms were to be rated on a 5-point Likert scale. For example, a PD would rate an intern's “Ability to appreciate a patient's illness in the context of their life” as “Unacceptable,” “Significantly below most PGY-1s,” “On par with most PGY-1s,” “Better than most PGY-1s,” or “Consistently at least one level higher than almost all PGY-1s”. Poor Internship Performance Using the PGY-1 survey as the outcome measure, we used three different criteria to classify poor performance on either the “Expertise” or “Professionalism” Factor. The first criterion was receiving a “1” or “2” on any single item within the associated factor. The second criterion was if the factor score for either Expertise or Professionalism was one standard deviation or more below the mean. The third criterion was if the Expertise or Professionalism factor score was two standard deviations or more below the mean. We calculated a factor scores by averaging the ratings of all the PGY-1 PD survey items under that factor. USMLE Step 3 The USMLE is a single program consisting of four separate examinations designed to assess an examinee's understanding of and ability to apply concepts and principles that are important in health, disease, and effective patient care. USMLE Step 3 is taken in the PGY-1 and places the emphasis on whether examinees can apply medical knowledge and understanding of the biomedical and clinical science essential for the unsupervised practice of medicine, and the overall USMLE Step 3 pass/fail result was used as an outcome variable. Statistical Analyses We conducted descriptive statistics of the study cohort. We conducted two-way contingency table analyses (or cross-tabulations) to examine the strength of associations between IM clerkship poor performance and the three measures of internship poor performance, and between IM clerkship poor performance and USMLE Step 3 failure. We repeated the contingency table analyses for all other five clerkships, both individually and in aggregate (i.e. collapsing the poor performance indicators across the five clerkships). Using these contingency tables, we compared the observed frequencies in each cell with the expected frequencies. Expected frequencies were calculated assuming the null hypothesis that there was no association between poor performance during clerkship and internship. A larger and consistent discrepancy between observed and expected frequencies would indicate a higher likelihood of rejection of the null hypothesis. A chi-square (χ2) test of independence was performed against the null hypothesis. However, whenever a cell count of five or smaller occurred, we reported Fisher's Exact Test result since chi-square test assumed large-sample approximation of chi-square distribution. RESULTS For the graduating classes of 2007 through 2011, 102 of 853 students (12%) were referred to the DOMEC for review of grade recommendations that were less than passing, for NBME subject examination failure or a score ≤10th percentile, and/or for comments from evaluators that were concerning to the clerkship and/or site director. After review, 25 of the 102 referred students (3% overall) were assigned a less than passing grade and were required to complete additional remediation. The overall response rate of the PGY-1 evaluation form was 81% (689/853), with year-to-year response rates ranging from 73 to 90%. For the specific cohort of 25 requiring remediation, the response rate of the PGY-1 evaluation was 76% (19/25). Two-Way Contingency Table Table I shows the results of two-way contingency table analysis between students requiring remediation from the IM clerkship and those who did not and poor performance in internship by four classification criteria (1 or 2 on any survey item within a factor of Expertise or Professionalism, 1 standard deviation below the mean or lower on a factor, 2 standard deviations below the mean or lower on a factor), and USMLE step 3 failure. There was a statistically significant association between internal medicine remediators and expertise and professionalism. Most striking is the association between receiving a 1 or 2 on any rating in the domain of professionalism (χ2 = 31.77, df = 1, P < 0.01) with an odds ratio of 9.49 (95% CI = [3.74, 24.11]), meaning that the odds of showing poor performance in the domain of professionalism are 9.49 times higher for those who performed poorly in internal medicine compared with those who did not. The sensitivity of requiring IM remediation on poor “Professionalism” was 45% and the specificity was 92% (See Fig. 1). TABLE I. Contingency Table Analysis Between Poor Performance in Internal Medicine Clerkship and Poor Performance in PGY-1 Outcome Measure  Remediators (N = 25)  Non Remediators (N = 776)  Likelihood Ratio χ2  Number  %  Number  %  1 or 2 Rating   Med Expertise  5/20  25  49/567  8.6  4.5   Professionalism  9/20  45  45/567  7.9  18.7  1 SD Below Mean   Med Expertise  5/20  25  40/566  7.1  5.8   Professionalism  7/20  35  77/566  13.6  5.6  2 SD Below Mean   Med Expertise  2/20  10  4/566  0.7  6.3   Professionalism  3/20  15  10/566  1.8  7.3  USMLE Step 3 Failure  4/19  21  18/577  3.1  8.5  Outcome Measure  Remediators (N = 25)  Non Remediators (N = 776)  Likelihood Ratio χ2  Number  %  Number  %  1 or 2 Rating   Med Expertise  5/20  25  49/567  8.6  4.5   Professionalism  9/20  45  45/567  7.9  18.7  1 SD Below Mean   Med Expertise  5/20  25  40/566  7.1  5.8   Professionalism  7/20  35  77/566  13.6  5.6  2 SD Below Mean   Med Expertise  2/20  10  4/566  0.7  6.3   Professionalism  3/20  15  10/566  1.8  7.3  USMLE Step 3 Failure  4/19  21  18/577  3.1  8.5  All comparisons in the table achieved statistical significance. View Large TABLE I. Contingency Table Analysis Between Poor Performance in Internal Medicine Clerkship and Poor Performance in PGY-1 Outcome Measure  Remediators (N = 25)  Non Remediators (N = 776)  Likelihood Ratio χ2  Number  %  Number  %  1 or 2 Rating   Med Expertise  5/20  25  49/567  8.6  4.5   Professionalism  9/20  45  45/567  7.9  18.7  1 SD Below Mean   Med Expertise  5/20  25  40/566  7.1  5.8   Professionalism  7/20  35  77/566  13.6  5.6  2 SD Below Mean   Med Expertise  2/20  10  4/566  0.7  6.3   Professionalism  3/20  15  10/566  1.8  7.3  USMLE Step 3 Failure  4/19  21  18/577  3.1  8.5  Outcome Measure  Remediators (N = 25)  Non Remediators (N = 776)  Likelihood Ratio χ2  Number  %  Number  %  1 or 2 Rating   Med Expertise  5/20  25  49/567  8.6  4.5   Professionalism  9/20  45  45/567  7.9  18.7  1 SD Below Mean   Med Expertise  5/20  25  40/566  7.1  5.8   Professionalism  7/20  35  77/566  13.6  5.6  2 SD Below Mean   Med Expertise  2/20  10  4/566  0.7  6.3   Professionalism  3/20  15  10/566  1.8  7.3  USMLE Step 3 Failure  4/19  21  18/577  3.1  8.5  All comparisons in the table achieved statistical significance. View Large FIGURE 1. View largeDownload slide Predictive validity of poor performance in IM clerkship and PGY-1 outcomes. FIGURE 1. View largeDownload slide Predictive validity of poor performance in IM clerkship and PGY-1 outcomes. We repeated the contingency table analyses for all other five clerkships, independently and in aggregate. The results indicated that no significant associations existed between any of the other clerkships and poor performance during internship and Step 3 failure, likely a result of the low number of students who were assigned less than passing grades in the other clerkships which ranged from 0 to 11. By aggregating the poor performance indicators of the five clerkships, we only found a significant association between aggregated poor performance indicators of the five clerkships and internship “Expertise” poor performance with the criterion of scoring “1” or “2” on the factor (P < 0.05). (Fig. 2) No other significant associations were found between less than passing performance in the other clerkships and the outcomes of interest. FIGURE 2. View largeDownload slide PGY-1 performance of Int Med remediators (n = 25) vs. all students <pass grade from other clerkships combined (n = 22). FIGURE 2. View largeDownload slide PGY-1 performance of Int Med remediators (n = 25) vs. all students <pass grade from other clerkships combined (n = 22). DISCUSSION In 1998, Lavin and Pangaro demonstrated that unsatisfactory internal medicine clerkship performance was strongly associated with future poor performance at the internship level, and that the methods of the internal medicine clerkship were better at identifying such students when compared to all other specialty-based clerkships at USU combined. In the present study, we sought to determine how durable this association is for the internal medicine clerkship 13 years later, to explore findings with an additional outcome measures including performance on USMLE Step 3, ratings on additional criteria for poor performance within the domains of Expertise and Professionalism on program director ratings of intern graduates of USU, and to assess if changes to clerkships since the first study resulted in any change in ability to detect poor performance in internship. Since the original publication, other clerkships at our institution have adopted an education committee review process to determine end of clerkship grades for at risk students, and all clerkships now use NBME subject examinations as part of their assessment process. Our expectation was that other clerkships' using the education committee oversight of the process would increase the detection rate of underperforming students and demonstrate stronger associations with future poor performance at the internship level as well. This turned out to be incorrect. The present study builds on the prior study by examining the outcomes of program directors' ratings of PGY-1 residents in the domains of domains of medical expertise and professionalism using a validated survey instrument. In the present study, we have again demonstrated that unsuccessful clinical performance in the USU third year internal medicine clerkship remains strongly associated with future poor performance in internship, specifically in the domains of medical expertise and professionalism. The strength of the association in the domain of professionalism was particularly striking with internal medicine clerkship “remediators” being 18 times more likely to garner adverse ratings on professionalism during internship. Further, these students were 8.5 times more likely to fail USMLE Step 3 than nonremediators. In the present analysis, we asked the question whether or not unsatisfactory performance in any other individual clerkship (such as pediatrics, general surgery, obstetrics and gynecology, psychiatry, or family medicine) carries an association with future performance at the postgraduate level. As the results above show, no other USU third year clerkship demonstrated a significant correlation with poor performance at the PGY-1 level, nor did they correlate with USMLE Step 3 failure. It could be argued that the longer duration of the IM clerkship could explain the predictive value (especially given the low threshold for referral to committee review) but using general surgery as a comparative group, internal med had more than twice as many less than passing grades during the study period. Further, we also compared the internal medicine clerkship was compared against all other clerkships combined which represents a much greater period of time for observation (effectively 36 weeks vs. 12 weeks). The primary reason for the discrepancy between internal medicine and all other clerkships at our institution in predicting poor performance during internship is due to the increased sensitivity in detecting students whose performance is unsatisfactory. For the study period, 25 students were assigned a less than passing grade for the internal medicine clerkship compared to only 13 students for the other five clerkships combined. We feel that the increased sensitivity in the medicine clerkship remains because of an evaluation system that includes a process that values real-time, in-clerkship, discussions between a trained site director and teachers working with students that allow one to calibrate the assessments of individual teachers to the assessment framework (RIME). As a result, this enhances the identification of students about whom there is concern for subsequent discussion at a clinical competence committee (DOMEC). Importantly, members of the DOMEC include the site directors who run the evaluation sessions at the distant sites, such that the site directors themselves are trained at the quarterly reviews of student performance in the assessment framework and minimum expectations of “success.” Although other clerkships at USU have adopted a competency committee review of students, none has a similar program of assessment as described above. As a result, internal medicine had a significantly higher number of below passing grades compared to all other specialties. This has resulted in an increased sensitivity of unsuccessful performance in internal medicine and poor performance at the internship level. Given the outcomes during PGY-1, we would argue that these presented true positives rather than false positives. What elements in our assessment system might explain the enhanced detection of marginal or at risk students? The first is that at the level of teacher-student interaction, the terminology of our evaluation RIME framework is easily understood by teachers.14 Further, teachers do not “give” a failing grade so much as make an educational diagnosis reflecting the student's own performance; we ask teachers for their summary judgment of a student's progress in RIME but this is a “recommendation” and not the assignment of a grade. At the second level of the assessment system, our practice is to have face-to-face, formal evaluation sessions led by the clerkship site directors, in which a conversation about individual students calibrates the teachers' expectations and achieves consistency in assessment.15 The value of group/committee discussion for students who may be exhibiting poor performance during the clerkship has been well documented in prior studies.16,–18 Parenti et al showed improved ability to detect marginally performing students by employing evaluation sessions.18 In 2000, Hemmer et al demonstrated that face-to-face formal evaluation sessions improved the detection of unprofessional behavior on the internal medicine clerkship.17 In addition to the increased sensitivity the above studies have demonstrated with face to face evaluation sessions, there is a consistent opportunity to provide faculty development in evaluation.16 The threshold for reviewing students at the competency committee includes not only less than passing recommendations from faculty or poor NBME shelf examination performance, but also comments (whether in writing or made verbally at an evaluation session) that the site director deems of sufficient concern to refer for review. As such, it capitalizes on faculty's willingness to say and discuss what they might not write down, but also relieves an individual faculty member's perception that they are responsible for “giving” a low grade. Finally, at the top of the assessment system, the review at the competency committee (DOMEC), which includes the clerkship site directors and residency program directors, uses the “wisdom of the group” to reach a judgment based on the entirety of the record, a process that gives society the benefit of the doubt.7 We believe that comprehensive nature of the system of assessment, which is only in place in the USU internal medicine clerkship, improves the detection of at risk students. Nevertheless, the overall sensitivity for detecting students during clinical clerkships who will have problems in internship remains low, being 25% of graduates with poor ratings during PGY-1 for expertise, and 45% for professionalism (see Fig. 1), albeit notably improved with the process in the internal medicine clerkship. In other words, there are graduates who demonstrate deficiencies in internship that were not detected during the internal medicine clerkship. There may be several explanations for this low sensitivity. First, not all issues with expertise or professionalism evident in internship will be manifest during medical school; our analysis assumes that the reports from internship directors are the “truth” about an individual, and that clinical clerkships are a time to “screen” for this existing “truth,” which may, in fact, not be present. In other words, the student could have developed personal health, social or other issues after they clerkship year. Second, although the assessment system in the internal medicine clerkship is rigorous and focuses on frame of reference training and discussions among faculty and site directors, the conceptual, social and available time barriers may still affect the faculty's ability to identify at-risk students. Third, we only examined the PGY-1 outcomes for those students presented to DOMEC who were required to remediate, and thus found deficient in some way. It may be helpful to look at all students presented to DOMEC and determine if this alters the sensitivity and specificity. Fourth, we did not stratify our analysis by choice of residency or ultimate match results for internship. It is possible that the associations may have been different depending on a student's career goals or match results. Finally, it would be of interest to know whether including “all” students from the clerkship year, identified by “any” clerkship, would improve the identification of students at risk. We recognize that even if students are identified by the DOMEC review process and are required to remediate, it does not mean they will struggle in internship, and this impacts the difficult decisions about allowing identified students to progress toward medical school graduation. Hopefully, remediation on subsequent clinical rotations will have been successful. For those “remediators” about whom there was data, 15/19 passed USMLE Step 3, although a 21% failure rate on Step 3 should give one pause. Furthermore, approximately 40% (6/15 who had both PD evaluations and Step 3 scores) of the “remediators” passed Step 3 and had acceptable PD ratings. It would be important to review each student's record to more fully understand the nature of the areas of concern that prompted remediation, or to determine the degree of improvement demonstrated from the internal medicine clerkship to their remediation, as this might help identify factors associated with “success.” Nevertheless, it bears noting that the specificity is very high for future problems in internship if one is required to remediate following a competency committee review. These data serve as further validation of our student assessment and review process. And while it is helpful to note that our increased detection rate of underperforming students is an accurate one, this has significant implications for the effort in remediation. Increasing numbers of students detected with poor performance in turn increases the number of students requiring labor-intensive remediation programs. In our current system, students' required remediation was typically additional clinical work at the clerkship and/or advanced clerkship level, and could have been from 4 to 12 weeks in duration. From the present study, we cannot determine whether the prescribed remediation sufficiently address the problems detected by the initial review process, although reviews of remediation and self-regulated learning suggest there is an opportunity to make remediation more effective.19 There are limitations to this study. It was a retrospective, cohort study, looking at a small number of students who required remediation. The program director survey outcome data relies on ratings and/or comments provided by the internship director. Although the overall survey response rates were good, we did not have complete data for all in the cohort. Although the outcome we used was internship performance, which by its proximity to medical school makes it more likely that what was identified in internship would have its roots in medical school, we cannot make any statement about longer term outcomes of success for these students. This is one factor that has led us to implement an additional survey of program directors for our graduates during their PGY-3 year. Further, internal medicine failures and failures in other clerkships were not stratified by causes of failure such as “cognitive” issues where medical knowledge and expertise were deficient or “noncognitive” issues such as poor professionalism. Finally, our findings are of associations only and do not imply causality. CONCLUSIONS The detection of unsatisfactory clinical performance in the internal medicine clerkship utilizing a rigorous evaluation process with committee review continues to be strongly associated with poor performance during internship both in clinical performance and in testing for licensure. In our analysis, the internal medicine clerkship outperformed all of the other specialties in predicting future poor performance. REFERENCES 1. Griffith CH 3rd, Wilson JF, Haist SA, et al.   Internal medicine clerkship characteristics associated with enhanced student examination performance. Acad Med  2009; 84( 7): 895– 901. Google Scholar CrossRef Search ADS PubMed  2. Peskun C, Detsky A, Shandling M Effectiveness of medical school admissions criteria in predicting residency ranking four years later. Med Educ  2007; 41( 1): 57– 64. Google Scholar CrossRef Search ADS PubMed  3. Andriole DA, Jeffe DB, Hageman HL, Whelan AJ What predicts USMLE Step 3 performance? Acad Med  2005; 80( 10 Suppl): S21– 4. Google Scholar CrossRef Search ADS PubMed  4. Lavin B, Pangaro L Internship ratings as a validity outcome measure for an evaluation system to identify inadequate clerkship performance. Acad Med  1998; 73( 9): 998– 1002. Google Scholar CrossRef Search ADS PubMed  5. Greenburg DL, Durning SJ, Cohen DL, Cruess D, Jackson JL Identifying medical students likely to exhibit poor professionalism and knowledge during internship. J Gen Intern Med  2007; 22( 12): 1711– 7. Google Scholar CrossRef Search ADS PubMed  6. Papadakis MA, Teherani A, Banach MA, et al.   Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med  2005; 353( 25): 2673– 82. Google Scholar CrossRef Search ADS PubMed  7. Gaglione MM, Moores L, Pangaro L, Hemmer PA Does group discussion of student clerkship performance at an education committee affect an individual committee member's decisions? Acad Med  2005; 80( 10 Suppl): S55– 8. Google Scholar CrossRef Search ADS PubMed  8. Durning SJ, Pangaro LN, Lawrence LL, Waechter D, McManigle J, Jackson JL The feasibility, reliability, and validity of a program director's (supervisor's) evaluation form for medical school graduates. Acad Med  2005; 80( 10): 964– 8. Google Scholar CrossRef Search ADS PubMed  9. Dong T, Durning SJ, Gilliland WR, Swygert K, Artino AR Development and initial validation of a program director's evaluation form for medical school graduates. Mil Med  2015; 180( 4 Suppl): 97– 103. Google Scholar CrossRef Search ADS PubMed  10. Pangaro L A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med  1999; 74( 11): 1203– 7. Google Scholar CrossRef Search ADS PubMed  11. Hemmer PA, Grau T, Pangaro LN Assessing the effectiveness of combining evaluation methods for the early identification of students with inadequate knowledge during a clerkship. Med Teach  2001; 23( 6): 580– 4. Google Scholar CrossRef Search ADS PubMed  12. Durning SJ, Pangaro LN, Denton GD, et al.   Intersite consistency as a measurement of programmatic evaluation in a medicine clerkship with multiple, geographically separated sites. Acad Med  2003; 78( 10 Suppl): S36– 8. Google Scholar CrossRef Search ADS PubMed  13. Holmes AV, Peltier CB, Hanson JL, Lopreiato JO Writing medical student and resident performance evaluations: beyond “performed as expected.” Pediatrics  2014; 133( 5): 766– 8. Google Scholar CrossRef Search ADS PubMed  14. Pangaro L, ten Cate O Frameworks for learner assessment in medicine: AMEE Guide No. 78. Med Teach  2013; 35( 6): e1197– 210. Google Scholar CrossRef Search ADS PubMed  15. Noel GL A system for evaluating and counseling marginal students during clinical clerkships. J Med Educ  1987; 62( 4): 353– 5. Google Scholar PubMed  16. Hemmer PA, Pangaro L The effectiveness of formal evaluation sessions during clinical clerkships in better identifying students with marginal funds of knowledge. Acad Med  1997; 72( 7): 641– 3. Google Scholar CrossRef Search ADS PubMed  17. Hemmer PA, Hawkins R, Jackson JL, Pangaro LN Assessing how well three evaluation methods detect deficiencies in medical students' professionalism in two settings of an internal medicine clerkship. Acad Med  2000; 75( 2): 167– 73. Google Scholar CrossRef Search ADS PubMed  18. Parenti CM A process for identifying marginal performers among students in a clerkship. Acad Med  1993; 68( 7): 575– 7. Google Scholar CrossRef Search ADS PubMed  19. Durning SJ, Cleary TJ, Sandars J, Hemmer P, Kokotailo P, Artino AR Perspective: viewing “strugglers” through a different lens: how a self-regulated learning perspective can help medical educators with assessment and remediation. Acad Med  2011; 86( 4): 488– 95. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.

MD, Vincent F. Capaldi,;PhD, Steven J. Durning, MD,;MD, Louis N. Pangaro,;MD, Rosalie Ber,

doi: 10.7205/MILMED-D-14-00564pmid: 25850127

ABSTRACT Background: Expertise in clinical reasoning is essential for high-quality patient care. The Clinical Integrative Puzzle (CIP) is a novel assessment method for clinical reasoning. The purpose of our study was to further describe the CIP, providing feasibility, reliability, and validity evidence to support this tool for teaching and evaluating clinical reasoning. Methods: We conducted a prospective, randomized crossover trial assessing the CIP in second-year medical students from a single institution. Feasibility was estimated through the time taken to complete a CIP during a CIP session and through comments from faculty developers. Reliability was addressed through calculating odd–even item reliability (split-half procedure) for grid questions within each CIP. Evidence for content, concurrent, and predictive validity was also measured. Results: 36 students participated in the study. Data suggested successful randomization of participants and nonparticipants. The CIP was found to have high feasibility, acceptable reliability (0.43–0.73 with a mean of 0.60) with a short time for CIP completion. Spearman–Brown correction estimated a reliability of 0.75 with completing two grids (estimated time of 50 minutes) and 0.82 for three grids (estimated time of 75 minutes). Validity evidence was modest; the CIP is consistent with clinical reasoning literature and the CIP modestly correlated with small group performance (r = 0.3, p < 0.05). Conclusions: Assessing clinical reasoning in medical students is challenging. Our data provide good feasibility and reliability evidence for the use of CIPs; validity data was less robust. INTRODUCTION Expertise in clinical reasoning is essential for high-quality patient care. Clinical reasoning involves establishing a diagnosis and treatment approach that is specific for a patient's circumstances and preferences; it pertains to nearly everything a physician does in practice. Much remains unknown regarding how expert performance in clinical reasoning is developed or maintained. There is also debate regarding whether clinical reasoning can be taught.1 Given these and other challenges, to include the fact that clinical reasoning is not directly observable with our standard assessment methods, it has been difficult to develop tools to evaluate clinical reasoning. In this report, we present preliminary psychometric data of a novel tool for assessing clinical reasoning called the Clinical Integrative Puzzle (CIP) and we compare these findings with commonly used measures to assess clinical reasoning followed by a discussion of clinical reasoning theories. Though there remains controversy regarding the assessment and teaching of clinical reasoning, we believe educational theories can help shed light on this debate. THEORETICAL FRAMEWORKS Studies in deliberate practice argue that to achieve expert performance in a domain (such as clinical reasoning), one has to spend many hours (approximately 10,000 hours) in effortful practice with the material, which at least initially should be under the guidance of a coach or mentor.2 Many scholars believe that for clinical reasoning this practice should entail the deliberate construction of illness scripts. Script theory3,–5 is germane to contemporary thinking on teaching and assessing clinical reasoning. A script can be thought of as a mental representation of the symptoms and findings that are seen with a diagnosis or illness. Indeed this application leads to the name “illness script” in medicine.6 Illness scripts are also believed to include the causal factors, pathogenesis, prognosis, and consequences for a disease. Further, illness scripts include a range of features that can be consistent with the illness. Script theory suggests that physicians use two systems for clinical reasoning. One system (System 1) is rapid, automatic, and its function is largely unconscious, and involves low cognitive effort. This rapid system is believed to entail script activation and is often referred to as nonanalytic reasoning, or pattern recognition. A physician can see a typical presentation of a disease and the diagnosis immediately comes to mind, or is activated, without much, if any, effort. Often, the physician arrives at the diagnosis through this nonanalytic reasoning. The second system (“System 2” or analytic reasoning) is slow and rule based, requires high cognitive effort and is consciously controlled. This system is believed at times to be involved in “script confirmation” when both system 1 and system 2 are used to arrive at a patient's diagnosis. In this circumstance, the physician actively compares and contrasts activated scripts from system 1 processes for a given patient presentation to arrive at the correct diagnosis. Limited studies suggest that physicians use both of these systems in caring for patients.7 There have been insights from fields outside of medical education regarding how scripts may develop. This literature from multiple fields2 proposes that expertise is an “adaptation”, and that experts expand and rearrange (i.e., adapt) their long-term memory through “chunking” or grouping multiple pieces of information together into large integrated cognitive units8 to enable processing of more information simultaneously than less advanced practitioners can do. This “chunking” of information is needed to cope with the limited number of independent pieces of information that the human brain can process in working memory at a time. Cognitive load theory addresses this limitation in human cognitive architecture and proposes that the number of interacting units (or “element interactivity”) is a key determinant of cognitive load.9 Additionally, a number of studies suggest that clinical reasoning is a not a generalized skill but rather is highly dependent on a relevant knowledge base and on the context of the encounter.10 Deliberate practice and script theories support this notion in that directed, explicit, focused effort in a domain is required, and involves high effort (System 2) for expertise to develop in clinical reasoning. Thus, these theories suggest that structured learning should improve clinical reasoning performance for a content domain. The CIP was first described by Ber, as a novel tool for the assessment of reasoning.11 The CIP uses a “grid” approach as outlined in Table I. Clinical diagnoses or syndromes are depicted on the horizontal rows, and findings from generic domains (e.g., history, physical examination, and laboratory findings) that will populate the illness scripts in the students' memories are depicted in the vertical columns. Next, a series of options for each cell in the grid are given. Trainees must match appropriate options with the rows and columns. The process allows for cross-referencing and reinforcement of concepts among a cluster of similar diagnoses with critical differences within and across domains. The rows generally involve similar syndromes or diagnostic entities that trainees often have difficulty differentiating, allowing trainees to practice comparing and contrasting of similar disease entities, allowing for, and perhaps requiring, “script confirmation” to take place. Groothoff12 have provided support for the construct validity of a slightly adapted version of the CIP, called MATCH (Measuring Analytic Thinking in Clinical Healthcare). At this stage of training, we would expect that students would mainly use the slower, “System 2” processing for these clinical reasoning activities. TABLE I. Sample CIP Grid on Thyrotoxicosis Disease  History  Physical  Labs  Pathology  RAIU  Treatment  Graves' Disease  1    7  13  19  25  31  Subacute Thyroiditis  2    8  14  20  26  32  Toxic MNG  3    9  15  21  27  33  Central Hyperthyroidism  4  10  16  22  28  34  T4 Ingestion  5  11  17  23  29  35  Struma Ovarii  6  12  18  24  30  36  Disease  History  Physical  Labs  Pathology  RAIU  Treatment  Graves' Disease  1    7  13  19  25  31  Subacute Thyroiditis  2    8  14  20  26  32  Toxic MNG  3    9  15  21  27  33  Central Hyperthyroidism  4  10  16  22  28  34  T4 Ingestion  5  11  17  23  29  35  Struma Ovarii  6  12  18  24  30  36  View Large TABLE I. Sample CIP Grid on Thyrotoxicosis Disease  History  Physical  Labs  Pathology  RAIU  Treatment  Graves' Disease  1    7  13  19  25  31  Subacute Thyroiditis  2    8  14  20  26  32  Toxic MNG  3    9  15  21  27  33  Central Hyperthyroidism  4  10  16  22  28  34  T4 Ingestion  5  11  17  23  29  35  Struma Ovarii  6  12  18  24  30  36  Disease  History  Physical  Labs  Pathology  RAIU  Treatment  Graves' Disease  1    7  13  19  25  31  Subacute Thyroiditis  2    8  14  20  26  32  Toxic MNG  3    9  15  21  27  33  Central Hyperthyroidism  4  10  16  22  28  34  T4 Ingestion  5  11  17  23  29  35  Struma Ovarii  6  12  18  24  30  36  View Large The purpose of our study was to further describe the CIP, providing psychometric data on this tool for teaching and evaluating clinical reasoning. As the CIP allows for deliberate comparing and contrasting of key features that comprise a diagnosis, it offers unique teaching and assessment moments. For example, this potentially allows for reduction in cognitive load by reducing the interaction of multiple pieces of the patient's presentation by answering questions within generic domains and by potentially facilitating adaptation and chunking through having learners work through the component parts to the diagnoses (completing cells in horizontal and vertical columns as opposed to just answering what is the most likely diagnosis). More specifically, we sought to establish the feasibility, reliability, and gather preliminary validity evidence for this emerging tool in medical education literature. In terms of validity, whereas multiple frameworks exist in the literature, we explored evidence for a number of commonly used arguments: content, construct and predictive validity. Our hypotheses were that the CIP would demonstrate evidence for feasibility, reliability, and validity. METHODS Study Context The Uniformed Services University (USU) is the United States' only federal medical school. Students graduating from USU then enter the same spectrum of specialties offered in the civilian community. During the study period, the curriculum was taught in a traditional approach with 2 years of preclinical courses followed by 2 years of clinical clerkships. The Introduction to Clinical Reasoning (ICR) Course occurs during the second year and was under the direction of one of the authors (SJD) during the study period. At the time of the study, the ICR course is a year-long course wherein students are introduced to a variety of reasoning processes by exploring a series of symptoms, physical examination findings, laboratory abnormalities, and syndromes. Generally speaking, each of the 30 topics has both a lecture and a case-based reasoning small group session. In the small group sessions, students work through 2 to 5 “paper cases,” which illustrate common diagnostic entities and key findings for the topic being discussed. During the study period, the sessions were 90 minutes in length and small group size ranged from 6 to 9 students per group. During these sessions, preceptors facilitate discussions by reviewing student answers to the cases and lead students through the clinical reasoning process (e.g., pointing out key terminology, pathophysiology, and decision points). There are assigned readings before the small groups and students are expected to arrive prepared to discuss the cases for the topic for the session. Following is an example excerpt from the introduction of a paper case. “A 60-year-old patient with hypertension, diabetes, and high cholesterol who presents with a three month history of progressive substernal chest pain. His physical examination is normal. Please list your differential diagnosis” The CIP sessions were conducted in addition to usual small group teaching sessions on topic areas—there were no other differences in terms of formal small group teaching instruction between those who completed a CIP and those who did not for topics with CIPs. Student evaluation measures were also the same regardless of CIP assignment. CIP Description The CIP resembles the extended-matching assessment, described initially by Case13 but extends upon this through its crossword puzzle–like appearance (Table I). This tool enables trainees to compare and contrast a group of related diagnoses in a variety of domains such as history, physical, histology, pathology, and/or laboratory patterns. The CIP can be either paper- or computer-based in format. It can also allow inclusion of images (such as radiographs and histology slides) and multiple other media formats, such as streaming audio files, which can be included in electronic CIPs. The tool also allows for integration of curriculum content through simultaneous inclusion of a variety of domains. A CIP allows for the division of a number of prototypical presentations into specific domains, which enables the learner to focus more attention on key discriminating findings. The study investigators created eight CIPs (5 × 5 to maximum of 6 × 6) for topics addressed in the course. Each CIP had a primary author. CIP topics were anemia, abdominal pain, dyspnea, chest pain, thyrotoxicosis, headache, and pediatric growth and development disorders [available upon request to the principal investigator (SJD)]. Each CIP was written by one primary author and then vetted with content and educational experts within their respective department. Primary authors then subsequently met with each other and further vetted each CIP, resulting in the CIPs used in this study. All participants underwent a 15-minute orientation session wherein the CIP was explained and a practice grid was completed with a nonmedical topic. CIP Sessions Each of the four CIP sessions lasted one hour in duration. Each CIP session is a small group teaching and assessment event for clinical reasoning, shaped by exercises with CIPs. More specifically, in each session, the task for students was to complete as many CIPs as possible. Participants worked independently and no instruction or feedback was given before the CIPs were completed within each session. Participants listed start and stop times on the CIP grid and time was also monitored by the research assistant. For each of the four sessions, there was a different “primary” CIP, i.e., the first CIP that all participants for the session received. Each of these sessions was monitored by a research assistant, and student participation was voluntary and involved informed consent. All students who agreed to participate were randomized to one of 6 groups on entry through a random-number generator. This was done as it was expected that not all students would sign up for this study and also to prevent students from completing CIPs only in topics of interest, which could confound our study finding correlations. Each of these six groups attended two of the four CIP sessions as illustrated below. The session numbers corresponded to time in the academic year (i.e., 1 = first quarter, 4 = last quarter). The six groupings were as follows: Group 1: CIP in session 1 and 2 Group 2: CIP in session 1 and 3 Group 3: CIP in session 1 and 4 Group 4: CIP in session 2 and 3 Group 5: CIP in session 2 and 4 Group 6: CIP in session 3 and 4 Students were assigned to participate in two of four total CIP sessions (above). The faculty involved in the CIP sessions did not teach any of the small group sessions. Measurements Measurements collected during the study period included baseline measures (collected before the CIP sessions), process measures (collected during the academic year), and outcome or after measures (collected at the end of the academic year). Baselines End-of-first-year Grade Point Average (GPA), grades in first-year courses, and Medical College Admissions Test (MCAT) scores. Process Performance on Introduction to Clinical Reasoning (ICR) Course 3 in-house multiple-choice examinations. Outcomes United States Medical Licensing Examination (USMLE) Step 1 (basic science knowledge) and internal medicine clerkship grade. We expected only small to moderate correlations with USMLE Step 1 performance as the construct of clinical reasoning is not fully captured in a multiple-choice test. Participants in our study also underwent unstructured exit interviews. Outcome measurements were collected to help measure the durability of effect of exposure to CIPs. This was felt to be particularly important given findings from transfer literature suggesting differences in durability between instructional modalities for complex tasks such as clinical reasoning.14 Participants All USU second-year medical students had the opportunity to participate in this study. There were no exclusionary criteria. All 160 students (both participants and nonparticipants in this study) received the same teaching and evaluation materials offered in the course; the only difference between participants and nonparticipants in our study was exposure to CIPs on selected topics. Students were notified of the opportunity for study participation by e-mail at the beginning of the academic year. Students who did not respond to the e-mail invitation received up to 3 electronic reminders by the research assistant. Students who expressed interest through e-mail invitations then received an overview presentation on the CIP by the research assistant. Following the overview presentation, students who agreed to participate signed an informed consent, were randomized to one of six groups, and given a unique identifier by the research assistant for data identification and analysis. The study was approved by the USU IRB. Statistical Analysis Feasibility was estimated through the time taken to complete a CIP during a CIP session and through comments from faculty developers. The time was listed by each participant on each CIP answer grid (start and stop time). The research assistant also timed participants during CIP sessions. Qualitative comments conducted by the research assistant from students on exit interviews also targeted feasibility of this tool. All responses were recorded (audio and/or written) and were analyzed by two members of the study team. Students and faculty were encouraged to make any suggestions or comments regarding the tool; the research assistant also asked specifically about the feasibility and value of this potential tool. Responses were analyzed by two of the study investigators for emergent themes and example comments. Reliability of the CIP was addressed through calculating odd–even item reliability (split-half procedure) for grid questions within each CIP. Odd–even reliability compares the performance of odd numbered items with even numbered items in the CIP cells as a measure of internal consistency of the tool and ensures a reasonable mix of items. CIP scores were calculated by adding the number of correct answers over all cells in a grid (i.e., maximum scores 25–36 per CIP, minimum score 0). Validity was assessed through a number of means. First, by comparing CIP content with end-of-year examinations and by discussing CIP content with both faculty (unstructured interviews) and participants (through unstructured exit interviews). Second, we assessed validity by comparing CIP method with current reasoning theory (content and construct). Third, we gathered validity evidence by comparing CIP performance with end of course performance in small groups and multiple-choice in-house examinations. Fourth, we compared CIP performance with USMLE examination and internal medicine clerkship grade as a means of assessing predictive validity. Finally, we gathered validity evidence through analysis of comments from trainees by unstructured exit interviews. We also recorded comments from exit interviews. These qualitative comments were used to help explain underpinnings of quantitative findings and to generate hypotheses. RESULTS A total of 36 participants completed CIPs in this study, 26 men and 10 women. There were no significant (all p > 0.05) “baseline” differences between the end-of-first-year GPA, grades in first-year courses, and MCAT scores between participants and nonparticipants at the USUHS during the academic year of the study. Feasibility Faculty reported that it took approximately 5 minutes for an instructor to create a cell in the grid, which compares favorably with the amount of time that it takes an instructor to write a multiple-choice question (MCQ). Thus a CIP grid with 25 cells (questions) would be estimated to take 2 hours. The mean time that it took a participant to complete the CIP was 25 minutes (range, 8–34 minutes). It took a participant approximately 1 minute to complete a question cell in the grid. Reliability Odd–even reliability per CIP ranged between 0.43 (growth and development disorders) and 0.73 (dyspnea, chest pain). The mean odd–even reliability across all CIPs was 0.60. If one uses the mean odd–even reliability (0.60) and Spearman–Brown correction, completing two 25-cell CIP grids would result in an estimated reliability of 0.75 and completing three 25-cell grids would result in a reliability of 0.82. Two 25-cell grids would require approximately 50 minutes (1 cell per minute) and three grids would require 75 minutes (1 question or cell in CIP per minute). Validity All the selected CIP topics (content) corresponded to the topics taught in the ICR Course. Validity evidence was gathered through interviews of faculty and exit interviews of students. These unstructured interviews provide evidence of this tool's content and construct validity. The majority of participants (n = 34) as well as all faculty involved in writing the CIPs (n = 6) endorsed high satisfaction with this tool. They endorsed that the content of the examination mirrored the content of the course as well as the clinical reasoning process. Sample quotes from faculty (quotes represent the 3 identified themes of making the clinical reasoning process explicit, integrating basic and clinical science assessment, and mirrors clinical practice, respectively) included: “Constructing a CIP compels faculty to focus on the key distinguishing features between a group of diagnoses; this is a different exercise than telling students all the common features seen in a diagnosis.” “CIPs enable the assessment of basic science and clinical science together which is very unique. I enjoyed choosing the domains to include and working with my basic science and clinical faculty on the topic.” “Taking care of patients is more than which of the following is the most likely diagnosis. Patients don't say I have a. CHF, b. COPD, c. etc. By reviewing the CIP with the student, I can see how they put the pieces of the puzzle together to arrive at their answer. I am going to use CIPs in other venues with trainees to include residents and faculty.” Examples of student comments from the unstructured exit interviews (n = 31 participants) are listed below and also provide evidence of content and construct validity. Student themes mirrored faculty themes (making the clinical reasoning process explicit, integrating basic and clinical sciences, and mirrors clinical practice). In terms of making the process more explicit, students noted that the CIP was a “unique format for teaching and evaluating reasoning,” “helps me connect the dots and arrive at a diagnosis,” “displays the intermediate steps to the diagnosis,” and “helps compare important elements of similar diagnoses.” In terms of the theme of integrating basic and clinical sciences, student stated the CIP “gives me a tool for reading about and contrasting textbook diagnoses” and “helps me put basic and clinical science information together.” Finally, in terms of mirroring clinical practice, example quotes included: “I prefer these to multiple-choice questions.” Patients do not come in saying “I have abdominal pain and please choose a diagnosis from A-E” and “please put these in the course, they are too valuable for study use only.” Further, all participants (n = 34) reported that the CIP reflected important clinical reasoning concepts and helped performance on the National Board of Medical Examiners examinations. Several students asked for additional CIPs to help prepare for examinations. No significant correlations emerged between the students' scores for each individual CIP and multiple-choice examinations of academic performance. Correlations for the following were all found to have an r < 0.2, and p > 0.05: first-year GPA, final grade in clinical reasoning course, IM clerkship grade. Significant small to moderate correlations were seen with small group performance (for all non-CIP-session topics) in the clinical reasoning course (r = 0.30, p < 0.05). DISCUSSION This study evaluated the feasibility, reliability and validity of the CIP in a pilot program at a single institution in a single academic year. Although the study did demonstrate evidence of feasibility, reliability, and validity, the evidence for the latter was the least robust in this small study with content and construct validity being provided through consistency with clinical reasoning theory as well as student and faculty qualitative comments. The correlation of CIP scores with small group grades from preceptors, whereby students display their clinical reasoning by working through paper cases of presentations of a variety of topics in clinical medicine, is an expected outcome and provides some evidence of construct validity. Further, the concordance of qualitative findings from unstructured interviews of faculty and students provides additional validity evidence. Given that the topics vary in content and scope, we believe this may be the reason that small to moderate correlations were found. The lack of association with MCQs is consistent with the notion that MCQs may be reflecting a different construct than clinical reasoning. The CIP is a tool for assessing clinical reasoning. The puzzle allows novice and more advanced learners to engage content in a format different from multiple-choice examinations and small-group case discussions that they usually encounter, where the typical focus is on the one most likely diagnosis for 1 single case at a time. The crossword puzzle format of the CIP allows students to put the component “parts” of a diagnosis together, explicitly compare and contrast symptoms and findings between diagnoses, and build basic illness scripts. Thus, the CIP approach is consistent with script theory, providing additional construct validity evidence with the design of brief descriptions in each block that enables the learner to focus on key information for each domain by comparing and contrasting findings vertically while composing illness scripts horizontally. According to studies by Bordage and Lemieux,15,16 medical learners do not organize medical knowledge in linear frameworks such as simple lists of signs, symptoms, and rules but rather develop a network of knowledge of abstract relationships. The CIP appears to represent a logical way of solidifying and organizing abstract clinical relationships, and allows comparing and contrasting between multiple diagnoses with the grid format. Bordage and Lemieux15,16 suggest that a major determinant of diagnostic competence is the ability to compare and contrast the signs and symptoms presented and relate these abstract qualities to stored memory structures and scripts. In addition to serving as an assessment format, the CIP appears to be a useful tool for helping learners to acquire, compare, and contrast strategies that support clinical reasoning. Further, the CIP can plausibly allow for unique testing and teaching of cognitive skills such as knowledge organization, discriminating key features,16 prioritization of and explicitly testing the discriminating key findings for a group of related diagnoses,17 semantic competence (or use of proper medical terminology),16 and encapsulation (establishing the intermediate steps to the diagnosis such as the syndrome)6 all of which have been proposed to improve clinical reasoning. Also, the CIP grid format reduces element interactivity (through limiting the amount of information contrasted for each grid—i.e., a section of the history as opposed to the entire history) and thus assists the learner with managing cognitive load. As medical students develop into physicians, their reliance on analytic reasoning decreases as they begin to “chunk” information and rely more on pattern recognition and their ability to draw on their experience during their years of training. This could represent one reason why we did not see an association with internal medicine clerkship grade though we suspect the lack of association more reflects our small sample size and the fact that the CIP represents only a small portion of the content covered in the internal medicine clerkship. We believe the CIP, as an analytic thinking tool, can be used for both teaching and assessment of clinical reasoning. Further, it can assist with developing this chunking and automatization through providing brief descriptions for each block or question as well as enabling horizontal and vertical comparisons on the CIP grid. The data from this study provide evidence for the feasibility and reliability of the CIP in assessing clinical reasoning in medical students. It takes students approximately 1 minute per item (cell) in the grid and the results of the test appear reliable. Additional research is needed with more participants to assess the validity of this test. Based on our data, completing two to three grids requires less than 1.5 hours for a participant to complete (with high stakes reliability results) and appears to take less time for faculty to construct than creating multiple-choice questions (30 minutes–1 hour per question for an MCQ, 5 minutes per question (or cell) for a CIP, 12.5–25 hours to create a 25-question MCQ, and an estimated 2 hours for a 25-question CIP) with similar reliability results. Our results are consistent with recent studies from others adding to the validity evidence of our findings.12 Additionally, speaking to the validity of the CIP is the tool's consistency with reasoning theory. The CIP was developed from these theories as a basis. Vertically, the participants compare and contrast domains, which may help develop encapsulations. Horizontally the participants build a “script” for a particular illness. There were several limitations to this study. The CIP was only studied in a single institution with a small number of participants. We suspect that this had major impact on our ability to assess the predictive validity of this assessment tool; the study was not powered to detect small effect sizes. A larger, multicenter trial would be better powered to assess a small effect. Future research may include multicenter trials of this tool at various time points during medical school and residency, comparable to what was done by Groothoff et al.12 We would predict that more advanced learners may be able to better accomplish the CIP tasks, and do so using, in part, the more rapid “system 1” processing of pattern recognition as well as a more robust experience base (system 1 and/or 2) to draw from. Specifically, their CIP examination scores would not only be higher, but would be accomplished in a much shorter time period per CIP. This could be studied in the future. Additional data and correlation studies with USMLE Step II and III board scores and in-service examinations should reveal whether the CIP does measure similar or different constructs than the factual and applied type of clinical knowledge usually tested. Reliability would likely be improved by having more potential answers than cells for each topic, reducing guessing. This has been suggested in prior studies.12 The challenge of assessing and developing clinical reasoning skills and finding a valid predictor of clinical excellence remains an elusive goal of medical education. More research is needed in this field to better establish and quantify the efficacy of assessment and developmental tools used for improving clinical reasoning in medical education today. As each student is unique, it is unlikely that a single tool exists that would help every learner equally well to develop expertise in clinical reasoning. The CIP represents a novel approach to clinical reasoning instruction and offers a unique view of assessment (within and across disciplines) that could generate helpful learner feedback. Our learners endorsed the value of this feedback and our psychometric data provided multiple arguments supporting its use in medical education. ACKNOWLEDGMENTS The authors thank Professor Olle T.J. Cate, PhD, for his helpful review and suggested revisions to the manuscript. REFERENCES 1. Ramaekers SPJ On the development of competence in solving clinical problems; Can it be taught? Or can it only be learned?  Available at http://igitur-archive.library.uu.nl/dissertations/2011-0825-203547/UUindex.html; accessed October 2, 2014. 2. Ericsson KA The Cambridge Handbook of Expertise and Expert Performance . Cambridge, New York, Cambridge University Press, 2006. Google Scholar CrossRef Search ADS   3. Charlin B, Tardif J, Boshuizen HP Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med  2000; 75: 182– 90. Google Scholar CrossRef Search ADS PubMed  4. Charlin B, Gagnon R, Pelletier J, et al.   Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel. Med Educ  2006; 40: 848– 54. Google Scholar CrossRef Search ADS PubMed  5. Charlin B, Boshuizen HP, Custers EJ, Feltovich PJ Scripts and clinical reasoning. Med Educ  2007; 41: 1178– 84. Google Scholar CrossRef Search ADS PubMed  6. Schmidt HG, Rikers RM How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ  2007; 41: 1133– 9. Google Scholar PubMed  7. Mamede S, Schmidt HG, Rikers RM, Penaforte JC, Coelho-Filho JM Breaking down automaticity: case ambiguity and the shift to reflective approaches in clinical reasoning. Med Educ  2007; 41: 1185– 92. Google Scholar CrossRef Search ADS PubMed  8. Gobet F Expert memory: a comparison of four theories. Cognition  1998; 66: 115– 52. Google Scholar CrossRef Search ADS PubMed  9. van Merrienboer JJ, Sweller J Cognitive load theory in health professional education: design principles and strategies. Med Educ  2010; 44: 85– 93. Google Scholar CrossRef Search ADS PubMed  10. Durning SJ, Artino AR, Boulet JR, Dorrance K, van der Vleuten C, Schuwirth L The impact of selected contextual factors on experts' clinical reasoning performance (does context impact clinical reasoning performance in experts?). Adv Health Sci Educ Theory Pract  2012; 17: 65– 79. Google Scholar CrossRef Search ADS PubMed  11. Ber R The CIP (comprehensive integrative puzzle) assessment method. Med Teach  2003; 25: 171– 6. Google Scholar CrossRef Search ADS PubMed  12. Groothoff JW, Frenkel J, Tytgat GA, Vreede WB, Bosman DK, ten Cate OT Growth of analytical thinking skills over time as measured with the MATCH test. Med Educ  2008; 42: 1037– 43. Google Scholar CrossRef Search ADS PubMed  13. Case S Extended-matching items: a practical alternative to free-response questions. TLM  1993; 5: 107– 15. 14. Bransford J, Brown A, Cocking R (editors): How People Learn. National Research Council Committee on Developments in the Science of Learning. Committee on Learning Research and Educational Practice . Washington, DC, National Academy Press, 2000. 15. Bordage G, Lemieux M Some cognitive characteristics of medical students with and without diagnostic reasoning difficulties. Res Med Educ  1986; 25: 185– 90. Google Scholar PubMed  16. Bordage G, Lemieux M Semantic structures and diagnostic thinking of experts and novices. Acad Med  1991; 66: S70– 2. Google Scholar CrossRef Search ADS PubMed  17. Yudkowsky R, Lowenstein T, Riddle J, Otaki J, Nishigori RH, Bordage G A hypothesis-driven physical exam for medical students: initial validity evidence. Med Educ  2009; 43: 729– 40. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.

USN, Mark B. Stephens, MC;MD, Grace Landers,;MA, Stephen W. Davis,;PhD, Steven J. Durning, MD,;PhD, Sonia J. Crandall,

doi: 10.7205/MILMED-D-14-00558pmid: 25850128

ABSTRACT This study examined a cohort of students attending the Uniformed Services University regarding their attitudes toward medical care in underserved populations. Using the previously validated Medical Student Attitudes Toward the Underserved (MSATU), repeated measures analysis of variance showed that student attitudes toward care in underserved populations was less favorable than limited national data at entry and declined over time (Mean MSATU total score Year 1: 46.2 [SD 10.95]; Year 4: 41.7 [SD 12.3] p < 0.01). Differences in medical school debt, exposure to underserved populations, and the definition of “service” in the context of active duty military status might explain some of our findings. Providing broad service learning opportunities within the curriculum could increase student exposure to underserved populations and strengthen the social contract between community and institution. INTRODUCTION According to U.S. Census Bureau estimates, nearly 1 in 6 Americans are uninsured.1 Low-income, minority, and other traditionally underserved populations are less likely to have adequate access to health care.2 With continued changes in population demographics across the United States, the need for medical care in underserved populations continues to expand. Concurrently, the United States is confronting a looming physician shortage, particularly in primary care. It is estimated that by 2020, nearly 100,000 additional physicians will be needed to provide adequate primary care for the U.S. population.3 These factors combine to make the need to find physicians willing to serve in underserved communities particularly acute. Medical schools have a social obligation to train competent health care professionals capable of providing high quality health care to all Americans.4 Students are often idealistic and eager to serve underserved populations when they enter medical school.5 Experiences that expose students to a diverse patient population help to develop professional and interpersonal skills necessary for a comprehensive and compassionate medical practice.6,7 Unfortunately, as students' progress through their medical education, issues within the “hidden curriculum” of medical education may negatively impact attitudes toward care for underserved patients.8 Specifically, prior work has shown that medical student attitudes toward underserved populations tend to decline after matriculation.9,10 For some institutions, work with underserved populations occurs in “free clinics” outside of the formal (graded) medical school curriculum. Work in clinics serving disadvantaged populations requires students to devote personal time to the activity and they often receive no formal academic grade or “credit.” Students can see this as extra work and, as a result, fail to take advantage of robust volunteer clinical opportunities in underserved areas.5 Specific factors have been found to predict who will engage with underserved populations in the future. These include a strong desire to practice in an underserved community before entering medical school and students who come from a minority background themselves.11 Currently, many medical students come from privileged backgrounds, with family incomes and educational levels well above national census averages.12 Although, physicians are generally sensitive to medical issues confronting underserved populations, they may also have a poor image of underserved patients' attitudes toward their personal health and illness.13 Exposure to underserved communities in medical school, therefore, represents one way to confront this bias and potentially increase student interest in helping to provide care for underserved populations in the future. With this background in mind, the present study used the Medical Student Attitudes Toward the Underserved (MSATU) questionnaire8 to describe attitudes of students at the Uniformed Services University of the Health Sciences (USU) regarding health care in traditionally underserved communities. The MSATU is a previously validated instrument that has been specifically created to assess medical student attitudes regarding medical care and underserved populations.8 METHODS Study participants were students enrolled at the F. Edward Hébert Uniformed Services University School of Medicine (USU) (n = 170). USU is the nation's only federal medical school. All students are commissioned officers in the uniformed services. As such, they are salaried government employees with health care benefits and no incumbent tuition burden. Institutional Review Board approval was obtained before beginning the investigation. Participation in the study was voluntary. Interested students were verbally consented and given the MSATU questionnaire in the first year of medical school, and again during their fourth year. To preserve anonymity, students created an acrostic using a unique mixture of letters and numbers. This acrostic served as a unique identifier that allowed investigators to match individual responses from Year 1 and Year 4. Outcome Measure: The MSATU The MSATU assesses student attitudes regarding medical care in underserved populations.5,11 The total MSATU score combines two subscales, one that explores general attitudes toward underserved care and one that explores issues relating to access and payment for medical services. The first subscale assesses societal expectations and professional responsibility. Questions from this subscale target attitudes involving personal (respondent) roles and responsibilities as well as organizational responsibilities for underserved care. Respondent attitudes regarding access to care for patients regardless of their ability to pay is assessed by the second subscale (14 items). Both subscales use a 5-point Likert scale to quantify the extent to which respondents agree or disagree with each statement (5 = strongly agree, 1 = strongly disagree).9 The MSATU subscales are standardized using T-scores, which facilitates comparison of the magnitude of effects. T-scores have a mean of 50, and a standard deviation of 10. A T-score of 58, therefore, is 0.8 standard deviation above the mean. The change in T-score can be interpreted as an effect size. Previous studies have gathered evidence supporting the reliability and validity of the MSATU.11,12 Data Analysis MSATU scores were calculated using established scoring methods.9 Responses were analyzed using SPSS, Version 19.0 (SPSS, Chicago, Illinois). Baseline demographics were examined to assure that the sample was representative. Repeated measures ANOVA matching each student's Year 1 and Year 4 response was used to analyze longitudinal data and control for baseline differences. RESULTS Baseline 170 students enrolled in the class of interest. The mean matriculate age was 24. 65% were male and 78% were Caucasian. Table I displays MSATU scores for matched pairs (Year 1/Year 4) and for those students with only Year 1 data. The response rate for students completing the MSATU at both Year 1 and Year 4 was 35%. Compared to limited normative data from the literature,8,–10 USU respondents had less favorable MSATU total scores as well as scores on both subscales (Attitudes and Access). TABLE I. MSATU Total and Subscale Scores by Year of Training Measure  Year 1 Mean (SD)  Year 4 Mean (SD)  Both Years EMM  Both Years SE  p  MSATU Total          <0.01  USU  46.2 (10.95)  41.7 (12.38)  43.9  1.28    Set 1          <0.01  USU  45.7 (10.98)  41.22 (13.34)  43.5  1.31    Set 2          <0.01  USU  47.7 (10.95)  44.2 (10.95)  46.0  1.22    Measure  Year 1 Mean (SD)  Year 4 Mean (SD)  Both Years EMM  Both Years SE  p  MSATU Total          <0.01  USU  46.2 (10.95)  41.7 (12.38)  43.9  1.28    Set 1          <0.01  USU  45.7 (10.98)  41.22 (13.34)  43.5  1.31    Set 2          <0.01  USU  47.7 (10.95)  44.2 (10.95)  46.0  1.22    EMM, estimated marginal mean; SE, standard error. Repeated measures analysis of variance. View Large TABLE I. MSATU Total and Subscale Scores by Year of Training Measure  Year 1 Mean (SD)  Year 4 Mean (SD)  Both Years EMM  Both Years SE  p  MSATU Total          <0.01  USU  46.2 (10.95)  41.7 (12.38)  43.9  1.28    Set 1          <0.01  USU  45.7 (10.98)  41.22 (13.34)  43.5  1.31    Set 2          <0.01  USU  47.7 (10.95)  44.2 (10.95)  46.0  1.22    Measure  Year 1 Mean (SD)  Year 4 Mean (SD)  Both Years EMM  Both Years SE  p  MSATU Total          <0.01  USU  46.2 (10.95)  41.7 (12.38)  43.9  1.28    Set 1          <0.01  USU  45.7 (10.98)  41.22 (13.34)  43.5  1.31    Set 2          <0.01  USU  47.7 (10.95)  44.2 (10.95)  46.0  1.22    EMM, estimated marginal mean; SE, standard error. Repeated measures analysis of variance. View Large We examined age, gender, race, and MSATU scores for responders and nonresponders (students who completed only Year 1 administration) to determine whether the samples were representative and there were no significant differences between groups (responders and nonresponders). Table II displays the Year 1 to Year 4 comparison of responders. Repeated measures analysis of variance showed that Attitudes and Access scale scores declined over time. There were main effects for time for the MSATU Total scale and Set 1 (Attitudes). In both settings, MSATU scores declined significantly from Year 1 to Year 4. For the second MSATU subscale (Set 2, Access), there was a main effect of time. Attitudes declined significantly (from 47.7 to 44.2, a 0.35 standard deviation change) from Year 1 to Year 4. TABLE II. MSATU Total and Subscale Scores by Year of Training    Year 1  Year 4  Both Years Total  Mean Change  Effect Size  Mean  SD  Mean  SD  Mean  SE  MSATU Total  46.2  10.9  41.7  12.4  43.9  1.3  4.5  0.45  Set 1: Attitude  45.7  10.9  41.2  13.3  43.5  1.3  4.5  0.45  Set 2: Access  47.7  10.9  44.2  10.9  46.0  1.2  3.5  0.35     Year 1  Year 4  Both Years Total  Mean Change  Effect Size  Mean  SD  Mean  SD  Mean  SE  MSATU Total  46.2  10.9  41.7  12.4  43.9  1.3  4.5  0.45  Set 1: Attitude  45.7  10.9  41.2  13.3  43.5  1.3  4.5  0.45  Set 2: Access  47.7  10.9  44.2  10.9  46.0  1.2  3.5  0.35  Repeated Measures ANOVA: Within-subject variable: Year 1 versus Year 4. Effect size for Year 1 to Year 4 is the mean change divided by 10. View Large TABLE II. MSATU Total and Subscale Scores by Year of Training    Year 1  Year 4  Both Years Total  Mean Change  Effect Size  Mean  SD  Mean  SD  Mean  SE  MSATU Total  46.2  10.9  41.7  12.4  43.9  1.3  4.5  0.45  Set 1: Attitude  45.7  10.9  41.2  13.3  43.5  1.3  4.5  0.45  Set 2: Access  47.7  10.9  44.2  10.9  46.0  1.2  3.5  0.35     Year 1  Year 4  Both Years Total  Mean Change  Effect Size  Mean  SD  Mean  SD  Mean  SE  MSATU Total  46.2  10.9  41.7  12.4  43.9  1.3  4.5  0.45  Set 1: Attitude  45.7  10.9  41.2  13.3  43.5  1.3  4.5  0.45  Set 2: Access  47.7  10.9  44.2  10.9  46.0  1.2  3.5  0.35  Repeated Measures ANOVA: Within-subject variable: Year 1 versus Year 4. Effect size for Year 1 to Year 4 is the mean change divided by 10. View Large DISCUSSION Our findings suggest that medical students attending a federal medical school (USU) have generally less favorable attitudes toward care in underserved populations when compared with limited normative data nationally. This attitude appears to further decline between the first and fourth year of medical school. How this directly compares to other medical schools is not specifically known. Our findings shed light on at least two potentially important issues regarding attitudes toward care in underserved populations: debt and exposure. USU is a federal institution. Tuition is waived and students are salaried as commissioned officers in the uniformed services. It could be hypothesized that an absence of medical school debt might make students more open to interest in primary care specialties that are less lucrative, but also more likely to care for underserved populations. Our findings, although preliminary, do not appear to support this. Despite a relative lack of incurred debt, USU students had relatively low MSATU scores upon entry to medical school. This trend continued throughout all 4 years in our limited pool of respondents at both time periods. The reason(s) for why our respondents had a less favorable attitude toward care in underserved populations at admission is unclear and deserves further exploration if confirmed with larger samples. For example, given that our students are asked to practice in our military health system (MHS) for 7 (or more) years following graduation could lead to fewer applications from students with a strong interest in serving traditionally viewed underserved communities. Another important concept our study highlights is that of “exposure” to underserved populations. The MHS is a closed system. There are arguably fewer health care disparities in the MHS based solely on socioeconomic status or demographics. Within the MHS, all eligible patients have open access to medical care. Most USU medical student clinical experiences occur at military treatment facilities located on military bases. Restricted access to base for noneligible patients, combined with regular access to care for MHS beneficiaries makes it less likely that USU students are regularly exposed to patients in traditionally underserved settings. Insufficient insurance and lack of access to care are not problematic for patients (or providers) within the MHS. Data are still emerging regarding what lower MSATU scores mean for USU and nationally. We believe that an important line of future investigation entails qualitative methodology to explore underpinnings of these ratings to help inform institutions with translating quantitative findings to actions. Such work could help bring clarity to questions such as why might our students appear to have less favorable attitudes toward caring for the underserved. We have proposed some of the many reasons for why this may have been seen in our pilot population. This study has several important limitations. One is the relatively low-response rate from Year 1 to Year 4. Although, we attempted to control for this by examining demographic characteristics of respondents and nonrespondents, it is possible that the attitudes of nonrespondents vary significantly from those who chose to complete the survey in both Year 1 and Year 4. An additional limitation is the use of only one school. USU is the only federal medical school in the United States. Overall, our pilot study found less favorable attitudes toward caring for underserved populations in students attending USU. The specific reasons for this are not clear. Issues regarding debt and exposure to underserved populations are important elements to consider. Previous work has shown that an environment that promotes broad clinical exposures is one potential way to improve attitudes toward underserved populations. In addition, service learning opportunities provided early in medical school could help address the social contract schools have with their local communities to provide care in underserved populations and to reduce health care disparities. REFERENCES 1. United States Census Bureau Available at http://www.census.gov/hhes/www/hlthins/data/incpovhlth/2010/highlights.html; accessed September 15, 2014. 2. Majerol M, Newkirk V, Garfield R The Uninsured: A Primer: Key Facts Health About Insurance and the Uninsured in America . Kaiser Family Foundation, 2011. Available at http://www.kff.org/uninsured/upload/7451-06.pdf; accessed September 15, 2014. 3. U.S. Department of Health and Human Services, Health Resources and Services Administration, National Center for Health Workforce Analysis Projecting the Supply and Demand for Primary Care Practitioners Through 2020 . Rockville, MD, U.S. Department of Health and Human Services, 2013. Available at http://bhpr.hrsa.gov/healthworkforce/supplydemand/usworkforce/primarycare/projectingprimarycare.pdf; accessed October 1, 2014. 4. McCurdy L, Goode LD, Daugherty RM Jr, et al.   Fulfilling the social contract between medical schools and the public. Acad Med  1997; 72( 12): 1063– 70. Google Scholar CrossRef Search ADS PubMed  5. Smith JK, Weaver DB Capturing medical students' idealism. Ann Fam Med  2006; 4 Suppl 1: S32– 7; discussion S58–60. Google Scholar CrossRef Search ADS PubMed  6. Iglehart J Forum on the future of academic medicine: Session VI—Issues of change and quality in U.S. health care. Acad Med  1999; 74: 764– 72. Google Scholar CrossRef Search ADS PubMed  7. Ring JJ The right road for medicine. Professionalism and the new American Medical Association. JAMA  1991; 266: 1694. Google Scholar CrossRef Search ADS PubMed  8. Crandall SJ, Reboussin BA, Michielutte R, Anthony JE, Naughton MJ Medical students' attitudes toward underserved patients: a longitudinal comparison of problem-based and traditional medical curricula. Adv Health Sci Educ Theory Pract  2007; 12( 1): 71– 86. Google Scholar CrossRef Search ADS PubMed  9. Crandall SJ, Volk RJ, Cacy D A longitudinal investigation of medical student attitudes toward the medically indigent. Teach Learn Med  1997; 9: 254– 60. Google Scholar CrossRef Search ADS PubMed  10. Crandall SJ, Volk RJ, Loemker V Medical students' attitudes toward providing care for the underserved. Are we training socially responsible physicians? JAMA  1993; 269: 2519– 23. Google Scholar CrossRef Search ADS PubMed  11. Rabinowitz HK, Diamond JJ, Veloski JJ, Gayle JA The impact of multiple predictors on generalist physicians' care of underserved populations. Am J Public Health  2000; 90: 1225– 8. Google Scholar CrossRef Search ADS PubMed  12. American Association of Medical Colleges Medical students' socioeconomic background and their completion of the first two years of medical school , pp 1– 2. Washington, DC, American Association of Medical Colleges, 2010. 13. Williams SJ, Swinnen W, DeMaeseneer JM The GP's perception of poverty: a qualitative study. Fam Pract  2005; 22: 177– 83. Google Scholar CrossRef Search ADS PubMed  Reprint & Copyright © Association of Military Surgeons of the U.S.