Medical Education

Situational Judgment Testing

Improving Medical School Selection Processes

By Mojca Remskar, MD, PhD, and Dimple Patel, MS

Why is there a need for consideration of a tool like SJT? Let’s admit it—the institutional student selection processes in medicine favor specific groups of people over others. As argued in a recent article in Annals of Internal Medicine, medical schools, professional organizations, academic departments and private practices are racialized organizations, which to a large extent, continue to use structures and processes that promote certain groups. Traditionally, we have relied heavily on the use of academic measures, such as cognitive tests, as a leading factor in our decision-making. It is well-documented that using these measures disadvantages certain populations of applicants. In evidence-based medicine, it seems counterintuitive and hard for us to admit, but studies show that “good looks’’ and being thin improve your chance of being selected into a residency program. Being Black, Asian or obese is a disadvantage, according to various studies. Practice of individual, one-on-one interviews with faculty, no matter how unbiased and open-minded faculty are, leads to acceptance of “the best fit” candidates, who often reflect the leadership structure/image, which in medicine, still predominantly consists of white, heterosexual, cisgender males.

Changing a paradigm

In recent years, we have seen a significant emphasis being given to non-cognitive domains related to interpersonal communication and professionalism, with the intent to counter the above mentioned trend within medical schools and professional medical institutions and hospitals. To support the need for non-cognitive competencies to be included in the selection process, there is evidence that competencies, such as teamwork and accountability, correlate with improved patient outcomes, patient satisfaction and adherence to treatment plans. The AAMC and the Accreditation Council for Graduate Medical Education (ACGME) both emphasize professionalism and communication skills as essential components of a well-rounded physician by including them into their assessment frameworks. However, at present time, we do not routinely use reproducible, validated, easy-to-use tools to measure the professionalism and communication domains in our interview processes. Instead, we rely on one-on-one interviews as the most important way to assess applicants’ interpersonal and communication skills, maturity, interest in the field, dependability and honesty. This process is not standardized and lacks inter-rater reliability, and therefore, its outcomes are biased by personal preferences of interviewers.



The SJT can measure non-cognitive skills, such as those related to professionalism and attributes of an individual that mimic actual real-life events related to specific professions. Its ability to objectively measure interpersonally oriented skills is what makes it attractive for the selection processes. 

The SJT can measure non-cognitive skills.

Development of the SJT starts with analysis of professional critical events that would present specific constructs or domains relevant to the profession. Experts and non-experts are asked to develop a list of best, worst and in-between responses or approaches to address different situations and incidents. In the final step, a scoring key is developed based on the ability of specific responses to differentiate between experts and novices and the correlation with job performance in specific domains. The administration of SJT is relatively simple. It comes in a series of online scenarios for which the candidate is asked to rank the responses from the best to the worst. It can be administered as a written test, a video or a cartoon.



As mentioned previously, AAMC has developed its own SJT which measures eight pre-professional competencies relevant to medical school applicants – service orientation, social skills, cultural competencies, teamwork, ethical responsibility to self and others, reliability and dependability, resilience and adaptability and capacity for improvement. Understanding the importance of the inclusion of the professionalism domains into candidate selection, the SJT presents around 30 scenarios with multiple responses required to each one. Test takers select a response most closely related to how they would handle that situation. The entire test takes about 75 minutes to complete. Since SJT requests individuals to respond to the questions to the best of their ability, there is a possibility that the individual will respond with the option they feel will satisfy the exam, and not necessarily respond as they would act in the situation. This has been studied and the impact of fakability has been determined to be relatively small.

Creating a holistic review

The University of Minnesota Medical School is one medical school with two campuses. The mission of the Medical School states that it seeks to enroll and educate skilled, compassionate and socially responsible physicians. Furthermore, the school is looking to educate individuals who appreciate working in diverse communities with an interdisciplinary focus and within inter-professional teams. The aforementioned attributes and others are being screened for in the current admissions process, but using a valid and reliable assessment tool that can accurately identify these non-cognitive attributes in applicants is a necessary tool for holistic review.

There are both opportunities and challenges to including a non-cognitive assessment. The challenges include the added cost to taking an examination on top of an already expensive endeavor. There are no fees in this initial year for taking the test, though the AAMC plans to add them moving forward. Exactly how they will be charged is undetermined at this point. It already costs on average $1,400 for an applicant to apply to medical school. Furthermore, applicants would need to spend additional time preparing on top of the many hours of academic coursework and co-curricular activities. Adding another requirement could cause additional stress and anxiety to an already demanding process. It will also require committee members to take additional time to learn a new tool, and some might feel unwilling to embrace new and unproven methodologies.



The opportunities, however, outweigh the challenges in the long run. A non-cognitive assessment directly compliments holistic review and fits with the mission of many medical schools. Implementing an assessment, like the SJT, has the potential to make medical education more accessible to a wider audience. There is already plenty of data that show that lower MCAT scores keep under-represented in medicine (UIM) populations out of medical education in greater numbers. The holistic review process takes a great deal of time and involves many people. A non-cognitive assessment could help make the review process more efficient for the institution and potentially free up time for an already highly committed group of people.

Finally, and maybe most importantly, could a non-cognitive assessment like the SJT be as highly valued as the MCAT? It would be worth figuring out, so that when we are speaking about applicant access and medical student failure and success, our first question is something other than “What is the candidate’s MCAT score?”

 Medicine as a whole has to find ways to diversify the physician population.

Improving resident selection

In parallel to the efforts by the U of M Medical School, U of M Graduate Medical Education Office, under the leadership of Drs. John Andrews and Michael Cullen, has also developed SJT geared towards resident selection. The model includes the following dimensions of professionalism – conscientiousness, aspiring to excellence, integrity, accountability, teamwork, patient centered care and stress tolerance. The resident-specific SJT has been shown to predict overall ACGME milestone performance and multisource professionalism assessment performance. SJT has been shown to predict overall ACGME milestone performance and multisource professionalism assessment performance.



In the resident selection process, any given department generally receives about 1,000 applications each year. The screening process frequently includes scores on a cognitive test as the primary filter due to the simplicity of using a number. This is combined with review of application packets, which include the applicant’s medical school transcripts, AOA membership, letters of recommendation, personal statement and description of their research activities, work experiences and volunteering activities. Based on these, the final decision is made to interview about one-tenth of the applicants. In the actual interview process, we have used United States Medical Licensing Exam scores and assessment of applicants by faculty as the most important components of applicant selection. As discussed earlier, this is a flawed process, which selects a specific population of applicants and more often than not works against minority applicants.

In the recent few years, the U of M anesthesiology residency program has worked on standardization of the interview process to minimize subjectivity. Several factors have led our residency program to look for opportunities to incorporate new measures of applicant non-cognitive domains into our process. Most importantly, our residency program is making a structured effort to increase the ratio of UIM applicants in our program. As part of this transition, we have started using standardized questions in our interview process in 2016-17. And in 2018, we have added SJT as part of the applicant interview process.

Unanswered questions

A lot remains to be investigated to inform the best use of a tool like SJT. We have yet to define what an appropriate weight of all the different new measures is in the decision-making process about applicants. Longitudinal studies of how inclusion of situational judgment tests change our successful applicant pools are not well described. We have yet to answer questions about how new measures correlate with medical student, resident and attending work performance.

However, at the U of M Medical School, we believe a standardized holistic process that relies less on traditional academic performance measures and emphasizes interpersonal and professionalism dimensions of individual applicants can diversify our medical population to better reflect population statistics.

We believe medicine as a whole has to find ways to diversify the physician population to better represent the U.S. population. A tool like SJT is a move in the right direction towards a more balanced physician pool in the future.

Mojca Remskar, MD, PhD, MACM, is a professor and the Executive Vice Chair in the Department of Anesthesiology at the University of Minnesota Medical Schools.

Dimple Patel, MS, is he Associate Dean of Admissions at the University of Minnesota Medical School.