Why it's time to remove time limits on tests, like the USMLE exams

A recent NEJM article and accompanying podcast episode (“Tough Love”) authored and hosted by the Journal’s national correspondent sound the alarm that a culture of grievance among medical students and trainees about the discomforts of medical training is threatening to undermine both their medical education and patient care. She also describes widespread anxiety among medical educators who feel fearful of speaking because of concerns of retaliation on social media.  Absent from the discussion, however, are the voices of students and trainees who, in the podcast, are referred to as “our children.” Medical Students and trainees we spoke with did not feel that their concerns are experiences were accurately characterized. We propose that medical educators are ill prepared for the shifting power dynamics, both in terms of knowing how to listen and how to lead.

“Sonny’s Blues” is a 1956 story by the author, James Baldwin, about a “sensible” and pragmatic algebra teacher and his younger musically gifted younger brother (“Sonny”), who struggles with heroin addiction. Both of them, raised in Harlem, are deeply affected by anti-Black racism.  Although the older brother, who narrates the story, feels responsible for Sonny, he struggles to relate to him. With the help of an English professor, Laura Greene at Augustana College, we reflect on some of the lessons of this story for the physician-patient relationship, especially when caring for individuals with substance use disorder. We explore the cost both to patients and to ourselves, as healthcare professionals, of holding patients at arm’s length because we fear engaging, especially in the face of suffering. A PDF of “Sonny’s Blues,” can be accessed from the story’s Wiki page (scroll down to external links).

In a 2021 episode that we reran last month, “About me being racist: a conversation that follows an apology,” Saul talked with a former Black colleague after apologizing to her for something racist he had done twenty years earlier that hurt her for a long time.   Since then, Saul has been thinking about how he got exposed to racist ideas and notions of power as a white male growing up in the United States (in his case in a liberal, highly educated community) and suggested that he and Stefan talk about it, taking to heart Toni Morrison’s admonition that, “White people have a very serious problem, and they should start thinking about what they can do about it – and leave me out of it!” Also, next month we’ll de discussing a short story by author James Baldwin with a special guest, and would like to encourage listeners to read “Sonny’s Blues,” which can be accessed from the story’s Wiki page (scroll down to external links).

We are re-running this episode from 2021 because we’re releasing a sequel next month in which Saul reflects on his journey confronting racist ideas he’d absorbed and that became impossible to ignore after he’d acknowledged his role in the incident described here.  We are also re-running the episode because it exemplifies our commitment to facing things -- about ourselves and our profession – to enhance our wellbeing, and our relationships with colleagues and patients.  Rather than disheartening, we find such conversations and the changes they bring rewarding and healing.

For years, when physicians order tests to assess lung function, or blood work to determine kidney function, or look up guidelines for managing high blood pressure the results have been adjusted for race. This practice has been based on studies that seemed to indicate that the same result means different things if the patient is Black vs white. So, for instance, an “uncorrected’ creatinine of 1.6 was thought to be less concerning in a Black than white patient as Blacks were thought to have greater muscle mass (not true). These correction factors masked underlying environmental and social stressors disproportionately affecting Black Americans. Regrettably they also contributed to delays in care for chronic conditions, as Black patients had to be sicker than white patients to trigger therapeutic interventions – further exacerbating disparities. We talk with two physicians who lead an anti-racism equity committee based in a Chicago VA hospital to understand the history and science that led to these “corrections,” and how they have successfully been removing them through education and advocacy across their organization and nationally. Their activism is especially meaningful because of its immediate, tangible, benefit for affected patients. The views expressed in this episode are those of the participants and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. 

The practice of urine drug testing during pregnancy and then often reporting positive results to Child Protective Services triggers a cascade that can result in separation of mother and newborn, with devastating consequence for both. These practices are more common when patients come from marginalized communities even when baseline substance use rates are the same. As our guest -- obstetrician/gynecologist and addiction medicine expert Mishka Terplan MD, MPH -- points out, illicit substances are not teratogens in comparison to, say, alcohol, tobacco or lead exposure. So why do we order these tests? He also discusses how talking with patients about substance use behaviors, especially with the help of screening instruments, is the only way to characterize substance use behaviors and formulate treatment strategies. This is the third episode in which we learn of common clinician practices in which clinicians are co-opted into punitive and even carceral systems of oppression.

Direct, covert observation of health care is a novel and underutilized tool to assess health care trainees and clinicians. In this episode we talk with experts about two such approaches: the unannounced standardized patient and patient-collected audio. In the former, actors are sent incognito into practice settings, and in the latter real patients volunteer to record their visits on behalf of a quality improvement team.  Both approaches address the question, “How are our learners and experienced clinicians performing in the real world?” They also identify those who may do well on simulations but underperform in the clinical setting. As one of our guests observed, “If McDonalds is using secret shoppers to improve services, shouldn’t we be doing the same in health care (but with a lot more rigor) where the stakes are so much higher?”

In the prior episode we learned that there is no evidence that time-limited testing improves test validity and that, in fact, there is ample research showing that it makes tests less valid and less equitable. In this episode we discuss how, despite the data, the NBME denies accommodations on the USMLE exams to over half of medical students who have a documented learning disability and are approved for accommodations at their medical school (e.g., extra time). We talk with a leading medical educator about a national survey she and her colleagues conducted to assess the scope and impact on medical schools and their students. And we conclude with a discussion about how the NBME could make the test fair and valid for everyone. 

There is a widely held perception that being able to complete a test quickly is an indication of mastery when compared with those who need more time. As a result, it is often difficult to obtain accommodations on high stakes examinations, including the USMLE exams.  Many students who request extra time because of a disability are denied accommodations and many other students who need it aren't eligible (e.g., English is a second language) or are inhibited from applying (e.g., Veterans, students from certain cultural backgrounds). But what does the evidence show? In this episode we interview an expert on the topic about a paper she authored titled Four Empirically Based Reasons Not to Administer Time-Limited Tests. The implications are profound because this is a problem we can fix, significantly improving high stakes assessment, equity, and inclusivity. 

Stefan interviews co-host Saul about his experiences becoming a doctor with a learning disability.  This episode, first run in 2020, sets the stage for two that will follow – in August and September, with experts on the science of student learning assessment and its implications for the USMLE examinations. These will address questions such as: Does struggling with multiple-choice tests under time pressure predict anything about future performance in the clinical setting? Do time limits make tests more or less valid and reliable?  What are implications of denying so many students accommodations on the USMLE examinations?  And, most importantly, what can we do about the documented perverse effects of our current system of assessment on equity and inclusion and, ultimately on the quality and diversity of our physician workforce?