Research
I study how people make inferences about sources of information, with a focus on social dynamics and applications in medicine.
Evidence evaluation and metascience
▸Reproducibility and robustness of economics and political science research
Brodeur, A., Mikola, D., …, Levin, J.M., et al. (2026). Nature, 652, 151-156.
Science aspires to be cumulative. Reproducibility efforts strengthen science by testing the reliability of published findings, promoting self-correction, and informing policy-making. Computational reproductions, whereby independent researchers reproduce the results of published studies, are an essential diagnostic tool. Such efforts should have greater visibility. However, little social science reproduction and robustness has been conducted at scale. Here we reproduced original analyses and conducted robustness checks of 110 articles that were published in leading economics and political science journals with mandatory data and code sharing policies. We found that more than 85% of published claims were computationally reproducible. In robustness checks, our reanalyses showed that 72% of statistically significant estimates remain significant and in the same direction, and the median reproduced effect size is nearly the same as the originally published effect size (that is, 99% of the published effect size). Additionally, 6 independent research teams examined 12 pre-specified hypotheses about determinants of robustness. Research teams with more experience found lower levels of robustness, and robustness did not correlate with author characteristics or data availability.
▸Barriers to adopting evidence from Bayesian adaptive clinical trials in critical care: a randomized experiment with US intensivists
Malley, B.E., Levin, J.M., Kahn, J.M., Bukowski, L.A., & Huang, D.T. (2026). CHEST Critical Care, 4(2), 100264.
Randomized clinical trials are a mainstay of medical research, but have drawbacks including time and cost. Bayesian adaptive trials can improve the speed and efficiency of clinical trials and are an increasingly common trial design in critical care. However, the complexity of Bayesian adaptive trials may prevent clinicians from easily interpreting their results. Do intensivist physicians perceive, understand, and accept results from Bayesian adaptive clinical trials differently than results from traditional frequentist trials when presented with otherwise identical data? We surveyed US intensivists from March through April 2022 about their perceptions of Bayesian adaptive trials. Within the survey, participants were randomized to read an abstract for a hypothetical new sepsis drug trial that used either Bayesian adaptive methods or traditional frequentist methods, with both abstracts based on the same simulated trial data. Participants then were asked about perceived understanding, validity, and general acceptance of the trial’s methods and results. Survey responses were compared between experimental groups using Mann-Whitney U tests and ordinal logistic regressions. We received complete results from 273 of 592 eligible physicians (response rate, 46.1%). Participants in the Bayesian group reported lesser understanding of the methods (mean rating in frequentist vs Bayesian groups, 3.18 vs 2.85; P < .001) and the results of the trial (mean rating, 3.36 vs 3.00; P < .001) compared with the frequentist group. Participants in the Bayesian group also expressed greater concerns about trial validity (eg, mean rating about the risk of type 1 error, 2.70 vs 2.99; P = .001) compared with the frequentist group. Participants in both groups reported similar beliefs about acceptance of trial results (eg, mean rating in frequentist vs Bayesian group about the effectiveness of the study drug, 3.87 vs 3.86; P = .47). Our results show that despite lower perceived understanding of Bayesian trials and skepticism about elements of their validity, no substantive differences were found in intensivist physician acceptance of Bayesian trial results compared with frequentist trial results.
▸Understanding provider-level variation in clinical practice: a longitudinal survey of corticosteroid use in septic shock
Soni, K., Bukowski, L.A., Davis, B.S., Levin, J.M., & Kahn, J.M. (2025). CHEST Critical Care, 3(2), 100134.
Clinician practice patterns vary widely in the ICU, particularly in the context of therapies with moderate-quality evidence. A better understanding of the psychological factors underlying provider-level practice variation may reveal targets to standardize practice and speed adoption of new evidence. Are provider-level psychological traits associated with practice patterns in the ICU? We administered a longitudinal questionnaire to intensivist clinicians in a single integrated health system from 2018 through 2021. We selected corticosteroids in septic shock as an example of a guideline-recommended practice supported by moderate-quality evidence. In response to a clinical vignette of septic shock, intensivists rated their willingness to administer corticosteroids, their perceptions of the strength of evidence, and their perception of how well the mechanism of action is understood. Via the same questionnaire, we measured psychological traits hypothesized to affect decision-making under uncertainty. We used multivariate regression to examine the relationship between these factors and respondents’ willingness to treat with corticosteroids. The overall survey completion rate was 50.5%. Among 201 participants, 101 participants (50%) were attending physicians, 77 participants (38%) were fellow physicians, and 23 participants (11%) were advanced practice providers. Willingness to treat with corticosteroids increased over time, from 15.3% to 24.4% in 2021. In multivariate regression analyses using generalized estimating equations for repeated measures, risk tolerance (OR, 1.34, 95% CI, 1.07-1.67; P = .01) and perceived strength of evidence (OR, 1.45; 95% CI, 1.18-1.77; P < .001) were associated with increased willingness to treat with corticosteroids, controlling for other factors. Willingness to treat was not associated with perceived understanding of the mechanism of action or other psychological factors. Our findings indicate that ICU providers’ individual level of risk tolerance is associated with their willingness to prescribe corticosteroids for septic shock.
▸The political polarization of COVID-19 treatments among physicians and laypeople in the United States
Levin, J.M., Bukowski, L.A., Minson, J.A., & Kahn, J.M. (2023). Proceedings of the National Academy of Sciences, 120(7), e2216179120.
Press: The Washington Post, The Atlantic, Medpage, San Francisco Chronicle, Ars Technica, Psychiatrist.com, Pennlive.com
In the United States, liberals and conservatives disagree about facts. To what extent does expertise attenuate these disagreements? To study this question, we compare the polarization of beliefs about COVID-19 treatments among laypeople and critical care physicians. We find that political ideology predicts both groups’ beliefs about a range of COVID-19 treatments. These associations persist after controlling for a rich set of covariates, including local politics. We study two potential explanations: a) that partisans are exposed to different information and b) that they interpret the same information in different ways, finding evidence for both. Polarization is driven by preferences for partisan cable news but not by exposure to scientific research. Using a set of embedded experiments, we demonstrate that partisans perceive scientific evidence differently when it pertains to a politicized treatment (ivermectin), relative to when the treatment is not identified. These results highlight the extent to which political ideology is increasingly relevant for understanding beliefs, even among expert decision makers such as physicians.
▸Willingness to treat with therapies of unknown effectiveness in severe COVID-19: a survey of intensivist physicians
Levin, J.M., Davis, B.S., Bukowski, L.A., & Kahn, J.M. (2022). Annals of the American Thoracic Society, 19(4), 633-639.
Little is known about how physicians develop their beliefs about new treatments or update their beliefs in the face of new clinical evidence. These issues are particularly salient in the context of the coronavirus disease (COVID-19) pandemic, which created rapid demand for novel therapies in the absence of robust evidence. To identify psychological traits associated with physicians’ willingness to treat with unproven therapies and willingness to update their treatment preferences in the setting of new evidence in the context of COVID-19. We administered a longitudinal e-mail survey to United States physicians board certified in intensive care medicine in April and May 2020 (phase one) and October and November 2020 (phase two). We assessed five psychological traits potentially related to evidence uptake: need for cognition, evidence skepticism, need for closure, risk tolerance, and research engagement. We then examined the relationship between these traits and physician preferences for pharmacological treatment for a hypothetical patient with severe COVID-19 pneumonia. There were 592 responses to the phase one survey, conducted prior to publication of trial data. At this time physicians were most willing to treat with macrolide antibiotics (50.5%), followed by antimalaria agents (36.1%), corticosteroids (24.5%), antiretroviral agents (22.6%), and angiotensin inhibitors (4.4%). Greater evidence skepticism (relative risk [RR], 1.40; 95% confidence interval [CI], 1.30-1.52; P < 0.001), greater need for closure (RR, 1.19; 95% CI, 1.06-1.34; P = 0.003), and greater risk tolerance (RR, 1.17; 95% CI, 1.08-1.26; P < 0.001) were associated with an increased willingness to treat, whereas greater need for cognition (RR, 0.85; 95% CI, 0.75-0.96, P = 0.010) and greater research engagement (RR, 0.91; 95% CI, 0.88-0.95; P < 0.0001) were associated with decreased willingness to treat. In phase two, most physicians updated their beliefs after publication of trial data about antimalarial agents and corticosteroids. Physicians with greater evidence skepticism were more likely to persist in their beliefs. Psychological traits associated with clinical decisions in the setting of uncertain evidence may provide insight into strategies to better align clinical practice with published evidence.
Human-AI collaboration
▸Indirect influences of artificial intelligence advice on complex sepsis treatment decisions
Sivaraman, V., Levin, J.M., Mason, E., …, & Perer, A. (2026). Under review at Nature Communications.
Sepsis is a major public health concern, and many sepsis patients die due to ineffective hemodynamic management. Artificial intelligence could support hemodynamic decisions, particularly in complex cases where conflicting information and comorbidities drive uncertainty and practice variation. However, these systems’ real-world value depends on not just accuracy, but how they influence the quality of clinician decisions. We first retrospectively evaluated an AI system’s recommendations in 6,928 sepsis patients, finding that they improved upon the original providers’ decisions overall but had significantly worse performance in complex cases. We then conducted a pre-registered experiment with 228 US critical care physicians to test their use of this imperfect advice in six complex treatment decisions, using one of three advice types designed to vary reasoning autonomy. Participants given simple model recommendations overcame poor advice, choosing optimal treatments in 56.3% of decisions compared to 31.8% if recommendations were always followed. They also made significantly better decisions than those who received a less prescriptive advice type (42.9%). Physicians selectively adapted and improved weak advice, likely because it prompted them to reconsider their assessment. Our findings suggest that improving sepsis care decisions with AI may require accounting for these indirect influences resulting from human-AI collaborative reasoning.
▸Ignore, trust, or negotiate: understanding clinician acceptance of AI-based treatment recommendations in health care
Sivaraman, V., Bukowski, L.A., Levin, J.M., Kahn, J.M., & Perer, A. (2023). Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.
Artificial intelligence (AI) in healthcare has the potential to improve patient outcomes, but clinician acceptance remains a critical barrier. We developed a novel decision support interface that provides interpretable treatment recommendations for sepsis, a life-threatening condition in which decisional uncertainty is common, treatment practices vary widely, and poor outcomes can occur even with optimal decisions. This system formed the basis of a mixed-methods study in which 24 intensive care clinicians made AI-assisted decisions on real patient cases. We found that explanations generally increased confidence in the AI, but concordance with specific recommendations varied beyond the binary acceptance or rejection described in prior work. Although clinicians sometimes ignored or trusted the AI, they also often prioritized aspects of the recommendations to follow, reject, or delay in a process we term “negotiation.” These results reveal novel barriers to adoption of treatment-focused AI tools and suggest ways to better support differing clinician perspectives.
Judgment and belief under uncertainty
▸When (and why) expressing uncertainty affects perceptions of competence
Levin, J.M. (2026). Working paper.
What do we infer about others when they express uncertainty? A broad literature on this question has yielded results that appear contradictory. In prior work, expressing more (versus less) certainty has been associated with being perceived as being both more and less competent. In this paper, I propose and test a theory to reconcile these findings. I argue that people make inferences about others on the basis of fit between the amount of certainty that those others express and their own beliefs about the epistemicness (predictability) of the environment. Fit between expressed certainty and perceived epistemicness suggests calibration and in turn competence, while a lack of fit suggests poor calibration and (relative) incompetence. I find support for this account across four experimental studies involving both real and hypothetical communication. In each study, I manipulate two factors: the amount of certainty that an advisor or teammate expresses and a characteristic that affects the epistemicness of a given prediction or estimation. Participants display a preference for others who express more certainty in more epistemic environments, but this preference attenuates (S1) or reverses (S2 - S4) in less epistemic environments. Participants’ perceptions of others’ calibration follow the same pattern (S3).
▸Understanding others’ models of uncertainty
Levin, J.M., Fox, C.R., & Ülkümen, G. (2026). Working paper.
Prior work has established that people intuitively distinguish between uncertainty that is reducible given additional information (epistemic uncertainty) and uncertainty that is inherently irreducible (aleatory uncertainty) and that they use distinct language when describing each. In epistemic domains, people commonly use language such as “I am N% sure”, whereas in aleatory domains, language such as “There is an N% chance” is more common. We hypothesize that people use such language as a cue when evaluating others’ competence – preferring others who use language that is consistent with their own beliefs about the type of uncertainty that characterizes a given outcome. We find support for this pattern across two studies. In Study 1, participants rate others as being more competent when they use confidence language (such as “… N% sure”) when offering advice about outcomes that are perceived to be relatively more epistemic and likelihood language (such as “… N% chance”) when offering advice about outcomes that are perceived to be relatively more aleatory. In Study 2, we employ a manipulation to shift perceptions of uncertainty type within the context of a single, real-world outcome. Priming participants to think about an outcome as being relatively more epistemic or more aleatory shifts their predictions about the judgment accuracy of others who use confidence language versus those who use likelihood language.
Expecting expectations: correcting for theory-laden perception in inference from experience
Levin, J.M., & McKenzie, C.R.M. In progress.
Education and group dynamics
▸The Maximizing Extubation Outcomes through Educational and Organizational Research (METEOR) trial
Girard, T.D., …, Levin, J.M., et al. In progress.
The METEOR Trial will compare four implementation strategies—traditional online education, protocol-directed care, interprofessional education, and a combination of protocol-directed care and interprofessional education—to test the hypotheses that interprofessional education is superior to traditional online education as an implementation strategy in the intensive care unit (ICU) and the benefits of interprofessional education are increased when interprofessional education is paired with a clinical protocol. Additionally, the trial will also test the hypothesis that preventive post-extubation NIV for high-risk patients and preventive post-extubation HFNC for low-risk patients are both superior to current clinical practice (i.e., conventional post-extubation oxygen therapy).