Large Language Models Answer Medical Questions Accurately, but Can’t Match Clinicians’ Knowledge (2024)

Table of Contents
Journals Podcasts Citation References

Our website uses cookies to enhance your experience. By continuing to use our site, or clicking "Continue," you are agreeing to our Cookie Policy|Continue

JAMA

    Sign In

    Individual Sign In

    Sign inCreate an Account

    Access through your institution

    Sign In

    Purchase Options:

    Buy this article

    Subscribe to the JAMA journal

    This Issue

    Views 11,174

    Comments 2

    JAMA Medical News

    visual abstract icon Visual Abstract editorial comment icon Editorial Comment related articles icon Related Articles author interview icon Interviews multimedia icon Multimedia audio icon Listen to this article
    • Viewpoint The Challenges for Regulating Medical Use of Large Language Models

      TimoMinssen,LLD; EffyVayena,PhD; I. GlennCohen,JD

      JAMA

    • Medical News in Brief National Health Care Leaders Will Develop AI Code of Conduct

      EmilyHarris

      JAMA

    • Medical News in Brief Study Tests Large Language Models’ Ability to Answer Clinical Questions

      EmilyHarris

      JAMA

    • Special Communication Creation and Adoption of Large Language Models in Medicine

      Nigam H.Shah,MBBS, PhD; DavidEntwistle,BS, MHSA; Michael A.Pfeffer,MD

      JAMA

    • RebeccaVoelker,MSJ

      JAMA

    • YulinHswen,ScD, MPH; JenniferAbbasi

      JAMA

    • Medical News in Brief AI-Drafted Responses to Patients Reduced Clinician Burnout

      EmilyHarris

      JAMA

    • Original Investigation Comparing Physician and Chatbot Responses to Patient Questions

      John W.Ayers,PhD, MA; AdamPoliak,PhD; MarkDredze,PhD; Eric C.Leas,PhD, MPH; ZechariahZhu,BS; Jessica B.Kelley,MSN; Dennis J.Faix,MD; Aaron M.Goodman,MD; Christopher A.Longhurst,MD, MS; MichaelHogarth,MD; Davey M.Smith,MD, MAS

      JAMA Internal Medicine

    • Research Letter Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations

      EricStrong,MD; AliciaDiGiammarino,MS; YingjieWeng,MHS; AndreKumar,MD, MEd; PoonamHosamani,MD; JasonHom,MD; Jonathan H.Chen,MD, PhD

      JAMA Internal Medicine

    Full Text

    When Google completes the sentence, “Coronary artery disease is caused by,” the search engine offers several possibilities: atherosclerosis, smoking, and inflammation, to name a few.

    This type of word completion—the province of large language models (LLMs) such as OpenAI’s ChatGPT or Google’s Pathways Language Model, known as PaLM—uses billions of pieces of information the models have trained on to predict what’s most likely to come next. In short, the models are “autocomplete on steroids,” bioinformatics expert Jonathan Chen, MD, PhD, an assistant professor of medicine at Stanford Medicine, said in an interview.

    Full Text

    Add or change institution

    Comment

    2 Comments for this article

    EXPAND ALL

    August 8, 2023

    Artificial Intelligence or Human Deficiency?

    Alessandro Capone, Medical Doctor | Statistics for Health Economic Evaluation Group (external member), Department of Statistical Science, University College London

    The author highlights the potential of large language models (LLMs), their limitations, and the relative lack of appropriate guidelines for implementing such powerful tools. Unfortunately, the history of scientific research is paved with horrifying mistakes (whether accidental or deliberate), and the use of LLMs could generate errors of judgment and lead to decisions that could undermine individual and public health, and/or the quality of life for many people.

    The core asset of scientific research - universal reproducibility - must be guaranteed and protected by clear and shared guidelines that strictly regulate use of LLMs.

    The self-learning procedures and

    related AIXIs must be tested for discrepancy with established and published clinical and economic data. This or other measures of accuracy and consistency must be applied to estimate the reliability of the system. The ultimate decision to implement a health policy, for instance, must be the sole and exclusive prerogative of the human being. It is artificial intelligence that must be subservient to mankind and not (hopefully) the other way around.

    CONFLICT OF INTEREST: None Reported

    READ MORE

    August 23, 2023

    The difference between human creativity and AI creativity

    Jun Xu | Private

    The difference in creativity between human intelligence and AI is that we create something new based primarily on our understanding, whereas AI creates something new based on statistical probability or correlation. Since statistical correlation is not human understanding, AI clinicians should highlight their own creation so that patients are aware of the parts that may contain hallucinations and errors.

    CONFLICT OF INTEREST: None Reported

    Read More About

    Artificial Intelligence

    Download PDF Full Text

    Cite This

    Citation

    Harris E. Large Language Models Answer Medical Questions Accurately, but Can’t Match Clinicians’ Knowledge. JAMA. 2023;330(9):792–794. doi:10.1001/jama.2023.14311

    Manage citations:

    Ris (Zotero) EndNote BibTex Medlars ProCite RefWorks Reference Manager Mendeley

    © 2024

    Comment

    Add or change institution

    Artificial Intelligence ResourceCenter

    Others Also Liked

    Select Your Interests

    Customize your JAMA Network experience by selecting one or more topics from the list below.

    • Academic Medicine
    • Acid Base, Electrolytes, Fluids
    • Allergy and Clinical Immunology
    • American Indian or Alaska Natives
    • Anesthesiology
    • Anticoagulation
    • Art and Images in Psychiatry
    • Artificial Intelligence
    • Assisted Reproduction
    • Bleeding and Transfusion
    • Cardiology
    • Caring for the Critically Ill Patient
    • Challenges in Clinical Electrocardiography
    • Climate and Health
    • Climate Change
    • Clinical Challenge
    • Clinical Decision Support
    • Clinical Implications of Basic Neuroscience
    • Clinical Pharmacy and Pharmacology
    • Complementary and Alternative Medicine
    • Consensus Statements
    • Coronavirus (COVID-19)
    • Critical Care Medicine
    • Cultural Competency
    • Dental Medicine
    • Dermatology
    • Diabetes and Endocrinology
    • Diagnostic Test Interpretation
    • Drug Development
    • Electronic Health Records
    • Emergency Medicine
    • End of Life, Hospice, Palliative Care
    • Environmental Health
    • Equity, Diversity, and Inclusion
    • Ethics
    • Facial Plastic Surgery
    • Gastroenterology and Hepatology
    • Genetics and Genomics
    • Genomics and Precision Health
    • Geriatrics
    • Global Health
    • Guide to Statistics and Methods
    • Guidelines
    • Hair Disorders
    • Health Care Delivery Models
    • Health Care Economics, Insurance, Payment
    • Health Care Quality
    • Health Care Reform
    • Health Care Safety
    • Health Care Workforce
    • Health Disparities
    • Health Inequities
    • Health Policy
    • Health Systems Science
    • Hematology
    • History of Medicine
    • Humanities
    • Hypertension
    • Images in Neurology
    • Implementation Science
    • Infectious Diseases
    • Innovations in Health Care Delivery
    • JAMA Infographic
    • Law and Medicine
    • Leading Change
    • Less is More
    • LGBTQIA Medicine
    • Lifestyle Behaviors
    • Medical Coding
    • Medical Devices and Equipment
    • Medical Education
    • Medical Education and Training
    • Medical Journals and Publishing
    • Melanoma
    • Mobile Health and Telemedicine
    • Narrative Medicine
    • Nephrology
    • Neurology
    • Neuroscience and Psychiatry
    • Notable Notes
    • Nursing
    • Nutrition
    • Nutrition, Obesity, Exercise
    • Obesity
    • Obstetrics and Gynecology
    • Occupational Health
    • Oncology
    • Ophthalmology
    • Orthopedics
    • Otolaryngology
    • Pain Medicine
    • Palliative Care
    • Pathology and Laboratory Medicine
    • Patient Care
    • Patient Information
    • Pediatrics
    • Performance Improvement
    • Performance Measures
    • Perioperative Care and Consultation
    • Pharmacoeconomics
    • Pharmacoepidemiology
    • Pharmacogenetics
    • Pharmacy and Clinical Pharmacology
    • Physical Medicine and Rehabilitation
    • Physical Therapy
    • Physician Leadership
    • Poetry
    • Population Health
    • Primary Care
    • Professional Well-being
    • Professionalism
    • Psychiatry and Behavioral Health
    • Public Health
    • Pulmonary Medicine
    • Radiology
    • Regulatory Agencies
    • Reproductive Health
    • Research, Methods, Statistics
    • Resuscitation
    • Rheumatology
    • Risk Management
    • Scientific Discovery and the Future of Medicine
    • Shared Decision Making and Communication
    • Sleep Medicine
    • Sports Medicine
    • Stem Cell Transplantation
    • Substance Use and Addiction Medicine
    • Surgery
    • Surgical Innovation
    • Surgical Pearls
    • Teachable Moment
    • Technology and Finance
    • The Art of JAMA
    • The Arts and Medicine
    • The Rational Clinical Examination
    • Tobacco and e-Cigarettes
    • Toxicology
    • Translational Medicine
    • Trauma and Injury
    • Treatment Adherence
    • Ultrasonography
    • Urology
    • Users' Guide to the Medical Literature
    • Vaccination
    • Venous Thromboembolism
    • Veterans Health
    • Violence
    • Women's Health
    • Workflow and Process
    • Wound Care, Infection, Healing

    Save Preferences

    Privacy Policy | Terms of Use

    X

    .

    ×

    Access your subscriptions

    Add or change institution

    Free access to newly published articles

    To register for email alerts, access free PDF, and more

    Purchase access

    Get full journal access for 1 year

    Get unlimited access and a printable PDF ($40.00)—
    Sign in or create a free account

    Rent this article from DeepDyve

    Access your subscriptions

    Add or change institution

    Free access to newly published articles

    To register for email alerts, access free PDF, and more

    Purchase access

    Get full journal access for 1 year

    Get unlimited access and a printable PDF ($40.00)—
    Sign in or create a free account

    Rent this article from DeepDyve

    Sign in to access free PDF

    Add or change institution

    Free access to newly published articles

    To register for email alerts, access free PDF, and more

    Save your search

    Free access to newly published articles

    To register for email alerts, access free PDF, and more

    Purchase access

    Customize your interests

    Free access to newly published articles

    To register for email alerts, access free PDF, and more

    Create a personal account or sign in to:

    • Register for email alerts with links to free full-text articles
    • Access PDFs of free articles
    • Manage your interests
    • Save searches and receive search alerts

      Privacy Policy

      Make a comment

      Free access to newly published articles

      To register for email alerts, access free PDF, and more

      Create a personal account or sign in to:

      • Register for email alerts with links to free full-text articles
      • Access PDFs of free articles
      • Manage your interests
      • Save searches and receive search alerts

        Privacy Policy

        Large Language Models Answer Medical Questions Accurately, but Can’t Match Clinicians’ Knowledge (2024)

        References

        Top Articles
        Latest Posts
        Article information

        Author: Ray Christiansen

        Last Updated:

        Views: 5767

        Rating: 4.9 / 5 (69 voted)

        Reviews: 92% of readers found this page helpful

        Author information

        Name: Ray Christiansen

        Birthday: 1998-05-04

        Address: Apt. 814 34339 Sauer Islands, Hirtheville, GA 02446-8771

        Phone: +337636892828

        Job: Lead Hospitality Designer

        Hobby: Urban exploration, Tai chi, Lockpicking, Fashion, Gunsmithing, Pottery, Geocaching

        Introduction: My name is Ray Christiansen, I am a fair, good, cute, gentle, vast, glamorous, excited person who loves writing and wants to share my knowledge and understanding with you.