Med-PaLM and Med-PaLM 2

What it is

Med-PaLM and Med-PaLM 2 are large language models built to answer medical questions, introduced and evaluated in two arXiv papers. The first, Large Language Models Encode Clinical Knowledge, was posted on 26 December 2022^[1]. The second, Towards Expert-Level Medical Question Answering with Large Language Models, was submitted on 16 May 2023 and covers Med-PaLM 2^[2]. Both papers centre on the same task: reading a question written in ordinary language — exam-style items as well as questions people actually type into a search box — and producing an answer in text.

Where it came from

The first paper also introduced a benchmark for measuring such models. MultiMedQA combines six existing medical question-answering datasets with HealthSearchQA, a new dataset of medical questions that people searched for online^[1]. The starting point for the experiments was Flan-PaLM, an instruction-tuned version of PaLM. On MedQA, a set of questions in the style of the US Medical Licensing Exam, Flan-PaLM reached 67.6%, surpassing the prior state of the art by over 17%^[1].

Med-PaLM itself was produced from Flan-PaLM by instruction prompt tuning, a parameter-efficient method that uses a few exemplars^[1]. The idea is not to retrain the whole model but to learn a small set of additional prompt parameters from a handful of worked examples, steering the model toward a task at modest cost.

How it was judged

The papers did not rely on multiple-choice accuracy alone. Human evaluation of Flan-PaLM's responses revealed key gaps, according to the first paper^[1], and the tuned Med-PaLM, though judged encouragingly, remained inferior to clinicians^[1]. Automatic scores on standard datasets plus clinician review of free-text answers frame what the numbers mean: a high exam score shows the model can pick the right option or produce a plausible answer, while human grading asks whether that answer is correct, complete and safe.

Med-PaLM 2

The follow-up paper reports Med-PaLM 2 scoring up to 86.5% on MedQA, improving on Med-PaLM by over 19%^[2]. The gain came within months of the first paper.

Where it sits in the development of medical AI

Med-PaLM matters less as a product than as a way of setting up the problem. It gave the field a shared public benchmark, MultiMedQA, and a reporting style that sets automatic accuracy beside human review^[1]. The move from 67.6% to 86.5% on MedQA became a reference point for later medical language models^[1][2]. The papers' own human evaluations also mark the limit: benchmark scores climbed faster than clinical judgement, which the first paper found the model had not reached^[1]. Exam-style questions test whether a model can select or generate the expected answer; they do not by themselves show how it would behave in front of a patient, which is why the authors paired the scores with review by clinicians.