PubMedQA Homepage (original) (raw)

About

The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts.

For more details about PubMedQA, please refer to this paper:

Dataset

PubMedQA has 1k expert labeled, 61.2k unlabeled and 211.3k artificially generated QA instances.

Please visit our GitHub repository to download the dataset:

Submission

To submit your model, please follow the instructions in the GitHub repository.

Citation

If you use PubMedQA in your research, please cite our paper by:

@inproceedings{jin2019pubmedqa, title={PubMedQA: A Dataset for Biomedical Research Question Answering}, author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua}, booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, pages={2567--2577}, year={2019} }

Model Code Size Accuracy (%) Macro-F1 (%)
1 Nov 28, 2023 GPT-4 (Medprompt) Microsoft(Nori et al. 2023) NA 82.0 NA
2 May 16, 2023 Med-PaLM 2 Google Research & DeepMind(Singhal et al. 2023) NA 81.8 NA
3 Nov 27, 2023 MEDITRON EPFL(Chen et al. 2023) 70B 81.6 NA
4 Jul 6, 2023 Palmyra-Med Writer Inc.(Kamble et al. 2023) 40B 81.1 NA
5 Dec 1, 2023 AntGLM-Med Ant Group(Li et al. 2023) 10B 80.6 NA
6 Apr 12, 2023 GPT-4-base Microsoft & OpenAI(Nori et al. 2023) NA 80.4 NA
7 Mar 4, 2024 Claude 3 Anthropic(Anthropic, 2024) NA 79.7 NA
8 Jan 11, 2023 GPT-3.5 + Z-Code++ Microsoft Azure AI(He et al. 2022) 175B 79.6 55.8
9 Dec 26, 2022 Flan-PaLM (3-shot) Google Research & DeepMind(Singhal et al. 2022) 540B 79.0 NA
10 Mar 14, 2024 HEAL DeepScribe Inc(Yuan et al. 2024) 13B 78.4 NA
11 Dec 20, 2022 Codex (5-shot) Technical University of Denmark & Copenhagen University Hospital(LiƩvin et al. 2022) 175B 78.2 NA
12 Sep 13, 2019 Human Performance University of Pittsburgh & Carnegie Mellon University(Jin et al. 2019) NA 78.0 72.2
13 Nov 16, 2022 Galactica Meta AI(Taylor et al. 2022) 120B 77.6 NA
14 May 22, 2023 GatorTronGPT University of Florida & NVIDIA(Peng et al. 2023) 20B 77.6 NA
15 Mar 4, 2024 MediSwift-XL Cerebras Systems(Thangarasa et al. 2024) 1.2B 76.8 NA
16 Mar 20, 2023 GPT-4 Microsoft & OpenAI(Nori et al. 2023) NA 75.2 NA
17 Apr 18, 2024 Reka Core Reka(Ormazabal et al. 2024) NA 74.6 NA
18 Dec 15, 2022 PubMedGPT Stanford University(Bolton et al. 2022) 2.7B 74.4 NA
19 Oct 17, 2022 DRAGON Stanford University & EPFL(Yasunaga et al. 2022) 360M 73.4 NA
20 Apr 27, 2023 PMC-LLaMA Shanghai Jiao Tong University(Wu et al. 2023) 7B 73.4 NA
21 Mar 29, 2022 BioLinkBERT (large) Stanford University(Yasunaga et al. 2022) 340M 72.2 NA
22 Mar 29, 2022 BioLinkBERT (base) Stanford University(Yasunaga et al. 2022) 110M 70.2 NA
23 Sep 13, 2019 BioBERT (multi-phase tuning) University of Pittsburgh & Carnegie Mellon University(Jin et al. 2019) 110M 68.1 52.7
24 June, 2021 BioELECTRA SAAMA AI Research Lab(Kanakarajan et al. 2019) 110M 64.0 NA
25 July 31, 2020 PubMedBERT Microsoft Research(Gu et al. 2020) 110M 55.8 NA