Welcome to the fourth edition of the Medical Domain Visual Question Answering Task!

Description

Motivation

With the increasing interest in artificial intelligence (AI) to support clinical decision making and improve patient engagement, opportunities to generate and leverage algorithms for automated medical image interpretation are currently being explored. Since patients may now access structured and unstructured data related to their health via patient portals, such access also motivates the need to help them better understand their conditions regarding their available data, including medical images.

The clinicians' confidence in interpreting complex medical images could also be enhanced by a “second opinion” provided by an automated system. In addition, patients may be interested in the morphology/physiology and disease-status of anatomical structures around a lesion that has been well characterized by their healthcare providers – and they may not necessarily be willing to pay significant amounts for a separate office- or hospital visit just to address such questions. Although patients often turn to search engines (e.g. Google) to disambiguate complex terms or obtain answers to confusing aspects of a medical image, results from search engines may be nonspecific, erroneous and misleading, or overwhelming in terms of the volume of information.

News

12/22/2020: Website goes public.

Tasks Description

The fourth edition of VQA-Med includes two subtasks:
1) Visual Question Generation (VQG): consists in generating relevant natural language questions about radiology images based on their visual content.
2) Visual Question Answering (VQA): consists in answering natural language questions from the visual content of associated radiology images.

AIcrowd projects:

Previous editions:

VQA-Med 2020: https://www.imageclef.org/2020/medical/vqa (Datasets & Evaluation Code: https://github.com/abachaa/VQA-Med-2020)
VQA-Med 2019: https://www.imageclef.org/2019/medical/vqa (Datasets: https://github.com/abachaa/VQA-Med-2019)
VQA-Med 2018:https://www.imageclef.org/2018/VQA-Med

Join our mailing list: https://groups.google.com/d/forum/imageclef-vqa-med

Schedule

16 November 2020: Registration opens
6 March 2021: Release of the validation sets
29 April 2021: Release of the test sets
30 April 2021: Registration closes
10 May 2021: Run submission deadline
18 May 2021: Release of the processed results by the task organizers
28 May 2021: Submission of participant papers [CEUR-WS]
28 May – 11 June 2021: Review process of participant papers
11 June 2021: Notification of acceptance
2 July 2021: Camera ready copy of participant papers and extended lab overviews [CEUR-WS]
21-24 September 2021: The CLEF Conference, Bucharest, Romania.

Participant Registration

Please refer to the general ImageCLEF registration instructions

Data

Training Set: We will use the VQA-Med 2020 training data: https://www.aicrowd.com/challenges/imageclef-2020-vqa-med-vqa
Validation Set: Consists of 500 radiology images and associated questions/answers about Abnormality (e.g., “what is most alarming about this ultrasound image?”).
Test Set: 500 radiology images and related questions about Abnormality.

The VQA-Med 2021 datasets will be also used in the ImageCLEF 2021 Caption task.

Evaluation Methodology

The following preprocessing methodology would be applied before running the evaluation metrics on each answer for the visual question answering task:

Each answer is converted to lower-case
All punctuations are removed and the answer is tokenized to individual words

The evaluation will be conducted based on the following metrics:

Accuracy (Strict)
We use an adapted version of the accuracy metric from the general domain VQA task that considers exact matching of a participant provided answer and the ground truth answer.
BLEU
We use the BLEU metric to capture the similarity between a system-generated answer and the ground truth answer.

Submission Instructions

Task 1: Visual Question Answering

Each team is allowed to submit a maximum of 10 runs.
We expect a txt file with the following format for the result submission file: <Image-ID><|><Answer>
For example:

rjv03401|answer of the first question in one single line
AIAN-14-313-g002|answer of the second question
wjem-11-76f3|answer of the third question
You need to respect the following constraints:
• The separator between <Image-ID> and <Answer> has to be the pipe character (|).
• Each <Image-ID> of the test set must be included in the run file exactly once.

Task 2: Visual Question Generation

Each team is allowed to submit a maximum of 10 runs.
Each run can include from 1 to 7 questions per image.
We expect a txt file with the following format for the result submission file: <Image-ID><|><Question1><|><Question2><|>...<|><QuestionN> (N≤7)
The separator between <Image-ID> and each <Question> has to be the pipe character (|).
Each <Image-ID> of the test set must be included in the run file exactly once.

For both tasks, participants are allowed to use other resources asides from the official training/validation datasets, however, the use of the additional resources must have to be explicitly stated. For a meaningful comparison, we will separately group systems that exclusively use the official training data and who incorporate additional sources.

CEUR Working Notes

All participating teams with at least one graded submission should submit a CEUR working notes paper.
Official detailed instructions for the CLEF 2021 working notes can be found here: http://clef2021.clef-initiative.eu/index.php?page=Pages/instructions_for_authors.html

Citations

When referring to the ImageCLEF VQA-Med 2021 task general goals, evaluation, dataset, results, etc. please cite the following publication which will be published by September 2021:

BibTex:
@Inproceedings{ImageCLEF-VQA-Med2021,
author = {Asma {Ben Abacha} and Mourad Sarrouti and Dina Demner-Fushman and Sadid A. Hasan and Henning M\"uller},
title = {Overview of the VQA-Med Task at ImageCLEF 2021: Visual Question Answering and Generation in the Medical Domain},
booktitle = {CLEF 2021 Working Notes},
series = {{CEUR} Workshop Proceedings},
year = {2021},
volume = {},
publisher = {CEUR-WS.org},
pages = {},
month = {September 21-24},
address = {Bucharest, Romania}
}

When referring to the ImageCLEF 2021 tasks in general, please cite the following publication which will be published by September 2021:

@inproceedings{ImageCLEF2021,
author = {Bogdan Ionescu and Henning M\"uller and Renaud P\’{e}teri
and Asma {Ben Abacha} and Mourad Sarrouti and Dina Demner-Fushman and
Sadid A. Hasan and Vassili Kovalev and Serge Kozlovski and Vitali
Liauchuk and Yashin Dicente and Obioma Pelka and Alba Garc\’{\i}a Seco
de Herrera and Janadhip Jacutprakart and Christoph M. Friedrich and
Raul Berari and Andrei Tauteanu and Dimitri Fichou and Paul Brie and
Mihai Dogariu and Liviu Daniel \c{S}tefan and Mihai Gabriel Constantin
and Jon Chamberlain and Antonio Campello and Adrian Clark and Thomas
A. Oliver and Hassan Moustahfid and Adrian Popescu and J\’{e}r\^{o}me
Deshayes-Chossart},
title = {{Overview of the ImageCLEF 2021}: Multimedia Retrieval in
Medical, Nature, Internet and Social Media Applications},
booktitle = {Experimental IR Meets Multilinguality, Multimodality, and
Interaction},
series = {Proceedings of the 12th International Conference of the CLEF
Association (CLEF 2021)},
year = {2021},
volume = {},
publisher = {{LNCS} Lecture Notes in Computer Science, Springer},
pages = {},
month = {September 21-24},
address = {Bucharest, Romania}
}

Organizers

Asma Ben Abacha <asma.benabacha(at)nih.gov>, National Library of Medicine, USA
Mourad Sarrouti <mourad.sarrouti(at)nih.gov>, National Library of Medicine, USA
Dina Demner-Fushman <ddemner(at)mail.nih.gov>, National Library of Medicine, USA
Sadid A. Hasan <sadidhasan(at)gmail.com>, CVS Health, USA
Henning Müller <henning.mueller(at)hevs.ch>, University of Applied Sciences Western Switzerland, Sierre, Switzerland