You are here

Visual Question Answering in the Medical Domain

Welcome to the 2nd edition of the Medical Domain Visual Question Answering Task!

Description

Motivation

With the increasing interest in artificial intelligence (AI) to support clinical decision making and improve patient engagement, opportunities to generate and leverage algorithms for automated medical image interpretation are currently being explored. Since patients may now access structured and unstructured data related to their health via patient portals, such access also motivates the need to help them better understand their conditions regarding their available data, including medical images.

The clinicians' confidence in interpreting complex medical images can be significantly enhanced by a “second opinion” provided by an automated system. In addition, patients may be interested in the morphology/physiology and disease-status of anatomical structures around a lesion that has been well characterized by their healthcare providers – and they may not necessarily be willing to pay significant amounts for a separate office- or hospital visit just to address such questions. Although patients often turn to search engines (e.g. Google) to disambiguate complex terms or obtain answers to confusing aspects of a medical image, results from search engines may be nonspecific, erroneous and misleading, or overwhelming in terms of the volume of information.

News

  • 09.11.2018: Website goes live.
  • 31.01.2019: Training and validation datasets released.
  • 18.03.2019: Test dataset released.

Task Description

Visual Question Answering is an exciting problem that combines natural language processing and computer vision techniques. Inspired by the recent success of visual question answering in the general domain, we conducted a pilot task in ImageCLEF 2018 to focus on visual question answering in the medical domain. Based on the success of the inaugural edition and the huge interest from both computer vision and medical informatics communities, we will continue the task this year with enhanced focus on a nicely curated enlarged dataset. Same as last year, given a medical image accompanied with a clinically relevant question, participating systems are tasked with answering the question based on the visual image content.

Data

The datasets include a training set of 3,200 medical images with 12,792 Question-Answer (QA) pairs, a validation set of 500 medical images with 2,000 QA pairs, and a test set of 500 medical images with 500 questions. In order to generate a more-focused set of questions for a meaningful task evaluation, we considered generating 4 categories of questions based on: Modality, Plane, Organ System, and Abnormality. Please see the readme file of the crowdAI dataset section for more detailed information.

Evaluation Methodology

The following pre-processing methodology is applied before running the evaluation metrics on each answer:

  • Each answer is converted to lower-case
  • All punctuations are removed and the answer is tokenized to individual words

The evaluation would be conducted based on the following metrics:

  1. Accuracy (Strict)
    We use an adapted version of the accuracy metric from the general domain VQA task that considers exact matching of a participant provided answer and the ground truth answer.
  2. BLEU
    We use the BLEU metric [1] to capture the similarity between a system-generated answer and the ground truth answer. The overall methodology and resources for the BLEU metric are essentially similar to the ImageCLEF 2017 caption prediction task.
  3. References:

    [1] Papineni, K.; Roukos, S.; Ward, T.; Zhu, W. J. (2002). BLEU: a method for automatic evaluation of machine translation (PDF). ACL-2002: 40th Annual meeting of the Association for Computational Linguistics. pp. 311–318.

    Preliminary Schedule

  • 19.11.2018: Registration opens for all ImageCLEF tasks (until 26.04.2019)
  • 31.01.2019: training and validation data release starts
  • 18.03.2019: Test data release starts
  • 01.05.2019: Deadline for submitting the participants runs
  • 13.05.2019: Release of the processed results by the task organizers
  • 24.05.2019: Deadline for submission of working notes papers by the participants
  • 07.06.2019: Notification of acceptance of the working notes papers
  • 28.06.2019: Camera-ready working notes papers
  • 09-12.09.2019CLEF 2019, Lugano, Switzerland

CrowdAI is shutting down and will move towards AICrowd. Please temporarily ignore the information below this paragraph. During the transition phase (until all challenges are migrated) we will have to provide the datasets and End User Agreement (EUA) handling ourselves. In order to get access to the dataset, please download the EUA at the bottom of this page and send a filled in and signed version to sadid.hasan(at)philips.com. Please refer to the ImageCLEF registration instructions to get some examples on how to fill in the EUA.

Participant Registration

Please refer to the general ImageCLEF registration instructions

Submission Instructions

  • Each team is allowed to submit a maximum of 10 runs.
  • We expect the following format for the result submission file: <Image-ID><|><Answer>

    For example:

    rjv03401|answer of the first question in one single line
    AIAN-14-313-g002|answer of the second question
    wjem-11-76f3|answer of the third question

  • You need to respect the following constraints:

    • The separator between <Image-ID> and <Answer> has to be the pipe character (|).
    • Each <Image-ID> of the test set must be included in the run file exactly once.
    • All <Image-ID> must be present in a participant’s run file in the same order as the given test file.

  • Participants are allowed to use other resources asides from the official training/validation datasets, however the use of the additional resources must have to be explicitly stated. For meaningful comparison, we will separately group systems who exclusively use the official training data and who incorporate additional sources.

CEUR Working Notes

  • All participating teams with at least one graded submission should submit a CEUR working notes paper.
  • The working notes paper should be submitted using this link:
    https://easychair.org/conferences/?conf=clef2019
    Click on "enter as an author", then select track "ImageCLEF - Multimedia Retrieval in CLEF".
    Add author information, paper title/abstract, keywords, select "Task 3 - ImageCLEFmedical" and upload your working notes paper as pdf.

Citations

When referring to the ImageCLEF VQA-Med 2019 task general goals, evaluation, dataset, results, etc. please cite the following publication which will be published by September 2019:

  • BibTex:
    @Inproceedings{ImageCLEFVQA-Med2019,
    author = {Asma {Ben Abacha} and Sadid A. Hasan and Vivek V. Datla and Joey Liu and Dina Demner-Fushman and Henning M\"uller},
    title = {{VQA-Med}: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019},
    booktitle = {CLEF2019 Working Notes},
    series = {{CEUR} Workshop Proceedings},
    year = {2019},
    volume = {},
    publisher = {CEUR-WS.org $<$http://ceur-ws.org$>$},
    pages = {},
    month = {September 09-12},
    address = {Lugano, Switzerland},
    }
  • When referring to the ImageCLEF 2019 tasks in general, please cite the following publication to be published by September 2019:

    • BibTex:

      @inproceedings{ImageCLEF19, author = {Bogdan Ionescu and Henning M\"uller and Renaud P\'{e}teri and Yashin Dicente Cid and Vitali Liauchuk and Vassili Kovalev and Dzmitri Klimuk and Aleh Tarasau and Asma {Ben Abacha} and Sadid A. Hasan and Vivek Datla and Joey Liu and Dina Demner-Fushman and Duc-Tien Dang-Nguyen and Luca Piras and Michael Riegler and Minh-Triet Tran and Mathias Lux and Cathal Gurrin and Obioma Pelka and Christoph M. Friedrich and Alba Garc\'ia Seco de Herrera and Narciso Garcia and Ergina Kavallieratou and Carlos Roberto del Blanco and Carlos Cuevas Rodr\'{i}guez and Nikos Vasillopoulos and Konstantinos Karampidis and Jon Chamberlain and Adrian Clark and Antonio Campello}, title = {{ImageCLEF 2019}: Multimedia Retrieval in Medicine, Lifelogging, Security and Nature}, booktitle = {Experimental IR Meets Multilinguality, Multimodality, and Interaction}, series = {Proceedings of the 10th International Conference of the CLEF Association (CLEF 2019)}, year = {2019}, volume = {}, publisher = {{LNCS} Lecture Notes in Computer Science, Springer}, pages = {}, month = {September 9-12}, address = {Lugano, Switzerland}}

    Organizers

    • Asma Ben Abacha <asma.benabacha(at)nih.gov>, National Library of Medicine, USA
    • Sadid A. Hasan <sadid.hasan(at)philips.com>, Philips Research Cambridge, USA
    • Vivek Datla <vivek.datla(at)philips.com>, Philips Research Cambridge, USA
    • Joey Liu <joey.liu(at)philips.com>, Philips Research Cambridge, USA
    • Dina Demner-Fushman <ddemner(at)mail.nih.gov>, National Library of Medicine, USA
    • Henning Müller <henning.mueller(at)hevs.ch>, University of Applied Sciences Western Switzerland, Sierre, Switzerland

    Join our mailing list: https://groups.google.com/d/forum/imageclef-vqa-med

    Acknowledgements

    AttachmentSize
    PDF icon ImageCLEFmedVQA2019EndUserAgreement.pdf612.95 KB