ImageCLEFmed Caption

Welcome to the 11th edition of the Caption Task!

Motivation

Interpreting and summarizing the insights gained from medical images such as radiology output is a time-consuming task that involves highly trained experts and often represents a bottleneck in clinical diagnosis pipelines.

Consequently, there is a considerable need for automatic methods that can approximate this mapping from visual information to condensed textual descriptions. The more image characteristics are known, the more structured are the radiology scans and hence, the more efficient are the radiologists regarding interpretation. We work on the basis of a large-scale collection of figures from open access biomedical journal articles (PubMed Central). All images in the training data are accompanied by UMLS concepts extracted from the original image caption.

Lessons learned:

• In the first and second editions of this task, held at ImageCLEF 2017 and ImageCLEF 2018, participants noted a broad variety of content and situation among training images. In 2019, the training data was reduced solely to radiology images, with ImageCLEF 2020 adding additional imaging modality information, for pre-processing purposes and multi-modal approaches.
• The focus in ImageCLEF 2021 lay in using real radiology images annotated by medical doctors. This step aimed at increasing the medical context relevance of the UMLS concepts, but more images of such high quality are difficult to acquire.
• As uncertainty regarding additional source was noted, we will clearly separate systems using exclusively the official training data from those that incorporate additional sources of evidence.
• For ImageCLEF 2022, an extended version of the ImageCLEF 2020 dataset was used. For the caption prediction subtask, a number of different additional evaluations metrics were introduced with the goal of replacing the primary evaluation metric in future iterations of the task.
• For ImageCLEF 2023, several issues with the dataset (large number of concepts, lemmatization errors, duplicate captions) were tackled and based on experiments in the previous year, BERTScore was used as the primary evaluation metric for the caption prediction subtask.
• For ImageCLEF 2026, two synthetical subtasks, based on newly generated captions and concepts derived from them, were introduced alongside the standard subtasks, and the caption prediction ranking was based on an average score over relevance and factuality metrics. The best overall caption prediction score was considerably higher on the synthetical data (0.5840) than on the standard data (0.3755).

Schedule

• TBD: Website goes live
• TBD: Registration opens
• TBD: Development dataset released
• TBD: Test dataset released
• TBD: Deadline for submitting participant runs
• TBD: Release of processed results by the task organizers
• TBD: Submission of participant papers (CEUR-WS)
• TBD: Notification of acceptance
• TBD: Camera Ready Submission Deadline

Task Description

For captioning, participants will be requested to develop solutions for automatically identifying individual components from which captions are composed in Radiology Objects in COntext version 2[2] images. ImageCLEFmedical Caption 2027 consists of the following subtasks:

Subtask 1: TBD

Subtask 2: TBD

Subtask 3: TBD

Subtask 4: TBD

Subtask 5: TBD

Submission Limits

Baselines

Concept Detection Task

Caption Prediction Task

Data

Information will be added soon.

Evaluation methodology

Information will be added soon.

Certificates

Participant registration

Please refer to the general ImageCLEF registration instructions

Results

CEUR Working Notes

Dataset image attribution

If you include dataset images in your paper, you must provide the correct attribution. Use the lookup file below to find the attribution string for each image ID:

https://fh-dortmund.sciebo.de/s/KT9AMjPtoq3pxTz

Insert the attribution in the figure caption or directly beside the image.

Working Notes Papers should cite both the ImageCLEF 2027 overview paper as well as the ImageCLEFmedical task overview paper and the ROCOv2 dataset paper, citation information is available in the Citations section below.

Reproducibility

We encourage you to make your work as reproducible as possible by releasing code, trained models, and detailed instructions on a public repository (e.g. GitHub) and pointing to it in your paper.

ArXiv references

Please refrain from citing preprints (e.g. arXiv) without checking if they have been published in the meantime. The publication details proof the value of the cited work and give attribution to the authors; an arXiv reference is only a preprint with no peer review. You can try https://preprintresolver.eu/ to give proper attribution to authors and improve the quality of your work. If no proper reference/doi is found, you can use the arXiv reference.

Citations

Contact

Organizers:

• Hendrik Damm <hendrik.damm(at)fh-dortmund.de>, University of Applied Sciences and Arts Dortmund, Germany
• Tabea M. G. Pakull, <tabea.pakull(at)uk-essen.de>, Institute for Transfusion Medicine, University Hospital Essen, Germany
• Asma Ben Abacha <abenabacha(at)microsoft.com>, Microsoft, USA
• Alba García Seco de Herrera <alba.garcia(at)essex.ac.uk>, University of Essex, UK
• Christoph M. Friedrich <christoph.friedrich(at)fh-dortmund.de>, University of Applied Sciences and Arts Dortmund, Germany
• Henning Müller <henning.mueller(at)hevs.ch>, University of Applied Sciences Western Switzerland, Sierre, Switzerland
• Raphael Brüngel <raphael.bruengel(at)fh-dortmund.de>, University of Applied Sciences and Arts Dortmund, Germany
• Lea Reinartz, University of Applied Sciences and Arts Dortmund, Germany
• Praveen Nath, University of Applied Sciences and Arts Dortmund, Germany
• Henning Schäfer <henning.schaefer(at)uk-essen.de>, Institute for Transfusion Medicine, University Hospital Essen, Germany
• Cynthia S. Schmidt, Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen
• Benjamin Bracke, University of Applied Sciences and Arts Dortmund, Germany
• Bahadir Eryilmaz, Institute for Artificial Intelligence in Medicine, Germany

Acknowledgments

[1] Rückert, J., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., Schäfer, H., Schmidt, C. S., Koitka, S., Pelka, O., Abacha, A. B., de Herrera, A. G. S., Müller, H., Horn, P. A., Nensa, F., & Friedrich, C. M. (2024). ROCOv2: Radiology objects in COntext version 2, an updated multimodal image dataset. https://doi.org/10.48550/ARXIV.2405.10004