ImageCLEFmed Caption

Welcome to the 10th edition of the Caption Task!

Motivation

Interpreting and summarizing the insights gained from medical images such as radiology output is a time-consuming task that involves highly trained experts and often represents a bottleneck in clinical diagnosis pipelines.

Consequently, there is a considerable need for automatic methods that can approximate this mapping from visual information to condensed textual descriptions. The more image characteristics are known, the more structured are the radiology scans and hence, the more efficient are the radiologists regarding interpretation. We work on the basis of a large-scale collection of figures from open access biomedical journal articles (PubMed Central). All images in the training data are accompanied by UMLS concepts extracted from the original image caption.

Lessons learned:

• In the first and second editions of this task, held at ImageCLEF 2017 and ImageCLEF 2018, participants noted a broad variety of content and situation among training images. In 2019, the training data was reduced solely to radiology images, with ImageCLEF 2020 adding additional imaging modality information, for pre-processing purposes and multi-modal approaches.
• The focus in ImageCLEF 2021 lay in using real radiology images annotated by medical doctors. This step aimed at increasing the medical context relevance of the UMLS concepts, but more images of such high quality are difficult to acquire.
• As uncertainty regarding additional source was noted, we will clearly separate systems using exclusively the official training data from those that incorporate additional sources of evidence.
• For ImageCLEF 2022, an extended version of the ImageCLEF 2020 dataset was used. For the caption prediction subtask, a number of different additional evaluations metrics were introduced with the goal of replacing the primary evaluation metric in future iterations of the task.
• For ImageCLEF 2023, several issues with the dataset (large number of concepts, lemmatization errors, duplicate captions) were tackled and based on experiments in the previous year, BERTScore was used as the primary evaluation metric for the caption prediction subtask.

Schedule

• 01.12.2025: Website goes live
• 26.01.2026: Registration opens
• 15.02.2026: Development dataset released
• 19.04.2026: Test dataset released
• 30.04.2026: Deadline for submitting participant runs
• 07.05.2026: Release of processed results by the task organizers
• 28.05.2026: Submission of participant papers (CEUR-WS)
• 30.06.2026: Notification of acceptance
• 06.07.2026: Camera Ready Submission Deadline

Task Description

For captioning, participants will be requested to develop solutions for automatically identifying individual components from which captions are composed in Radiology Objects in COntext version 2[2] images. ImageCLEFmedical Caption 2026 consists of the following subtasks:

Standard Tasks

• Concept Detection
• Caption Prediction

Synthetical Tasks NEW

• Concept Detection Synthetical
• Caption Prediction Synthetical

New this year, we introduce two synthetical subtasks. For these tasks, we generated new captions and derived new concepts from those captions. The evaluation methodology remains the same as for the standard tasks, with the exception that there is no secondary score for the Synthetical Concept Detection subtask.

Explainability Task

In addition, we ask participants to provide explanations for the captions of a small subset (will be released with the test dataset) of images. We encourage people to be creative. There are no technical limitations to this task. The explanations will be manually evaluated by a radiologist for interpretability, relevance, and creativity. Examples of how such an explanation might look like are provided as follows:

Submission Limits

• Development set: Up to 10 submissions (maximum 5 per day)
• Test set: Up to 3 submissions

Baselines

We provide trained baselines for reference:

• Caption Prediction: Qwen3-4B-VL, fine-tuned with Unsloth
• Concept Detection: EfficientNet-based model

Concept Detection Task

The first step to automatic image captioning and scene understanding is identifying the presence and location of relevant concepts in a large corpus of medical images. Based on the visual image content, this subtask provides the building blocks for the scene understanding step by identifying the individual components from which captions are composed. The concepts can be further applied for context-based image and information retrieval purposes.

Evaluation is conducted in terms of set coverage metrics such as precision, recall, and combinations thereof.

Caption Prediction Task

On the basis of the concept vocabulary detected in the first subtask as well as the visual information of their interaction in the image, participating systems are tasked with composing coherent captions for the entirety of an image. In this step, rather than the mere coverage of visual concepts, detecting the interplay of visible elements is crucial for strong performance.

This year, we will use BERTScore as the primary evaluation metric and ROUGE as the secondary evaluation metric for the caption prediction subtask. Other metrics such as MedBERTScore, MedBLEURT, and BLEU will also be published.

Data

The data for the caption task will contain curated images from the medical literature including their captions and associated UMLS terms that are manually controlled as metadata. A more diverse data set will be made available to foster more complex approaches.

For questions regarding the dataset please use the challenge website forum or contact hendrik.damm@fh-dortmund.de.

For the development dataset, Radiology Objects in COntext Version 2 (ROCOv2) [2], an updated and extended version of the Radiology Objects in COntext (ROCO) dataset [1], is used for both subtasks. As in previous editions, the dataset originates from biomedical articles of the PMC OpenAccess subset, with the test set comprising a previously unseen set of images.

• Training Set: Consists of tba radiology images
• Validation Set: Consists of tba radiology images
• Test Set: Consists of tba radiology images

Concept Detection Task

The concepts were generated using a reduced subset of the UMLS 2022 AB release. To improve the feasibility of recognizing concepts from the images, concepts were filtered based on their semantic type. Concepts with low frequency were also removed, based on suggestions from previous years.

Caption Prediction Task

For this task each caption is pre-processed in the following way:

• removal of links from the captions

Evaluation methodology

The source code of the evaluation is available on Github (https://github.com/taubsity/clef-caption-evaluation-2026).

The repository also provides scripts to check submission formats.

For questions regarding the evaluation scripts please use the challenge website forum or contact tabea.pakull@uk-essen.de.

Concept Detection

Evaluation is conducted in terms of F1 scores between system predicted and ground truth concepts, using the following methodology and parameters:

• The default implementation of the Python scikit-learn (v0.17.1-2) F1 scoring method is used. It is documented here.
• A Python (3.x) script loads the candidate run file, as well as the ground truth (GT) file, and processes each candidate-GT concept sets.
• For each candidate-GT concept set, the y_pred and y_true arrays are generated. They are binary arrays indicating for each concept contained in both candidate and GT set if it is present (1) or not (0).
• The F1 score is then calculated. The default 'binary' averaging method is used.
• All F1 scores are summed and averaged over the number of elements in the test set, giving the final score.
• The primary score considers any concept. The secondary score filters both predicted and GT concepts to the set of manually annotated concepts before repeating the same F1 scoring steps. Note: There is no secondary score for the Synthetical Concept Detection subtask.

The ground truth for the test set was generated based on the same reduced subset of the UMLS 2022 AB release which was used for the training data (see above for more details).

Caption Prediction

This year, ranking of participants is based on an average score over all used metrics. In total 6 metrics are computed that fall into the aspects of relevance and factuality.

Relevance

In order to evaluate the relevance aspect of generated captions the following metrics are used:

• Image and Caption Similarity
• BERT-Score (Recall) with inverse document frequency (idf) scores computed from the test corpus for importance weighting
• Recall-Oriented Understudy for Gisting Evaluation (ROUGE) for overlap of unigrams (ROUGE-1) (F-measure)
• Bilingual Evaluation Understudy with Representations from Transformers (BLEURT)

Image and Caption Similarity is computed using the following methodology:

Using a medical imaging embedding model for calculating embeddings of the caption and the image and calculating similarity of these embeddings.

BERTScore is calculated using the following methodology and parameters:

The native Python implementation of BERTScore is used. This scoring method is based on the paper "BERTScore: Evaluating Text Generation with BERT" and aims to measure the quality of generated text by comparing it to a reference. We use Recall BERTScore with inverse document frequency (idf) scores computed from the test corpus for importance weighting as this setting correlates the most with human ratings for the image captioning task reported in the BERTScore paper.

To calculate BERTScore, we use the microsoft/deberta-xlarge-mnli model, which can be found on the Hugging Face Model Hub. The model is pretrained on a large corpus of text and fine-tuned for natural language inference tasks. It can be used to compute contextualized word embeddings, which are essential for BERTScore calculation.

To compute the final BERTScore, we first calculate the individual score (Recall idf) for each caption. The BERTScore is then averaged across all captions to give the final score.

The ROUGE score is calculated using the following methodology and parameters:

The native python implementation of ROUGE scoring method is used. It is designed to replicate results from the original perl package that was introduced in the paper "ROUGE: A Package for Automatic Evaluation of Summaries".

Specifically, we calculate the ROUGE-1 (F-measure) score, which measures the number of matching unigrams between the model-generated text and a reference. The final score is the average ROUGE-1 over all captions.

For calculation of BLEURT the following methodology and parameters are used:

The native Python implementation of BLEURT is used. This scoring method is based on the paper "BLEURT: Learning Robust Metrics for Text Generation". The aim of BLEURT is to provide an evaluation metric for text generation by learning from human judgments using BERT-based representations. In this evaluation, the recommended BLEURT-20 checkpoint is employed.

Factuality

In order to evaluate the factuality aspect of generated captions the following metrics are used:

• Unified Medical Language System (UMLS) Concept F1
• AlignScore

We calculate the UMLS F1 using the following methodology:

We use MedCAT to get the medical entities (UMLS concepts) of the caption and the predicted caption. We only match entities with the semantic types that are used to calculate MEDCON as described in the paper "Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation".

The AlignScore is calculated using the following methodology and parameters:

The native Python implementation of the AlignScore is used. It implements the metric based on RoBERTa introduced in the paper "AlignScore: Evaluating Factual Consistency with A Unified Alignment Function". The checkpoints are available on the Huggingface Model Hub. The AlignScore is designed to evaluate factual consistency in text generation by assessing the alignment of information between two pieces of text. The model calculates a score by splitting long contexts into manageable chunks and matching each claim sentence with the most supportive context chunk. The final AlignScore is the average alignment score across all claim sentences.

Certificates

This year, we will provide certificates for participating teams that submit working notes, as well as for the best-ranked teams per subtask.

Participant registration

Please refer to the general ImageCLEF registration instructions

Results

Note to the participants: Use the official leaderboards on sciebo to report your results.

Explainability:

The complete evaluation files are available here: https://portal.uk-essen.de/enqsig/link?id=BAgAAAAJv8BhoGWDRokAAADJpWasQj89hCJV7oZDXFby4doGcgUknLjkoPsElm0v-xlwVtTUHVlucxwcIY7_4tCPqGTp8WDQSM1Ji2tyvQnJ71xtt1mMEdp_fMM_mDFkIkSVWfdqxJfEHNhMOFzbH9rIhsRAW5rUhAZptb1gCWWfvTKBI5KfCGuoCQ7aPdFyj9zXJ46spqa6Jw2

Password: ALMZHQd5sD

Caption, Caption Synthetic, Concept, Concept Synthetic:

The complete evaluation files are available here: https://portal.uk-essen.de/enqsig/link?id=BAgAAACWnrM8NrmR24kAAABC9CxGTgB9jx8FdIN5HLdRQgCwyFE-u25ycZSM49XTQ3WX_7-xn5rmUlzcAIupiz23VheFZXHGqnlxa7HNaaFp86_QpOlljytIGkFNXJOmYdsOfd2n9_-zPNd-jAHHdKk6wpyzvW_UKyqZpwxLMXrXYPwtEOWSkt05jZIkC6LKGIiSiXfkSwb4ig2

Password: clef_caption_2026

Concept Detection

Concept Detection (Synthetic)

Caption Prediction

Caption Prediction (Synthetic)

Explainability Task

CEUR Working Notes

The working-notes paper is your opportunity to describe your approach, present all submitted runs and discuss the results. All participating teams with at least one graded submission, regardless of the score, should submit a CEUR working notes paper. Teams who participated in both tasks should generally submit only one report.

Use the official CEUR-WS template

https://clef2026.clef-initiative.eu/calls/submitting/#labs-working-notes--task-overview-papers

Make sure the EasyChair metadata (author names and order, title, affiliations) exactly match the PDF, as these fields feed directly into the proceedings.

• Camera-ready + signed copyright form:28.05.2026

Dataset image attribution

If you include dataset images in your paper, you must provide the correct attribution. Use the lookup file below to find the attribution string for each image ID:

https://fh-dortmund.sciebo.de/s/KT9AMjPtoq3pxTz

Insert the attribution in the figure caption or directly beside the image.

Working Notes Papers should cite both the ImageCLEF 2026 overview paper as well as the ImageCLEFmedical task overview paper and the ROCOv2 dataset paper, citation information is available in the Citations section below.

Reproducibility

We encourage you to make your work as reproducible as possible by releasing code, trained models, and detailed instructions on a public repository (e.g. GitHub) and pointing to it in your paper.

ArXiv references

Please refrain from citing preprints (e.g. arXiv) without checking if they have been published in the meantime. The publication details proof the value of the cited work and give attribution to the authors; an arXiv reference is only a preprint with no peer review. You can try https://preprintresolver.eu/ to give proper attribution to authors and improve the quality of your work. If no proper reference/doi is found, you can use the arXiv reference.

Citations

It is mandatory to cite the overview papers and the dataset.

Contact

Organizers:

• Hendrik Damm <hendrik.damm(at)fh-dortmund.de>, University of Applied Sciences and Arts Dortmund, Germany
• Tabea M. G. Pakull, <tabea.pakull(at)uk-essen.de>, Institute for Transfusion Medicine, University Hospital Essen, Germany
• Asma Ben Abacha <abenabacha(at)microsoft.com>, Microsoft, USA
• Alba García Seco de Herrera <alba.garcia(at)essex.ac.uk>, University of Essex, UK
• Christoph M. Friedrich <christoph.friedrich(at)fh-dortmund.de>, University of Applied Sciences and Arts Dortmund, Germany
• Henning Müller <henning.mueller(at)hevs.ch>, University of Applied Sciences Western Switzerland, Sierre, Switzerland
• Raphael Brüngel <raphael.bruengel(at)fh-dortmund.de>, University of Applied Sciences and Arts Dortmund, Germany
• Lea Reinartz, University of Applied Sciences and Arts Dortmund, Germany
• Praveen Nath, University of Applied Sciences and Arts Dortmund, Germany
• Henning Schäfer <henning.schaefer(at)uk-essen.de>, Institute for Transfusion Medicine, University Hospital Essen, Germany
• Cynthia S. Schmidt, Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen
• Benjamin Bracke, University of Applied Sciences and Arts Dortmund, Germany
• Bahadir Eryilmaz, Institute for Artificial Intelligence in Medicine, Germany

Acknowledgments

[1] Rückert, J., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., Schäfer, H., Schmidt, C. S., Koitka, S., Pelka, O., Abacha, A. B., de Herrera, A. G. S., Müller, H., Horn, P. A., Nensa, F., & Friedrich, C. M. (2024). ROCOv2: Radiology objects in COntext version 2, an updated multimodal image dataset. https://doi.org/10.48550/ARXIV.2405.10004