You are here

ImageCLEFmed Tuberculosis

Motivation

Description

Welcome to the 4th edition of the Tuberculosis Task!

Tuberculosis (TB) is a bacterial infection caused by a germ called Mycobacterium tuberculosis. About 130 years after its discovery, the disease remains a persistent threat and a leading cause of death worldwide according to WHO. This bacteria usually attacks the lungs, but it can also damage other parts of the body. Generally, TB can be cured with antibiotics. However, the different types of TB require different treatments, and therefore the detection of the TB type and the evaluation of lesion characteristics are important real-world tasks.

Lessons learned:

  • Tuberculosis task exists in ImageCLEF since 2017 and it was modified from year to year.
  • In the first and second editions of this task, held at ImageCLEF 2017 and ImageCLEF 2018, participants had to detect Multi-drug resistant patients (MDR subtask) and to classify the TB type (TBT subtask) both based only on the CT image. After 2 editions we concluded that the MDR subtask was not possible to solve based only on the image. In the TBT subtask, there was a slight improvement in 2018 with respect to 2017 on the classification results, however, not enough considering the amount of extra data provided in the 2018 edition, both in terms of new images and meta-data.
  • In the third edition Tuberculosis task was restructured to allow usage of uniform dataset, and included two subtasks - continued Severity Score (SVR) prediction subtask and a new subtask based on providing an automatic report (CT Report) on the TB case.
  • In this year edition, we decided to concentrate on the automated CT report generation task, since it has important outcome that can have a major impact in the real-world clinical routines. In order to make the task both more attractive for participants and practically valuable, this year report generation is lung-based rather than CT-based, which means the labels for left and right lungs will be provided independently. The set of target labels in the CT Report was updated with accordance to the opinion of medical experts. This year we provide 3 labels for each lung: presence of TB lesions in general, presence of pleurisy and caverns in particular. Also the dataset size was increased compared to the previous year.

News

  • 10.12.2019: Task web-page goes live
  • 20.01.2020: Task web-page updated. Finishing competition page on AIcrowd
  • 27.01.2020: Competition page available on AIcrowd

Task description

In this task participants have to generate automatic lung-wise reports based on the CT image data.
Each report should include the probability scores (ranging from 0 to 1) for each of the three labels and for each of the lungs (resulting in 6 entries per CT). The resulting list of entries includes: LeftLungAffected, RightLungAffected, CavernsLeft, CavernsRight, PleurisyLeft, PleurisyRight.

Data

In this edition, a dataset containing chest CT scans of 403 (283 for train and 120 for test) TB patients is used. Since the labels are provided on lung-wise scale rather than CT-wise scale, the total number of cases is virtually increased twice.

For all patients we provide 3D CT images with an image size per slice of 512*512 pixels and number of slices being around 100. All the CT images are stored in NIFTI file format with .nii.gz file extension (g-zipped .nii files). This file format stores raw voxel intensities in Hounsfield units (HU) as well the corresponding image metadata such as image dimensions, voxel size in physical units, slice thickness, etc. A freely-available tool called "VV" can be used for viewing the image files. Currently, there are various tools available for reading and writing NIFTI files. Among them there are load_nii and save_nii functions for Matlab and Niftilib library for C, Java, Matlab and Python, NiBabel package for Python.

Moreover, for all patients we provide two versions of automatically extracted masks of the lungs. These data can be downloaded together with the patients CT images.
The first version of segmentation was retrieved using the same technique as the previous years. The details of this segmentation can be found here.
The second version of segmentation was retrieved using non-rigid image registration scheme. The details of this segmentation and open-source implementation can be found here.
The first version of segmentation provides more accurate masks, but it tends to miss large abnormal regions of lungs in the most severe TB cases. The second segmentation on the contrary provides more rough bounds, but behaves more stable in terms of including lesion areas.
In case the participants use the provided masks in their experiments, please refer to the section "Citations" at the end of this page to find the appropriate citation for the corresponding lung segmentation technique.

Evaluation methodology

This task is considered as a multi-binary classification problem.
The ranking of this task will be done first by average AUC and then by min AUC over the 3 target labels.
The AUC values will be evaluated in a lung-wise manner.

Preliminary Schedule

  • 22.01.2020: registration opens (until 27.04.2020)
  • 22.01.2020: development data release starts
  • 27.03.2020: test data release starts (tentative)
  • 05.06.2020: deadline for submitting the participants runs
  • 12.06.2020: release of the processed results by the task organizers
  • 10.07.2020: deadline for submission of working notes papers by the participants
  • 07.08.2020: notification of acceptance of the working notes papers
  • 21.08.2020: camera ready working notes papers
  • 22-25.09.2020: CLEF 2020, Thessaloniki, Greece - foreseen as a virtual conference

Participant registration

Please refer to the general ImageCLEF registration instructions

Submission instructions

Submit a plain text file named with the prefix CTR (e.g. CTRfree-text.txt) with the following format:

  • <Filename>,<Probability of "left lung affected">,<Probability of "right lung affected">,<Probability of "presence of caverns in the left lung">,<Probability of "presence of caverns in the right lung">,<Probability of "pleurisy in the left lung">,<Probability of "pleurisy in the right lung">

e.g.:

  • CTR_TST_001.nii.gz,0.89,0.1,0.84,0.05,0.9,0.2
  • CTR_TST_002.nii.gz,0.1,0.6,0.222,0.333,0.444,0.55
  • CTR_TST_003.nii.gz,0.1,0.7,0.0,0.2,0.1,0.46
  • CTR_TST_004.nii.gz,0.88,0.78,0.59,0.65,0.8,0.4

You need to respect the following constraints:

  • Filenames must be the same as the original test file names
  • All filenames must be present in the runfiles
  • Only use numbers between 0 and 1 for the probabilities. Use the dot (.) as a decimal point (no commas accepted)

Results

DISCLAIMER : The results presented below have not yet been analyzed in-depth and are shown "as is". The results are sorted by descending mean AUC (if those are equal, by descending min AUC).
Submission ID Group name Mean AUC Min AUC Rank
68148 SenticLab.UAIC 0.924 0.885 1
68150 SenticLab.UAIC 0.922 0.860 2
68158 SenticLab.UAIC 0.899 0.862 3
67799 SenticLab.UAIC 0.892 0.830 4
67656 SenticLab.UAIC 0.887 0.821 5
67950 SDVA-UCSD 0.875 0.811 6
68152 SDVA-UCSD 0.875 0.811 7
68154 SDVA-UCSD 0.875 0.811 7
67839 SDVA-UCSD 0.874 0.809 7
67838 SDVA-UCSD 0.872 0.810 8
68098 SDVA-UCSD 0.869 0.817 9
67514 SenticLab.UAIC 0.860 0.772 10
68032 SDVA-UCSD 0.859 0.807 11
67658 SenticLab.UAIC 0.853 0.788 12
67920 SDVA-UCSD 0.832 0.779 13
63914 SenticLab.UAIC 0.825 0.766 14
65041 SenticLab.UAIC 0.793 0.703 15
68118 chejiao 0.791 0.682 16
67806 chejiao 0.789 0.684 17
68033 chejiao 0.787 0.677 18
68122 chejiao 0.787 0.682 19
67888 chejiao 0.781 0.662 20
67732 CompElecEngCU 0.767 0.733 21
67438 CompElecEngCU 0.764 0.698 22
68080 CompElecEngCU 0.759 0.714 23
67622 CompElecEngCU 0.757 0.713 24
67840 CompElecEngCU 0.757 0.727 25
67731 CompElecEngCU 0.756 0.724 26
67621 CompElecEngCU 0.755 0.707 27
60707 KDE-lab 0.753 0.698 28
60489 KDE-lab 0.747 0.699 29
61076 KDE-lab 0.742 0.681 30
60989 KDE-lab 0.738 0.668 31
67921 SDVA-UCSD 0.737 0.708 32
66630 KDE-lab 0.735 0.664 33
64346 CompElecEngCU 0.731 0.722 34
68004 chejiao 0.728 0.662 35
60491 KDE-lab 0.716 0.654 36
67947 FAST_NU_DS 0.705 0.644 37
67905 chejiao 0.697 0.677 38
66569 KDE-lab 0.692 0.635 39
68081 uaic2020 0.659 0.562 40
61469 CompElecEngCU 0.656 0.626 41
67229 KDE-lab 0.654 0.615 42
66617 KDE-lab 0.642 0.596 43
66583 KDE-lab 0.633 0.573 44
68121 uaic2020 0.626 0.513 45
67548 uaic2020 0.617 0.537 46
67573 uaic2020 0.609 0.598 47
68134 uaic2020 0.608 0.513 48
67681 JBTTM 0.601 0.432 49
68061 sztaki_dsd 0.595 0.546 50
67549 uaic2020 0.591 0.503 51
68125 FAST_NU_DS 0.567 0.458 52
68052 sztaki_dsd 0.539 0.514 53
68058 sztaki_dsd 0.529 0.470 54
68049 sztaki_dsd 0.515 0.404 55
67930 SDVA-UCSD 0.511 0.452 56
67977 uaic2020 0.505 0.402 57
68128 FAST_NU_DS 0.496 0.481 58
68055 sztaki_dsd 0.492 0.426 59
68135 uaic2020 0.491 0.382 60
60495 JBTTM 0.484 0.471 61
67902 sztaki_dsd 0.483 0.379 62
68050 sztaki_dsd 0.462 0.339 63
68059 sztaki_dsd 0.461 0.422 64
63627 CompElecEngCU 0.427 0.360 65

Citations

  • When referring to the ImageCLEFtuberculosis 2020 task general goals, general results, etc. please cite the following publication (also referred to as ImageCLEF tuberculosis task overview):
    • Serge Kozlovski, Vitali Liauchuk, Yashin Dicente Cid, Aleh Tarasau, Vassili Kovalev, Henning Müller, Overview of ImageCLEFtuberculosis 2020 - Automatic CT-based Report Generation and Tuberculosis Severity Assessment, CLEF working notes, CEUR, 2020.
    • BibTex:
      @Inproceedings{ImageCLEFTBoverview2020,
        author = {Kozlovski, Serge and Liauchuk, Vitali and Dicente Cid, Yashin and Tarasau, Aleh and Kovalev, Vassili and M\"uller, Henning},
        title = {Overview of {ImageCLEFtuberculosis} 2020 - Automatic {CT}-based Report Generation},
        booktitle = {CLEF2020 Working Notes},
        series = {{CEUR} Workshop Proceedings},
        year = {2020},
        volume = {},
        publisher = {CEUR-WS.org $<$http://ceur-ws.org$>$},
        pages = {},
        month = {September 22-25},
        address = {Thessaloniki, Greece}
        }
    • When referring to the ImageCLEF 2020 lab general goals, general results, etc. please cite the following publication which will be published by September 2020 (also referred to as ImageCLEF general overview):
      • Bogdan Ionescu, Henning Müller, Renaud Péteri, Asma Ben Abacha, Vivek Datla, Sadid A. Hasan, Dina Demner-Fushman, Serge Kozlovski, Vitali Liauchuk, Yashin Dicente Cid, Vassili Kovalev, Obioma Pelka, Christoph M. Friedrich, Alba García Seco de Herrera, Van-Tu Ninh, Tu-Khiem Le, Liting Zhou, Luca Piras, Michael Riegler, På l. Halvorsen, Minh-Triet Tran, Mathias Lux, Cathal Gurrin, Duc-Tien Dang-Nguyen, Jon Chamberlain, Adrian Clark, Antonio Campello, Dimitri Fichou, Raul Berari, Paul Brie, Mihai Dogariu, Liviu Daniel Ștefan, Mihai Gabriel Constantin, ImageCLEF 2020: Multimedia Retrieval in
        Lifelogging, Medical, Nature, and Internet Applications In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020), Thessaloniki, Greece, LNCS Lecture Notes in Computer Science, Springer (September 22-25 2020)
      • BibTex:
        @inproceedings{ImageCLEF20,
        author = {Bogdan Ionescu and Henning M\"uller and Renaud P\'{e}teri
        and Asma Ben Abacha and Vivek Datla and Sadid A. Hasan and Dina
        Demner-Fushman and Serge Kozlovski and Vitali Liauchuk and Yashin
        Dicente Cid and Vassili Kovalev and Obioma Pelka and Christoph M.
        Friedrich and Alba Garc\'{\i}a Seco de Herrera and Van-Tu Ninh and
        Tu-Khiem Le and Liting Zhou and Luca Piras and Michael Riegler and
        P\aa l Halvorsen and Minh-Triet Tran and Mathias Lux and Cathal Gurrin
        and Duc-Tien Dang-Nguyen and Jon Chamberlain and Adrian Clark and
        Antonio Campello and Dimitri Fichou and Raul Berari and Paul Brie and
        Mihai Dogariu and Liviu Daniel \c{S}tefan and Mihai Gabriel
        Constantin},
        title = {{Overview of the ImageCLEF 2020}: Multimedia Retrieval in
        Lifelogging, Medical, Nature, and Internet Applications},
        booktitle = {Experimental IR Meets Multilinguality, Multimodality, and
        Interaction},
        series = {Proceedings of the 11th International Conference of the CLEF
        Association (CLEF 2020)},
        year = {2020},
        volume = {12260},
        publisher = {{LNCS} Lecture Notes in Computer Science, Springer},
        pages = {},
        month = {September 22-25},
        address = {Thessaloniki, Greece}
        }
  • When using any of the provided mask of the lungs, please cite the corresponding publication:
      for segmentation #1 (fully automatic multistage):
      • Yashin Dicente Cid, Oscar A. Jiménez-del-Toro, Adrien Depeursinge, and Henning Müller. Efficient and fully automatic segmentation of the lungs in CT volumes. In: Goksel, O., et al. (eds.) Proceedings of the VISCERAL Challenge at ISBI. No. 1390 in CEUR Workshop Proceedings (Apr 2015)
      • BibTex:

        @inproceedings{DJD2015,

          Title = {Efficient and fully automatic segmentation of the lungs in CT volumes},
          Booktitle = {Proceedings of the {VISCERAL} Anatomy Grand Challenge at the 2015 {IEEE ISBI}},
          Author = {Dicente Cid, Yashin and Jim{\'{e}}nez del Toro, Oscar Alfonso and Depeursinge, Adrien and M{\"{u}}ller, Henning},
          Editor = {Goksel, Orcun and Jim{\'{e}}nez del Toro, Oscar Alfonso and Foncubierta-Rodr{\'{\i}}guez, Antonio and M{\"{u}}ller, Henning},
          Keywords = {CAD, lung segmentation, visceral-project},
          Month = may,
          Series = {CEUR Workshop Proceedings},
          Year = {2015},
          Pages = {31-35},
          Publisher = {CEUR-WS},
          Location = {New York, USA}
          }
    • for segmentation #2 (via non-rigid image registration):
      • Liauchuk, V., Kovalev, V. ImageCLEF 2017: Supervoxels and co-occurrence for tuberculosis CT image classification. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, Dublin, Ireland, CEUR-WS.org http://ceur-ws.org (September 11-14 2017) (URL: http://ceur-ws.org/Vol-1866/paper_146.pdf)
      • BibTex:

        @inproceedings{CLEF2017WN,

          Title = {ImageCLEF 2017: Supervoxels and co-occurrence for tuberculosis CT image classification},
          Author = {Liauchuk, Vitali and Kovalev, Vassili},
          Keywords = {Supervoxels, Dictionary, Co-occurrence, Image Classification},
          Booktitle = {{CLEF2017 Working Notes},
          Series = {{CEUR} Workshop Proceedings},
          Year = {2017},
          Publisher = {CEUR-WS},
          Month = {September 11-14},
          Address = {Dublin, Ireland}
          }

      Organizers

      • Serge Kozlovski <kozlovski.serge(at)gmail.com>, Institute for Informatics, Minsk, Belarus
      • Vitali Liauchuk <vitali.liauchuk(at)gmail.com>, Institute for Informatics, Minsk, Belarus
      • Yashin Dicente Cid <yashin.dicente(at)warwick.ac.uk>, University of Warwick, Coventry, England, UK
      • Vassili Kovalev <vassili.kovalev(at)gmail.com>, Institute for Informatics, Minsk, Belarus
      • Henning Müller <henning.mueller(at)hevs.ch>, University of Applied Sciences Western Switzerland, Sierre, Switzerland