Revision of ImageCLEF 2008: Visual Concept Detection Task from Thu, 05/08/2008 - 11:20

In 2008, ImageCLEF will offer a Visual Concept Detection Task continuing the Object Annotation Task of ImageCLEF 2006 and the Object Retrieval Task of ImageCLEF 2007. In contrast to its predecessors, the Visual Concept Detection Task of ImageCLEF 2008 will be strongly interacting with the ImageCLEF 2008 Photographic Retrieval Task.

Outline

The Visual Concept Detection Task has the objective to identify visual concepts that would help in solving the photographic retrieval task in ImageCLEF 2008.

Therefore we will publish a training database of approximately 1,800 images which are classified according to the concept hierarchy described in the following section along with their classification. Only these data may be used to train concept detection/annotation techniques.

At a later stage, we will publish a test database of 1,000 images. For each of these images participating groups are required to determine the presence/absence of the concepts.

At a later stage, the usefullness of the detected topics for the retrieval procedure will be evaluated. Groups who are interested in this will be required to detect the defined concepts in the 20,000 images which are used in the photo retrieval task. If you are interested in this, please contact the organizers as we are currently not yet sure how this can be evaluated.

Hierarchy

The images are labelled according to this class hierarchy:

Data

The visual concept detection task will work on a subset of the extended IAPR TC-12 database which is also used for the Photo Retrieval Task of ImageCLEF 2008.

You need to be registered to access these data.

Username and password to access these data is provided upon registration for ImageCLEF 2008.

Training and Test data is now available.

• Training Data
• Training Data Annotation
  

  May 8, 2008, NOTE: Gao Sheng from Singapore found a mistake in the training annotation:
  Image 27-27858.jpg should not not annotated to contain the person-concept.
  Feel free to change this for your training.

• Test Data

Format of the annotation

27-27700.jpg 0 1 0 0 1 0 0 1 1 1 0 0 1 0 0 0 0
27-27704.jpg 0 1 0 1 0 0 0 0 0 1 0 0 1 0 1 0 0
27-27705.jpg 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0
27-27706.jpg 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27-27709.jpg 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 1 0
27-27712.jpg 0 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 0
...

The first column gives the filename of the image and the succeeding columns denote presence/absence of the concepts in this order: 0 indoor 1 outdoor 2 person 3 day 4 night 5 water 6 road or pathway 7 vegetation 8 tree 9 mountains 10 beach 11 buildings 12 sky 13 sunny 14 partly cloudy 15 overcast 16 animal

Evaluation

Please note that it is not permitted to use any additional data for training and setup of the systems. If you need test data for system tuning, you need to split the available training data into a training and evaluation set. We will use equal error rate to evaluate the performance of the individual runs.

Eval Script

We provide a python program for the evaluation of runs on the training data to allow for system tuning. Click here to download. Put it into the same directory as the training data annotation, and the list of classes. Running the program on a file will output the Equal Error Rate for each class. The program requires valid submission files for an arbitrary subset of the training data. Adding a -roc option will additionally output a set of files to plot with gnuplot.

Submission

Submission format is equal to the annotation format of the training data, except that you are expected to give some confidence scores for each concept to be present or absent. That is, you have to submit a file containing the same number of columns, but each number can be an arbitrary floating point number, where higher numbers denote higher security regarding the presence of a particular concept. The submission site is available now

Annotation for the IAPR database

Additionally we encourage participants to provide us with annotations for all images in the IAPR TC-12 database which we will distribute among the participants of the retrieval task to study the usefulness of visual concept detections for the retrieval of images. IAPR database is available here For this we do not put a strict deadline, but will create a repository of submitted annotations. We encourage to submit results as fast as possible but no later than May 15 to allow for proper use in the retrieval task. Submitted runs are being made available here.

Schedule

Submission of Results: May 15, 0:00h CET, no extension

Organisers

Thomas Deselaers, RWTH Aachen University, Aachen, Germany Allan Hanbury, Vienna University of Technology Institute of Computer Aided Automation, Pattern Recognition and Image Processing Group, Vienna, Austria