You are here

Home › ImageCLEF Large Scale Visual Concept Detection and Annotation Task › Revisions

Revision of ImageCLEF Large Scale Visual Concept Detection and Annotation Task from Thu, 03/26/2009 - 11:55

Outline

Following the tradition from 2008, ImageCLEF will offer a visual concept detection and annotation task in 2009 again.
This year, the focus lies on the extension of the task concerning the amount of data available and the amount of concepts to be annotated. In 2008 there were about 1800 images for training and 1000 images for testing available. This year the training and test set will consist of several thousand images respectively of the MIR Flickr 25.000 image dataset.

All images have multiple annotations. Most annotations refer to holistic visual concepts and are annotated at an image-based level.
Categories for the visual concepts are for example:

  • Abstract Categories (Landscape, Family&Friends, Partylife …)
  • Seasons
  • Time of Day
  • Persons (no, single, big groups)
  • Quality (blurred, underexposed …)
  • Representation (portrait, macro image, canvas …)

Altogether we provide the annotations for 53 concepts. While most of the holistic concepts can be objectively determined (e.g., the presence or absence of objects) there are also some concepts that are influenced through the subjective impression of the annotators. A description of each concept was provided to the annotators to reduce the subjective impact, followed by a validation step after annotation.

The visual concepts are organized in a small ontology. Participants may use the hierarchical order of the concepts and the relations between concepts for solving the annotation task.

The Task

ANNOTATE THE IMAGES WITH All DEPICTED VISUAL CONCEPTS.

This task poses two main challenges:
1) Can image classifiers scale to the large amount of concepts and data?
2) Can an ontology (hierarchy and relations) help in large scale annotations?

Training Data

The training data is now available HERE. Please use the login data you got after registration. The training set consists of 5000 images with 53 visual concepts annotated. The annotations are provided in two formats: as rdf files and as plain txt files.

Please note that it is not permitted to use any additional data for training and setup of the systems. If you need test data for system tuning, you need to split the available training data into a training and evaluation set.

Evaluation Measures

The evaluation of the Visual Concept Detection and Annotation Task will consider several evaluation measures.

1) Evaluation per concept
We will use equal error rate to evaluate the performance of the individual runs for each concept. The script from Thomas from the last year can be used for evaluation. The tool is implemented in octave but works also with matlab. The annotation folder in the download area contains one file with all groundtruth annotations in the correct format to use the script.

2) Evaluation per image
A hierarchical measure that determines the annotation performance for each image, will be used as main score to judge the annotation system. It considers partial matches between system output and groundtruth and calculates misclassification costs for each missing or wrongly annotated concept per image. The score is based on structure information (distance between concepts in the hierarchy), relationships from the ontology and the agreement between annotators for a concept.
Further information about the hierarchical evaluation measure is provided soon.

3) Processing Times
We would like to assess the time that is needed to annotate each image. It should be regarded as reference value and is not further used.

How to cheat (but please don´t)

Please don´t use the annotation information that is delivered with the MIR Flickr 25.000 image dataset. We renamed all files and trust you that you don´t try to find out the original filenames.

Tentative Schedule


AttachmentSize
occurence_of_concepts2.png41.2 KB

ImageCLEF - Cross-language image retrieval evaluations