Visual Concept Detection and Annotation Task 2009: Results

Outline

The results for the Visual Concept Detection and Annotation Task are now available. We had submissions of 19 groups with altogether 73 runs.
We used two measures to determine the quality of the annotations. One for the evaluation per concept and the other for the evaluation per photo. The first one is the Equal Error Rate (EER) and the Area under Curve (AUC). The second measure is the proposed hierarchical measure that considers the relations between concepts and the agreement of annotators on concepts.

On the following sites you can find the results:
* Results of the Equal Error Rate
* Results of the Hierarchical Measure

Regrettably we cannot provide all testset annotations, as we cannot perform the annotation effort every year and would like to have some annotations for future tasks. But we are happy to provide you the annotations of 3000 photos of the testset for your own evaluations at our ftp. Here you can also find the results for the evaluation on the small testset. The results differ a little bit from the results on the complete testset.