Revision of ImageCLEF 2008: WikipediaMM Task from Fri, 02/29/2008 - 10:33

In 2008, ImageCLEF wikipediaMM will use the image collection created and employed by the INEX Multimedia (MM) Track (2006-2007). This (INEX MM) wikipedia image collection contains approximately 170,000 images that cover diverse topics of interest. These images are associated with unstructured and noisy textual annotations in English. In addition, the classification scores for the 101 MediaMill concepts are provided for each of these images.

This is an ad-hoc image retrieval task; the evaluation scenario is thereby similar to the classic TREC ad-hoc retrieval task and the ImageCLEFphoto task: simulation of the situation in which a system knows the set of documents to be searched, but cannot anticipate the particular topic that will be investigated (i.e. topics are not known to the system in advance). The goal of the simulation is: given a textual query (and/or sample images and/or concepts) describing a user's (multimedia) information need, find as many relevant images as possible from the (INEX MM) wikipedia image collection (with the query language either being identical or different from that used to describe the images).

Any method can be used to retrieve relevant documents. We encourage the use of both concept-based and content-based retrieval methods and, in particular, multimodal approaches that investigate the combination of evidence from different modalities.

IMPORTANT CONDITION

The wikipediaMM task adopts the user model followed in INEX, whereby the participants in the various tracks create the topics and perform the relevance assessments themselves.

Therefore, participation in ImageCLEF's wikipediaMM task requires that each participating group:

• creates a minimum of 2 topics
• performs the relevance assessments on the created topics

Our experience on the INEX MM track indicates that the creation of topics does not require much effort, whereas the assessments usualy take around 2 working days per topic. This procedure is also reflected in the schedule of the task (see below).

Further information about the image collection can be found in:

T. Westerveld and R. van Zwol. The INEX 2006 Multimedia Track. In N. Fuhr, M. Lalmas, and A. Trotman, editors, Advances in XML Information Retrieval:Fifth International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2006, Lecture Notes in Computer Science/Lecture Notes in Artificial Intelligence (LNCS/LNAI). Springer-Verlag, 2007.

Additional sources of information are also provided to help participants in the retrieval tasks. These resources are:

• Image classification scores:  For each image, the classification scores for the 101 different MediaMill concepts are provided by University of Amsterdam (UvA). The UvA classifier is trained on manually annotated TRECVID video data and the concepts are selected for the broadcast news domain.
  
  
  More details can be found in:

  C. G. M. Snoek, M. Worring, J. C. van Gemert, J.-M. Geusebroek, and A. W. M. Smeulders. The challenge problem for automated detection of 101 semantic concepts in multimedia. In MULTIMEDIA ’06: Proceedings of the 14th annual ACM international conference on Multimedia, pages 421–430, New York, NY, USA, 2006. ACM Press.
  
  
  

• Image features: For each image, the set of the 120D feature vectors that has been used to derive the above image classification scores is available. Participants can use these feature vectors to custom-build a CBIR system, without having to pre-process the image collection.
  
  
  More details can be found in:

  J. C. v. Gemert, J.-M. Geusebroek, C. J. Veenman, C. G. M. Snoek, and A. W. M. Smeulders. Robust scene categorization by learning image statistics in context. In CVPRW ’06: Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop, page 105, Washington, DC, USA, 2006. IEEE Computer Society.
  
  
  

• how well do the retrieval approaches cope with larger scale image collections?
• how well do the retrieval approaches cope with noisy and unstructured textual annotations?
• how well do the content-based retrieval approaches cope with images of varying quality?
• how well can systems exploit and combine different modalities given a user's multimedia information need? Can they outperform uni-modal approaches like query-by-text, query-by-concept or query-by-image?

In the context of INEX MM 2006-2007, mainly text-based retrieval approaches have been examined. Here, we hope to attract more visually-oriented approaches and most importantly, multimodal approaches that investigate the combination of evidence from different modalities.

• 20.2.2008: registration opens for all CLEF tasks
• 15.3.2008: data release
• 17.3.2008: instructions and formatting criteria for candidate topics/queries provided to participants
• 4.4.2008: submission deadline for candidate topics
• 15.4.2008: topic release
• 15.5.2008: submission of runs
• 22.5.2008: distribution of merged results to participants for relevance assessments
• 1.7.2008: submission deadline for relevance assessments
• 15.7.2008: release of results
• 15.8.1008: submission of working notes papers
• 17.-19.9.2008: CLEF workshop in Aarhus, Denmark

• Theodora Tsikrika, Database Architectures and Information Access Group, CWI, Amsterdam, The Netherlands, Theodora.Tsikrika@cwi.nl
• Jana Kludas, University of Geneva, Switzerland