You are here

Home › Wikipedia Retrieval › Revisions

Revision of Wikipedia Retrieval from Mon, 03/15/2010 - 11:44

image header
Introduction
ImageCLEF's Wikipedia Retrieval task provides a testbed for the system-oriented evaluation of visual information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs.

In 2010, ImageCLEF's Wikipedia Retrieval will use a new collection of approximately 250,000 Wikipedia images that cover diverse topics of interest. These images are associated with unstructured and noisy textual annotations in English, French, and German.

This is an ad-hoc image retrieval task; the evaluation scenario is thereby similar to the classic TREC ad-hoc retrieval task and the ImageCLEF photo retrieval task: simulation of the situation in which a system knows the set of documents to be searched, but cannot anticipate the particular topic that will be investigated (i.e. topics are not known to the system in advance). The goal of the simulation is: given a textual query (and/or sample images) describing a user's (multimedia) information need, find as many relevant images as possible from the Wikipedia image collection.

Any method can be used to retrieve relevant documents. We encourage the use of both concept-based and content-based retrieval methods and, in particular, multi modal and - new this year - multi lingual approaches that investigate the combination of evidence from different modalities and language resources.

More information will be provided soon.

Wiki MM Collection
The Wiki MM collection consists of around 250,000 images and associated user-supplied annotations. The collection was built to cover similar topics in English, German and French. Topical similarity was obtained by selecting only Wikipedia articles which have versions in all three languages and are illustrated with at least one image in each version. 44664 such articles were extracted from September 2009 Wikipedia dumps, containing a total number of 265987 images. The collection is intended to be freely distributed and we decided to remove all images with unclear copyright status. After this operation, the remaining number of images in the collection is 251866, with the following language distribution:

-English only: 77,365
-German only: 68,831
-French only: 42,751
-English and German: 14,729
-English and French: 16,887
-German and French: 12,128
-English, German and French: 19,175

The main difference between the Wiki MM collection and the INEX MM collection (Westerveld and van Zwol, 2007) used in previous WikipediaMM tasks is that the multilingual aspect was reinforced and both mono- and cross-lingual evaluations can be carried out. Another difference is that this year, participants will receive both the image annotation provided in the previous year and links to the article(s) which contain the image.

(Westerveld and van Zwol, 2007) T. Westerveld and R. van Zwol. The INEX 2006 Multimedia Track. In N. Fuhr, M. Lalmas, and A. Trotman, editors, Advances in XML Information Retrieval:Fifth International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2006, Lecture Notes in Computer Science/Lecture Notes in Artificial Intelligence (LNCS/LNAI). Springer-Verlag, 2007.
Evaluation Objectives
The characteristics of the new Wiki MM collection allow for the investigation of the following objectives:

  • how well do the retrieval approaches cope with larger scale image collections?
  • how well do the retrieval approaches cope with noisy and unstructured textual annotations?
  • how well do the content-based retrieval approaches cope with images that cover diverse topics and are of varying quality?
  • how well can systems exploit and combine different modalities given a user's multimedia information need? Can they outperform mono modal approaches like query-by-text, query-by-concept or query-by-image?
  • how well can systems exploit the multiple language resources? Can they outperform mono-lingual approaches that use for example only the English text annotations?
In the context of INEX MM 2006-2007, mainly text-based retrieval approaches have been examined. Here, we hope to attract more visually-oriented approaches and most importantly, multi modal and multi lingual approaches that investigate the combination of evidence from different modalities and languages. The results of WikipediaMM at ImageCLEF 2008/2009 showed that multimedia retrieval approaches outperformed for certain topics the text-based approaches, but globally the retrieval based on text remains unbeaten. The retrieval of multimedia documents will stay in the focus of attention for 2010. This year, a second focus will be the effectiveness of multi lingual approaches for multimedia document retrieval.
Schedule
The schedule can be found here:
  • 15.2.2010: registration opens for all ImageCLEF tasks
  • 25.3.2010: data release (images + metadata)
  • 19.4.2010: topic release
  • 15.5.2010: registration closes for all ImageCLEF tasks
  • 11.6.2010: submission of runs
  • 16.7.2010: release of results
  • 15.8.2010: submission of working notes papers
  • 20.09.2010-23.09.2010: CLEF 2010 Conference, Padova, Italy
Organisers

ImageCLEF - Cross-language image retrieval evaluations