- ImageCLEF 2024
- LifeCLEF 2024
- ImageCLEF 2023
- LifeCLEF 2023
- ImageCLEF 2022
- LifeCLEF2022
- ImageCLEF 2021
- LifeCLEF 2021
- ImageCLEF 2020
- LifeCLEF 2020
- ImageCLEF 2019
- LifeCLEF 2019
- ImageCLEF 2018
- LifeCLEF 2018
- ImageCLEF 2017
- LifeCLEF2017
- ImageCLEF 2016
- LifeCLEF 2016
- ImageCLEF 2015
- LifeCLEF 2015
- ImageCLEF 2014
- LifeCLEF 2014
- ImageCLEF 2013
- ImageCLEF 2012
- ImageCLEF 2011
- ImageCLEF 2010
- ImageCLEF 2009
- ImageCLEF 2008
- ImageCLEF 2007
- ImageCLEF 2006
- ImageCLEF 2005
- ImageCLEF 2004
- ImageCLEF 2003
- Publications
- Old resources
You are here
ImageCLEF 2009 Photo Retrieval Task
~~~~~~~~~ Please visit our ImageCLEFPhoto twitter page to get updates of what the organisers are working on at the moment ~~~~~~~~
UPDATE (23 September 2009)
We have released the relevance judgment and the cluster judgment information.
Introduction
The photo retrieval task of ImageCLEF 2009 is intended to provide a further study in the importance of diversity in image search result, which has started being studied since ImageCLEF last year. This year’s task elevates the research scope by using a new data set containing half a million images.
Diversity is highly needed when users put poorly specified or ambiguous queries. Since search engine knows nothing about users’ preference of relevant images, search engine could increase probability of finding relevant images by presenting different images which cover all possible representations within the query. Duplicate images should be reduced, as they are not considered useful by users. Diversity is also needed when users put same queries, but require different sets of results. These reasons are evidences of the importance of diversity, and image search engine should be able to tackle these problems by presenting a diverse result. This task requires participants to analyse the best way to improve diversity while still presenting images with high relevance.
The Task – Promote Diversity
Participants will run each provided topic on their image search system and produce a ranking that in the top 20, holds as many relevant images that are representative of the different sub-topics within the results. The definition of what consitutes diversity will vary across the topics, but there will be a clear indication in the topic, which is defined by the tag "cluster", indicating what the clustering criteria the evaluators will use.
For each topic in the ImageCLEFPhoto set, relevant images will be manually clustered into sub-topics and relevance judgements will be augmented to indicate which cluster an image belongs to. Relevance assessors will be instructed to look for simple clusters based on the form of a topic. For example if a topic asks for images of beaches in Brazil, clusters will be formed based on location; if a topic asks for photos of animals, clusters will be formed based on animal type.
Participating groups will return to us, for each topic, a ranked list of images IDs. We will determine which images are relevant and count how many clusters are represented in the ranking. We do not require you to identify or label clusters in the ranked list how you choose to do the clustering is an internal matter for you.
Evaluation will be based on two measures: precision at 20 and instance recall at rank 20 (also called S-recall), which calculates the percentage of different clusters represented in the top 20. It will be important to maximise both measures: simply getting lots of relevant images from one cluster or filling the ranking with diverse, but non-relevant images, will result in a poor overall effectiveness score.
Note, that it is quite possible to submit runs from a "standard" non-clustering image search system, though we would expect clustering systems to out-perform the standard systems in producing a diverse ranked list in the top 20.
Query Topics
ImageCLEFPhoto 2009 will use new topics developed based on the analysis of Belga query logs in 2008. Therefore, these topics highly represent the real life situation of diversity need in information retrieval. There will be around 50 topics in this year's ImageCLEFPhoto task's evaluation. All topics are in English language and they will have a variable number of clusters, ranging from 2 to 10.
The format of this year's queries would be different to the ones in the previous year, as we decided to eliminate the cluster tag. Instead, we will give cluster titles which represent what the relevant images would be about. Each of the cluster title would have an example image. An example of this year's topics is shown below:
|
The topic above contains five main clusters (shown in blue) and one 'other' cluster (shown in red). This 'other' cluster is created because there are other needs of diversity in the search results, but the numbers of these diverse queries were not big enough to be shown as different clusters. Due to this situation, we created the last cluster to encourage participants to find other images which are relevant to the initial query, but are not included in the main clusters.
Topics Download
There are fifty topics available for this year's photo retrieval task. These topics were constructed based on Belga's query logs in 2008. The coordinators are now developing a document containing how these queries were constructed and this information will be released very soon. You can download the topics by using the links below:
- Topics - part 1
All topics in this list adopt the format shown in the previous section, where each query title has several cluster titles, cluster descriptions and their relevant images. - Topics - part 2
In the part two of the topics, only the query title and image examples are given out to the participants. Three relevant images are given out for each query, but this number does not represent the number of clusters in that query. Participants are encouraged to decide on how broad the results should be for each of these topics.
IMPORTANT: Please note that the image examples given in these lists must not be included in the search results.
Data Collections
This year’s ImageCLEFPhoto task will use a new collection, which contains 498,920 images from Belga News Agency, which is an image search engine for news photographs. Each photograph will be up to a maximum of 512 pixels in either width or height, accompanied by a caption composed of English text up to a few sentences in length. Different to the data last year, captions are provided without a specific format to increase the challenges to participants. Caption might contain date and place where image was captured.
Invalid Images
Due to an unknown format, we unfortunately had to eliminate 881 images from the collection. Please update your collection by deleting these invalid images. Thanks to Gao Sheng for the list.
An example of image and its annotations are shown below:
 BELGIUM DENDERMONDE COMMEMORATION KNIFE ATTACK |
| 20090126 - DENDERMONDE, BELGIUM: Lots of people pictured during a commemoration for the victims of the knife attack in Sint-Gilles, Dendermonde, Belgium, on Monday 26 January 2009. Last friday 20-Year old Kim De Gelder killed three people, one adult and two childs, in a knife attack at the chidlren's day care center "Fabeltjesland" in Dendermonde. BELGA PHOTO BENOIT DOPPAGNE |
Submission Format and Guidelines
The submission format and guidelines are available here.
Important Dates
|
| : | Registration opens for Photo Retrieval Task |
|
| : | Release of
data collection |
| Release of topic examples has been cancelled. Participants, however, can see an example of the new topics in the "Query Topics" section in this web page. | ||
|
| : | Topic release |
|
| : | Submission of runs |
|
| : | Release of Results |
|
| : | Submission of Working Notes Papers |
|
| : | CLEF Workshop in Corfu, Greece |
Organisers of ImageCLEF Photo
Primary Contact : Monica Lestari Paramita, Department of Information Studies, University of Sheffield (m.paramita@shef.ac.uk)
Mark Sanderson, Department of Information Studies, University of Sheffield (m.sanderson@shef.ac.uk)
Paul Clough, Department of Information Studies, University of Sheffield (p.d.clough@shef.ac.uk)
Mailing List
We have set up a mailing list: imageclef@sheffield.ac.uk for participants. Please contact Paul Clough to be added to the list.