You are here

Home › Frequently asked questions › Revisions

Revision of Frequently asked questions from Mon, 10/15/2007 - 17:24

How can I register for ImageCLEF?

Please have a look at the CLEF web pages http://www.clef-campaign.org/) where the exact procedure is described. There is no separate registration for ImageCLEF and all the administrative part is handled centrally for CLEF.

Can I obtain the databases of ImageCLEF without registering?


In general, it is not possible to obtain the databases without registering as the copyright owners often ask us to do so. Still, some datasets were made available publicly already without the need to register. It does not depend on us but rather on the copyright owners, although we prefer of course a high participation at the workshop and many submissions.

Datasets that can be obtained independently:

What are my obligations when I register?

The main obligations are described in the forms that you sign. Most of it concerns the handling and use terms for the datasets. The datasets are not your property but you are allowed to use the in the ImageCLEF context and partly also for your research afterwards.

What happens if I am not able to submit results after registration?

There is no penalty for registering without submitting, so there is nothing to worry about. Still, you can only get the most from your participation when you are registered and can compare your research with other people.

Can I register for ImageCLEF at any time of the year?

Similar to TREC and other benchmarking events, CLEF has a yearly cycle of making available registration, data, tasks, and then evaluating the results. In principal you can register between the moments when registration opens in January up until the submission deadline for results some time in June. You are of course better off to register as early as possible as this gives you the possibility to better work on your submission.

Can I participate at the workshop without having submitted results?

Yes, everybody is free to participate at the CLEF workshop or more particular the ImageCLEF sessions. Registration for CLEF is currently handled by the ECDL conference. Also people who were not registered to submissions or groups that did not submit anything can register.

How many different tasks are there at ImageCLEF?

The number of tasks may change every year. It depends in general on the interest of research groups on a variety of tasks proposed the preceding year, on the availability of datasets and on the motivation of someone to run a separate task (which is a lot of work!!). In 2003, there was one task, two in 2004, three in 2005, and 4 in 2006 and 2007. Detailed information on the number of tasks and the tasks themselves will be published timely on http://www.imageclef.org.

Can I reuse the image databases for other research after the benchmark is over?

This depends mainly on the copyright constraints of the databases. Often it is indeed possible to reuse the datasets after the competition, when the sources are correctly cited. In case this is not clear, please contact us and we can tell you the exact terms of the database that you are planning to use.

How is the selection of the oral presentations at the workshop done?

We select the small number of oral presentations at the workshop based on several criteria. We would like to have some of the better submissions give the possibility to present their techniques. Still, we would also like to give the possibility to techniques that seem interesting and innovative to be presented. The organization committee takes the final decisions. Still, all participants at the workshop have a possibility to present their work at a poster.

How important is it to win ImageCLEF?

As is often stated in other competitions such as TREC, the main goal is not to win the competition but compare techniques based on the same data, so everyone can learn from the results. Everyone who has been to a workshop can tell that the discussions at the posters are always very vivid and many approaches that work or do not are discussed. It is a chance for everyone to learn from each other.

What is the difference between automatic, manual and feedback submissions?

Automatic submission means that your system is not taking into account any information that is based on the used database nor is a user interacting with the system to improve results. When you modify, for example weight of feature based on learning from the collection it is already a manual run. Query modifications for the retrieval are also manual submissions. If based on first results a real user supplies feedback to the system to refine the queries, then it should be labeled a feedback run.

How can I make my collection available via ImageCLEF?

Please contact us organizers if you have (preferably large) collections that could be made available for CLEF. We are always looking for collection with a copyright that permits us to distribute it among participants. Of course the content of the collection needs to fit into our strategy for it to be useful. Some databases might not be used directly but a year or two later.

How are the articles of the ImageCLEF workshop published?

The publication modalities for ImageCLEF can change from one year to another. At the conference, working notes are made available to the participants, sometimes in paper, or on CD. In the future, these might be distributed via the Internet as CD images to reduce workload. Then, the proceedings are usually printed in the Springer Lecture Notes in Computer Science after a review process of the papers.

How are the tasks for the various ImageCLEF tracks developed?

The tasks of ImageCLEF have been developed over a fairly long time and the method for creating them is often different from one year to another. It is adapted to the propositions of the participants to be really helpful in research. In general we try to create a realistic user model and then create tasks for this user. Often surveys among professionals (medical, library, …) are use or log files of systems such as demonstrations on the web. Propositions for creating realistic tasks are also always welcome.

Can I propose my own tasks for a future version of ImageCLEF?

Yes, you can of course propose new tasks and/or databases or other resources to us, particularly if you are willing to spend some work on this and help organizing it (which really is a lot of work). We can never guarantee to accept your proposition bit the organization committee discusses the possible new tasks every year. It is important to not run the same thing every year but to evolve with research interests in the field.

What is the pre-CLEF workshop about?

As CLEF is in general rather a text-based workshop we wanted to create an additional motivation for visual retrieval researchers. This workshop is thus mainly addressing issues around visual information retrieval evaluation but includes parts on text as well. As most visual retrieval researchers are not interested in ECDL, the workshop takes place at the same time as ECDL. This makes CLEF a four-day event, and allows us to better collect ideas for new tasks, even before the actual CLEF workshop.

What are the main objectives of ImageCLEF?

The main objective of ImageCLEF is to advance the field of image retrieval and offer evaluation in various fields of image information retrieval. We particularly identified the mixed use of text and visual features as important because this is relatively new as a research domain and most research groups work either on text or on images but only a few groups work on the two. By making available visual and textual results we give these groups a tool to obtain the information that they do not have themselves.

Where are the differences between ImageCLEF and other initiatives such as TREC, TRECVID, ImageEval, or the Benchathlon?

There are quite a few differences between all these evaluation campaigns but also a few similarities. TREC is surely the role model for CLEF and ImageCLEF with respect to organization. Still, TREC concentrates on text where as ImageCLEF deals with still images plus annotations for these images. Most annotations are also in several languages. The Benchathlon was more a forum to create evaluation resources but no real evaluation ever took place, although many papers were published in this context and also a database created. TRECVID is rather using videos instead of images as the target for retrieval but otherwise there are a few similarities as single shots of the videos can be regarded as images for retrieval. The spoken language can also be translated into text and is available to participants, so multimodal retrieval and since recently multilingual retrieval are covered. We work together with the TRECVID organizers to avoid too much overhead. We are also having other user models such as medical practitioners or home photo users, whereas TRECVID rather has a BBC journalist as user model. ImageEval has a variety of interesting tasks but is currently fairly concentrated on the French research domain, although it is accessible to other researchers as well. Many tasks are rather trying to test invariance of systems or stability with respect to image changes rather than real-world tasks based on identified needs of users of retrieval systems. Again, we are trying to stay in contact with the organizers to coordinate efforts.

How is ImageCLEF financed?

Financing is always a difficult question when it comes to benchmarks and evaluation campaigns. Most of us work without any financial aid, mainly because ImageCLEF is very interesting to work on. For the general CLEF organization the DELOS Network of Excellence finances part of the management activities and also of the ground truthing. For ImageCLEFmed, there is a financing from the America NSF to create cooperations between American and EU projects. This helps us to finance part of the organization, travel and particularly the ground truthing, which is very time consuming. The acceptance of the Multimatch EU project might also help CLEF a bit as several partners participate in Multimatch and get resources that are consecrated for retrieval issues.

ImageCLEF - Cross-language image retrieval evaluations