Participants are required to submit ranked lists of (up to) the top 1000 images ranked in descending order of similarity (i.e. the highest nearer the top of the list). It is important that the submitted runs follow these guidelines in order to be evaluated properly. Please note that runs which do not follow these guidelines will not be evaluated. There is a limit of five runs per participants in this year's ImageCLEF Photo Retrieval Task. If more runs are being submitted, the organisers unfortunately will have to choose five of them and only these runs are being evaluated.

Please submit your runs using DIRECT system. You will receive a username and password soon from DIRECT administrators. Please use them to log in and click "Submit Experiment". Upload each of the run in zip file, then a form will show up and you should fill the information needed. You could use the name of the runs in the identifier and description, and please choose the topic fields which you used in your experiment. Click "Validate" and you will be asked to confirm your submission, click "Submit" and your run should be submitted to the system. You can always delete and add new runs if necessary.

If you have any questions regarding the submission, please contact the organisers.

We would require participants to specify the methods used in detail. There are two main dimensions which must be included and these are described as follows.

There are four main tags which may be used in the experiments: title (T), cluster title (CT), cluster description (CD) and image (I). These tags are shown in the query example below:

Please specify which tags used by your runs in retrieving the images. If you use more than one tags, please use the correct order of the appearance in the topic (T, CT, CD and I) and separate them by using "-". Since there are two types of topics, and the second one does not have tags other than the title and image, please name your run based on the tags you used in the topics part one. Therefore, if you use the title and image in topics part one, we will assume that you use the same tags in topics part two.

A list of the code examples are shown in the table below. You may use other permutations which are not listed in the table.

Apart from the tags, you should also specify the modality of your runs. This data describes the use of visual or text features in your submission.

Having specified both of the dimensions above, the submitted runs should follow these guidelines:
     [Unique name from your group]_[Query Type]_[Modality]

For example, the Sheffield group who used all tags and text feature only would have the name:
     Sheffield1_T-CT-CD-I_TXT

Submission for the ImageCLEF task follows the standard TREC format and the relevant section from the general TREC guidelines have been reproduced almost verbatim below. Note in ImageCLEF, the document reference refers to both the image and caption.

The format to use when submitting results is as follows, using a *space* as the delimiter between columns. The width of the columns in the format is not important, but it is important to include all columns and have at least one space between the columns.

• The first column is the topic number -- these will be numbered 1-60 in 2008.
• The second column is the query number within that topic and these allow for variation between the translations. This field is not used in ImageCLEF 2009 and should be set to 1.
• The third column is the official document number of the retrieved document. This will take the form of: directory/filename, e.g. "belga15/15001234" where the filename has the extension removed.
• The fourth column is rank position starting from 0.
• The fifth column shows the score (integer or floating point) that generated the ranking. This score MUST be in descending (non-increasing) order and is important to include so that we can handle tied scores (for a given run) in a uniform fashion (the evaluation routines rank documents from these scores, not from your ranks).
• The sixth column is called the "run tag" and should be a unique identifier for your group AND for the method used. That is, each run should have a different tag that identifies the group and the method that produced the run.