You are here

Personal Photo Retrieval 2013

Schedule


Motivation

Although obtaining large amounts of images from the web has become very easy, the actual contents of personal photo collections remain a blind spot of research. Personal photo collections contain a lot of duplicate images, and vary in quality and image content. To adress this issue, this year's retrieval task will concentrate on the retrieval from such a collection that has been sampled from real photographers with a demographic span that models a lifetime's photo collection. Besides the noisy content in a personal photo collection, an objective of this task is to find out whether the participating retrieval systems can exploit data from different search strategies, i.e., query-by-example and browsing data, in order to find both visual concepts (see Fig. 3) and photos depicting events (see Fig. 4). This discrimination is motivated by a conducted user study that was accompanying the collection of the personal photos.

asian temple 1
Figure 3. Samples of the Visual Concept "Asian Temple Interior".

rock concert event
Figure 4. Samples of the Event Class "Rock Concert".

Overview

This year's subtask will extend the pilot task of 2012 with a focus on different usage scenarios and user groups. That is, the subtask will reveal if the tested algorithms are stable in terms of retrieval quality for different user groups. This becomes possible because each image's relevance has been judged by multiple assessors on a gradual scale. As of January 2013 the ground truths for the following user groups are available (the list might be extended):

  1. Average (equivalent to the relevance assessments used in Cranfield-based experiments)
  2. CBIR/MIR experts
  3. CBIR/MIR laypersons
  4. People with an IT background
  5. People without an IT background
  6. Female users
  7. Male users

The subtask will be ad-hoc, i.e., no additional training data is released. The participants will get multiple QBE documents and/or browsing data and will have to find the best matching documents illustrating an event or depicting a visual concept.

Dataset

The data set is described in detail in a paper which is available below. It consists of 5,555 images plus rich metadata as they have found on the hard disks of the 19 contributors ranging from year of birth 1944 to 1985. Thus, one can interpret the content of the collection as a mirror of a photographer’s life span with typical changing usage behaviors, cameras, topics, and places.

Contribution per Photographer   Global Distribution of Images

Figure 4: Contribution by each Photographer (left); Global Distribution of Photographs (right)

Visual Features and Baseline System

We provide the following low-level visual features for the data set in order to improve the comparability of each submitted run. Participants are free to combine them freely or to use their own implementations.
Additionally, we will provide a baseline system for the participants that do not wish to concentrate on the implementation of feature extraction and similarity calculation. The system is implemented in C++ and can be compiled on Linux, Mac OS X, and Windows. The release of the system will be announced soon on this website.

  1. Auto Color Correlogram
  2. BIC
  3. BRIEF (5 variants)
  4. CEDD
  5. Color Histogram
  6. Color Layout (MPEG-7)
  7. Color Structure (MPEG-7)
  8. Contour Shape (MPEG-7)
  9. Dominant Color (MPEG-7)
  10. Edge Histogram (MPEG-7)
  11. FCTH
  12. Gabor
  13. Region Shape (MPEG-7)
  14. Scalable Color (MPEG-7)
  15. SURF (5 variants)
  16. SURF Hash (5 variants
  17. Tamura

Using face detection features is not encouraged in this task because some faces had to be anonymized to preserve individual privacy rights. To compensate, a manually created Excel sheet with the number of depicted persons for each image is provided and can be used as face detection input. The intervals of depicted people and their frequency in the data set can be found in Fig. 7.

Number of People Depicted on an Image
Figure 7: Number of People Depicted on Images

Building the Baseline System

The baseline system is licensed under the Apache License, Version 2.0 and subject to copyright 2010-2013 BTU Cottbus, Chair of Database and Information Systems. The full license text is included within the source code's archive.
It is known to compile with the following compilers:

  • Mac OS X Snow Leopard and Lion: latest shipped gcc suite from Apple (we have no experience with the compilers from MacPorts etc.)
  • gcc v4.4.5 (32 bit) and gcc v4.4.3 (64 bit)
  • gcc v4.5 (mingw32) and Visual Studio 9 2008 x64

In order to build the system from scratch make sure that you have the following tools and libraries installed:

  • Qt 4 (we have tested 4.7.0)
  • CMake
  • The shipped 3rd party libraries for your architecture that can be found in the lib/ subdirectory in the source code archive

Please note that we cannot provide assistance in building the system. If you follow the instructions in the readme files that come with the source, CMake will find all required libraries and tools. Also note that OpenCV is optional but will reduce the amount of available features. SQLlite is also not needed.

Recommended Build Steps

  1. Ensure that your systems meets the aforementioned requirements.
  2. Unzip the archive.
  3. Navigate to the path containing the extracted files using a terminal.
  4. Create a subdirectory called "build" at the same level as the src/ folder, e.g., using the command  .
  5. Change to this directory:  
  6. Run cmake and make it use the correct source, i.e.,  
  7. Run make to start the build:  
  8. Deploy the application using  

If you follow these steps you will find all binaries below the build/ directory in a folder called "applications". In order to run the binaries, make sure that your systems includes the build/libs/ subdirectory in the path for dynamic library/shared object look-up. You will also have to create an environment variable called "PYTHIA_PLUGIN_PATH" pointing to build/plugins/.

To obtain the baseline system, please contact the david.zellhoefer[at]tu-cottbus.de for a password.

Operating the System

There is no GUI for the system, it runs in a terminal. All tools have an online help. A short manual can be found at the bottom of this page.

Registering for the Task and Accessing the Data

To obtain the data set, please contact the david.zellhoefer[at]tu-cottbus.de for a password.

Topics

This year's subtask on personal photo retrieval consists of 74 topics. In contrast to the setup of the pilot task in 2012, the topics are no longer separated into visual concepts or events. Hence, you will have to find out which retrieval strategy will be appropriate based on the provided query data.

In order to retrieve the documents a user is looking for, the participants can rely on one QBE document and/or up to three browsed documents. For some topics, there are no QBE documents in order to model the following usage behavior: a user browsed a personal photo collection and toggled an action to show more similar images without stating an explicit preference. In contrast, whenever a QBE document is present, the user has deliberately chosen to search for more documents that resemble the QBE document. In this scenario, the browsed documents might give deeper insights into the user's current information need. The QBE documents have been assessed relevant by the assessors while some browsed document might be irrelevant or only relevant to a low degree.

A file containing all query data for each topic can be obtained via the link at the bottom of this page.

The topic file is structured as follows:

Topic ID;QBE;Browse -3;Browse -2;Browse -1
1;SL373313.jpg;SL373312.jpg;;
2;PICT5079.jpg;;;
3;090719_Schloss_Babelsberg_12.jpg;090719_Schloss_Babelsberg_15.jpg;;
4;;PICT0029.jpg;P1020728.jpg;P1020729.jpg

All fields are separated by a semicolon. The semantics of the fields from left to right are:

  1. The topic ID
  2. A QBE document (if available)
  3. The 3rd last browsed document
  4. The 2nd last browsed document
  5. The last browsed document

Each document field refers to a file from the collection.

Participants are free to rely only on the provided QBE documents or to use data derived from the browsed documents as well. Please state on submission if you have used the browsing data. Participants are free to experiment with whatever methods they wish for image retrieval, e.g., relevance feedback or the integration of other modalities such as spatial or temporal information. We ask participants to indicate which of the following applies to each of their runs. Please note that you must not use the IPTC metadata fields that are present in the image documents.

Submission Details

The participants are requested to hand in their results in trec_eval format. After the publication of the results, the ground truths will be made available to the participants in trec_eval's QREL format for later usage.
The submissions will be received through the ImageCLEF 2013 system using subtask "ImageCLEFphoto-retrieval".
The trec_eval format is as follows:

42	Q0	542	1	0.427824	YourRunID
42	Q0	560	2	0.379514	YourRunID
42	Q0	506	3	0.373361	YourRunID
42	Q0	555	4	0.367622	YourRunID
42	Q0	538	5	0.356699	YourRunID
42	Q0	527	6	0.337737	YourRunID
42	Q0	577	7	0.337599	YourRunID
42	Q0	524	8	0.33359		YourRunID
42	Q0	587	9	0.325482	YourRunID
42	Q0	174	10	0.320099	YourRunID
...

All fields are seperated by a tab (\t) and every line has the following semantics (from left to right):

  1. A topic ID which will be specified by the topics that have been released in March.
  2. This field will be ignored and we recommend to leave it as shown above.
  3. The file ID of the retrieved document out of an interval of [1,5555].
  4. The rank of the retrieved document. Please make sure that the documents are ordered ascendingly.
  5. The calculated similarity score of the retrieved document.
  6. An arbitrary identifier that will help you to recognize the used retrieval technique.

The participants will be permitted to submit up to 5 runs.

The full submission details are available at this page.

Performance Measures

In order to reflect the gradual relevance scale for the topics, the retrieval performance evaluation will be primarily based on NDCG. More information will follow shortly.


Results


Average User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3333 0,7516 0,6761 0,6258 0,5969 0,5571 NOFB IMGMETBRO
DBIS run2 0,3767 0,7694 0,7141 0,6669 0,6407 0,6082 NOFB IMGMETBRO
DBIS run3 0,3954 0,7773 0,7197 0,6798 0,6546 0,6084 GRADED IMGMETBRO
FINKI run1 0,1360 0,6813 0,5479 0,4384 0,3853 0,3109 NOFB IMGBRO
FINKI run2 0,1375 0,6891 0,5510 0,4398 0,3881 0,3133 NOFB IMGBRO
FINKI run3 0,1354 0,6878 0,5526 0,4410 0,3909 0,3158 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1118 0,6594 0,5152 0,4125 0,3725 0,3077 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1082 0,6303 0,4955 0,3899 0,3499 0,2910 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0771 0,5769 0,4141 0,3138 0,2741 0,2226 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1162 0,6627 0,5152 0,4173 0,3713 0,3126 NOFB IMG
isi 1 0,4937 0,8055 0,7621 0,7370 0,7251 0,7001 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,4946 0,8053 0,7633 0,7375 0,7264 0,7001 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5027 0,8105 0,7734 0,7425 0,7302 0,6893 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1855 0,7181 0,6069 0,5193 0,4829 0,4236 NOFB IMGBRO
isi 5 0,4937 0,8055 0,7621 0,7370 0,7251 0,7001 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5028 0,8086 0,7738 0,7425 0,7288 0,6878 GRADED IMGMETBRO
isi2 2 0,4965 0,8047 0,7633 0,7379 0,7271 0,6986 GRADED IMGMETBRO
isi2 3 0,4952 0,8057 0,7620 0,7365 0,7267 0,6984 GRADED IMGMETBRO
isi2 4 0,5034 0,1376 0,2167 0,3132 0,3716 0,5034 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0682 0,5547 0,3941 0,2954 0,2579 0,2071
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0529 0,4476 0,3107 0,2302 0,1982 0,1494
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0696 0,5606 0,3974 0,3001 0,2611 0,2104
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0700 0,5584 0,4005 0,3051 0,2676 0,2126
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0700 0,5572 0,4009 0,3050 0,2675 0,2126
VCTLab 1 0,0756 0,4206 0,3420 0,2950 0,2731 0,2386 NOFB IMGBRO
VCTLab 2 0,0783 0,4446 0,3574 0,3047 0,2754 0,2382 NOFB IMGBRO
VCTLab 3 0,0751 0,4282 0,3488 0,2943 0,2654 0,2336 NOFB IMGBRO
VCTLab 4 0,0662 0,3778 0,3128 0,2616 0,2412 0,2093 NOFB IMGBRO
VCTLab 5 0,0676 0,3816 0,3148 0,2712 0,2447 0,2080 NOFB IMGBRO
WIDE_IO WideIO 0,0584 0,4431 0,3253 0,2501 0,2192 0,1845
Mean 0,2127 0,6023 0,5067 0,4433 0,4166 0,3770        
Std. Deviation 0,1909 0,1749 0,1804 0,1906 0,1977 0,2057        

Non-expert User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3379 0,7530 0,6792 0,6281 0,5979 0,5583 NOFB IMGMETBRO
DBIS run2 0,3789 0,7632 0,7134 0,6649 0,6403 0,6088 NOFB IMGMETBRO
DBIS run3 0,3984 0,7723 0,7181 0,6786 0,6536 0,6084 GRADED IMGMETBRO
FINKI run1 0,1340 0,6757 0,5417 0,4293 0,3776 0,3051 NOFB IMGBRO
FINKI run2 0,1352 0,6836 0,5445 0,4307 0,3803 0,3069 NOFB IMGBRO
FINKI run3 0,1332 0,6845 0,5489 0,4337 0,3837 0,3096 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1113 0,6529 0,5115 0,4093 0,3689 0,3042 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1077 0,6230 0,4917 0,3866 0,3463 0,2878 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0762 0,5691 0,4088 0,3088 0,2688 0,2188 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1157 0,6549 0,5138 0,4144 0,3673 0,3092 NOFB IMG
isi 1 0,4991 0,7980 0,7634 0,7374 0,7249 0,7005 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,5001 0,7980 0,7650 0,7384 0,7262 0,7006 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5084 0,8072 0,7772 0,7432 0,7300 0,6900 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1866 0,7118 0,6070 0,5177 0,4800 0,4196 NOFB IMGBRO
isi 5 0,4991 0,7980 0,7634 0,7374 0,7249 0,7005 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5083 0,8044 0,7767 0,7426 0,7282 0,6883 GRADED IMGMETBRO
isi2 2 0,5019 0,7974 0,7653 0,7381 0,7271 0,6988 GRADED IMGMETBRO
isi2 3 0,5006 0,7982 0,7636 0,7362 0,7266 0,6986 GRADED IMGMETBRO
isi2 4 0,5089 0,8062 0,7762 0,7428 0,7288 0,6891 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0670 0,5431 0,3873 0,2882 0,2513 0,2025
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0538 0,4462 0,3098 0,2287 0,1971 0,1494
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0684 0,5501 0,3916 0,2930 0,2549 0,2064
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0686 0,5482 0,3944 0,2989 0,2617 0,2074
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0687 0,5490 0,3948 0,2988 0,2617 0,2073
VCTLab 1 0,0764 0,4254 0,3445 0,2963 0,2731 0,2380 NOFB IMGBRO
VCTLab 2 0,0792 0,4483 0,3616 0,3053 0,2758 0,2382 NOFB IMGBRO
VCTLab 3 0,0760 0,4333 0,3525 0,2955 0,2658 0,2334 NOFB IMGBRO
VCTLab 4 0,0672 0,3827 0,3150 0,2626 0,2420 0,2095 NOFB IMGBRO
VCTLab 5 0,0692 0,3872 0,3185 0,2734 0,2461 0,2097 NOFB IMGBRO
WIDE_IO WideIO 0,0592 0,4445 0,3257 0,2486 0,2172 0,1831
Mean 0,2141 0,6230 0,5265 0,4569 0,4273 0,3819
Std. Deviation 0,1935 0,1495 0,1788 0,1983 0,2075 0,2143

Expert
User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3502 0,7503 0,6827 0,6363 0,6094 0,5784 NOFB IMGMETBRO
DBIS run2 0,3888 0,7769 0,7200 0,6743 0,6498 0,6225 NOFB IMGMETBRO
DBIS run3 0,4108 0,7823 0,7314 0,6902 0,6638 0,6260 GRADED IMGMETBRO
FINKI run1 0,1387 0,6860 0,5479 0,4347 0,3818 0,3102 NOFB IMGBRO
FINKI run2 0,1402 0,6936 0,5511 0,4359 0,3845 0,3127 NOFB IMGBRO
FINKI run3 0,1378 0,6885 0,5519 0,4364 0,3859 0,3142 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1136 0,6591 0,5163 0,4116 0,3715 0,3092 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1098 0,6347 0,4974 0,3911 0,3494 0,2935 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0795 0,5808 0,4163 0,3136 0,2737 0,2256 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1188 0,6647 0,5154 0,4177 0,3718 0,3167 NOFB IMG
isi 1 0,5129 0,8142 0,7736 0,7488 0,7370 0,7156 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,5139 0,8137 0,7741 0,7483 0,7379 0,7157 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5224 0,8175 0,7833 0,7532 0,7429 0,7056 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1889 0,7246 0,6081 0,5194 0,4809 0,4249 NOFB IMGBRO
isi 5 0,5129 0,8142 0,7736 0,7488 0,7370 0,7156 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5220 0,8162 0,7842 0,7519 0,7402 0,7042 GRADED IMGMETBRO
isi2 2 0,5156 0,8142 0,7748 0,7491 0,7383 0,7144 GRADED IMGMETBRO
isi2 3 0,5143 0,8146 0,7737 0,7475 0,7376 0,7142 GRADED IMGMETBRO
isi2 4 0,5225 0,8183 0,7840 0,7528 0,7410 0,7048 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0701 0,5574 0,3985 0,2975 0,2595 0,2123
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0549 0,4397 0,3054 0,2272 0,1960 0,1501
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0714 0,5650 0,4019 0,3014 0,2623 0,2148
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0719 0,5587 0,4026 0,3075 0,2692 0,2171
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0719 0,5568 0,4032 0,3073 0,2690 0,2170
VCTLab 1 0,0784 0,4270 0,3457 0,2975 0,2739 0,2401 NOFB IMGBRO
VCTLab 2 0,0819 0,4495 0,3607 0,3065 0,2774 0,2405 NOFB IMGBRO
VCTLab 3 0,0784 0,4343 0,3515 0,2964 0,2668 0,2355 NOFB IMGBRO
VCTLab 4 0,0687 0,3744 0,3130 0,2622 0,2409 0,2125 NOFB IMGBRO
VCTLab 5 0,0707 0,3799 0,3144 0,2728 0,2458 0,2119 NOFB IMGBRO
WIDE_IO WideIO 0,0606 0,4280 0,3148 0,2437 0,2153 0,1872
Mean 0,2201 0,6306 0,5310 0,4623 0,4329 0,3902
Std. Deviation 0,1986 0,1564 0,1822 0,2018 0,2116 0,2189

Non-IT
User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3445 0,7543 0,6833 0,6345 0,6053 0,5653 NOFB IMGMETBRO
DBIS run2 0,3851 0,7639 0,7103 0,6655 0,6404 0,6142 NOFB IMGMETBRO
DBIS run3 0,4037 0,7760 0,7195 0,6832 0,6577 0,6142 GRADED IMGMETBRO
FINKI run1 0,1361 0,6766 0,5425 0,4324 0,3812 0,3076 NOFB IMGBRO
FINKI run2 0,1375 0,6847 0,5454 0,4339 0,3836 0,3099 NOFB IMGBRO
FINKI run3 0,1351 0,6853 0,5489 0,4352 0,3867 0,3123 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1116 0,6514 0,5093 0,4054 0,3670 0,3044 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1080 0,6200 0,4859 0,3816 0,3431 0,2874 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0775 0,5678 0,4085 0,3085 0,2696 0,2204 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1158 0,6531 0,5101 0,4113 0,3660 0,3094 NOFB IMG
isi 1 0,5045 0,7977 0,7606 0,7350 0,7241 0,7023 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,5055 0,7974 0,7615 0,7356 0,7256 0,7021 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5145 0,8072 0,7743 0,7406 0,7296 0,6930 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1848 0,7081 0,5990 0,5143 0,4771 0,4190 NOFB IMGBRO
isi 5 0,5045 0,7977 0,7606 0,7350 0,7241 0,7023 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5145 0,8052 0,7733 0,7402 0,7283 0,6916 GRADED IMGMETBRO
isi2 2 0,5074 0,7969 0,7612 0,7357 0,7259 0,7006 GRADED IMGMETBRO
isi2 3 0,5061 0,7979 0,7601 0,7335 0,7253 0,7004 GRADED IMGMETBRO
isi2 4 0,5151 0,8071 0,7728 0,7405 0,7289 0,6924 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0683 0,5425 0,3883 0,2901 0,2541 0,2070
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0547 0,4529 0,3145 0,2320 0,2001 0,1515
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0696 0,5490 0,3918 0,2948 0,2579 0,2101
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0699 0,5451 0,3938 0,2993 0,2640 0,2118
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0700 0,5447 0,3941 0,2992 0,2639 0,2118
VCTLab 1 0,0776 0,4289 0,3478 0,2996 0,2763 0,2409 NOFB IMGBRO
VCTLab 2 0,0806 0,4580 0,3680 0,3111 0,2802 0,2407 NOFB IMGBRO
VCTLab 3 0,0775 0,0436 0,0501 0,0594 0,0644 0,0775 NOFB IMGBRO
VCTLab 4 0,0686 0,3924 0,3219 0,2677 0,2475 0,2140 NOFB IMGBRO
VCTLab 5 0,0706 0,3895 0,3218 0,2770 0,2500 0,2129 NOFB IMGBRO
WIDE_IO WideIO 0,0604 0,4484 0,3297 0,2529 0,2218 0,1877
Mean 0,2165 0,6092 0,5147 0,4482 0,4210 0,3786
Std. Deviation 0,1956 0,1821 0,1961 0,2086 0,2156 0,2202

IT User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3336 0,7503 0,6770 0,6262 0,5970 0,5583 NOFB IMGMETBRO
DBIS run2 0,3796 0,7758 0,7195 0,6707 0,6435 0,6121 NOFB IMGMETBRO
DBIS run3 0,3976 0,7795 0,7262 0,6806 0,6536 0,6112 GRADED IMGMETBRO
FINKI run1 0,1368 0,6915 0,5592 0,4425 0,3876 0,3132 NOFB IMGBRO
FINKI run2 0,1382 0,6988 0,5622 0,4434 0,3904 0,3156 NOFB IMGBRO
FINKI run3 0,1363 0,6948 0,5613 0,4436 0,3921 0,3183 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1133 0,6658 0,5218 0,4148 0,3738 0,3103 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1092 0,6406 0,5034 0,3931 0,3506 0,2934 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0776 0,5842 0,4176 0,3141 0,2737 0,2242 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1176 0,6705 0,5192 0,4185 0,3714 0,3149 NOFB IMG
isi 1 0,4956 0,8100 0,7703 0,7414 0,7279 0,7032 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,4966 0,8098 0,7718 0,7418 0,7290 0,7035 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5040 0,8181 0,7824 0,7471 0,7328 0,6921 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1862 0,7269 0,6158 0,5225 0,4832 0,4254 NOFB IMGBRO
isi 5 0,4956 0,8100 0,7703 0,7414 0,7279 0,7032 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5042 0,8162 0,7828 0,7474 0,7314 0,6908 GRADED IMGMETBRO
isi2 2 0,4985 0,8095 0,7720 0,7421 0,7300 0,7023 GRADED IMGMETBRO
isi2 3 0,4972 0,8098 0,7700 0,7406 0,7294 0,7018 GRADED IMGMETBRO
isi2 4 0,5047 0,8180 0,7822 0,7479 0,7320 0,6913 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0688 0,5608 0,3991 0,2967 0,2583 0,2076
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0530 0,4398 0,3048 0,2252 0,1938 0,1473
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0702 0,5672 0,4022 0,3008 0,2609 0,2111
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0707 0,5650 0,4058 0,3076 0,2684 0,2135
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0707 0,5644 0,4062 0,3074 0,2682 0,2134
VCTLab 1 0,0751 0,4240 0,3433 0,2946 0,2726 0,2400 NOFB IMGBRO
VCTLab 2 0,0782 0,4537 0,3626 0,3067 0,2767 0,2395 NOFB IMGBRO
VCTLab 3 0,0749 0,4355 0,3526 0,2962 0,2665 0,2343 NOFB IMGBRO
VCTLab 4 0,0659 0,3787 0,3132 0,2618 0,2414 0,2106 NOFB IMGBRO
VCTLab 5 0,0679 0,3852 0,3165 0,2716 0,2455 0,2096 NOFB IMGBRO
WIDE_IO WideIO 0,0583 0,4354 0,3217 0,2470 0,2166 0,1836
Mean 0,2135 0,6327 0,5330 0,4614 0,4308 0,3857
Std. Deviation 0,1915 0,1544 0,1808 0,1990 0,2077 0,2140

Male User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3448 0,7511 0,6797 0,6312 0,6025 0,5671 NOFB IMGMETBRO
DBIS run2 0,3896 0,7630 0,7131 0,6689 0,6443 0,6165 NOFB IMGMETBRO
DBIS run3 0,4078 0,7748 0,7211 0,6770 0,6509 0,6125 GRADED IMGMETBRO
FINKI run1 0,1406 0,6818 0,5470 0,4348 0,3813 0,3100 NOFB IMGBRO
FINKI run2 0,1421 0,6899 0,5505 0,4364 0,3844 0,3128 NOFB IMGBRO
FINKI run3 0,1397 0,6852 0,5507 0,4361 0,3858 0,3139 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1136 0,6527 0,5055 0,4033 0,3640 0,3048 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1099 0,6243 0,4864 0,3809 0,3397 0,2877 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0795 0,5700 0,4097 0,3103 0,2707 0,2222 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1181 0,6567 0,5063 0,4081 0,3621 0,3092 NOFB IMG
isi 1 0,5090 0,8044 0,7614 0,7344 0,7216 0,7045 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,5101 0,8044 0,7625 0,7349 0,7231 0,7048 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5185 0,8110 0,7736 0,7401 0,7267 0,6947 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1855 0,7145 0,6014 0,5090 0,4700 0,4145 NOFB IMGBRO
isi 5 0,5090 0,8044 0,7614 0,7344 0,7216 0,7045 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5188 0,8090 0,7739 0,7397 0,7251 0,6933 GRADED IMGMETBRO
isi2 2 0,5121 0,8050 0,7639 0,7356 0,7241 0,7037 GRADED IMGMETBRO
isi2 3 0,5107 0,8047 0,7619 0,7333 0,7231 0,7031 GRADED IMGMETBRO
isi2 4 0,5192 0,8108 0,7736 0,7401 0,7257 0,6938 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0698 0,5469 0,3864 0,2886 0,2520 0,2043
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0549 0,4469 0,3109 0,2298 0,1980 0,1507
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0712 0,5516 0,3908 0,2941 0,2555 0,2074
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0717 0,5506 0,3944 0,3007 0,2627 0,2097
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0717 0,5494 0,3952 0,3004 0,2625 0,2096
VCTLab 1 0,0769 0,4148 0,3386 0,2915 0,2679 0,2360 NOFB IMGBRO
VCTLab 2 0,0805 0,4407 0,3531 0,3019 0,2740 0,2375 NOFB IMGBRO
VCTLab 3 0,0771 0,4227 0,3445 0,2922 0,2637 0,2321 NOFB IMGBRO
VCTLab 4 0,0674 0,3755 0,3087 0,2578 0,2385 0,2077 NOFB IMGBRO
VCTLab 5 0,0695 0,3800 0,3134 0,2710 0,2445 0,2082 NOFB IMGBRO
WIDE_IO WideIO 0,0598 0,4365 0,3201 0,2448 0,2140 0,1812
Mean 0,2188 0,6239 0,5239 0,4550 0,4253 0,3838
Std. Deviation 0,1971 0,1543 0,1795 0,1976 0,2065 0,2160

Female User
Group Run ID map_cut_100 ndcg_cut_5 ndcg_cut_10 ndcg_cut_20 ndcg_cut_30 ndcg_cut_100 Remark RF? Type
DBIS run1 0,3644 0,7786 0,7094 0,6611 0,6316 0,5931 NOFB IMGMETBRO
DBIS run2 0,4049 0,7975 0,7445 0,6984 0,6708 0,6452 NOFB IMGMETBRO
DBIS run3 0,4250 0,8051 0,7509 0,7161 0,6903 0,6468 GRADED IMGMETBRO
FINKI run1 0,1434 0,6974 0,5661 0,4554 0,4020 0,3244 NOFB IMGBRO
FINKI run2 0,1448 0,7055 0,5692 0,4565 0,4042 0,3262 NOFB IMGBRO
FINKI run3 0,1420 0,7002 0,5670 0,4563 0,4065 0,3290 NOFB IMGBRO
InformationProcessingLaboratory IPL13_visual_r1 0,1150 0,6784 0,5269 0,4196 0,3784 0,3164 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r2 0,1126 0,6449 0,5062 0,3988 0,3611 0,3033 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r3 0,0810 0,5824 0,4201 0,3198 0,2809 0,2315 NOFB IMG
InformationProcessingLaboratory IPL13_visual_r4 0,1180 0,6782 0,5288 0,4244 0,3764 0,3198 NOFB IMG
isi 1 0,5352 0,8323 0,7971 0,7779 0,7697 0,7467 uses IPTC city tag GRADED IMGMETBRO
isi 2 0,5363 0,8321 0,7984 0,7779 0,7713 0,7468 uses IPTC city tag GRADED IMGMETBRO
isi 3 0,5473 0,8388 0,8097 0,7849 0,7783 0,7395 uses IPTC city tag GRADED IMGMETBRO
isi 4 0,1887 0,7381 0,6169 0,5285 0,4924 0,4339 NOFB IMGBRO
isi 5 0,5352 0,8323 0,7971 0,7779 0,7697 0,7467 uses IPTC city tag GRADED IMGMETBRO
isi2 1 0,5473 0,8385 0,8102 0,7861 0,7780 0,7383 GRADED IMGMETBRO
isi2 2 0,5386 0,8318 0,7980 0,7804 0,7725 0,7461 GRADED IMGMETBRO
isi2 3 0,5373 0,8327 0,7965 0,7790 0,7719 0,7459 GRADED IMGMETBRO
isi2 4 0,5479 0,8407 0,8098 0,7868 0,7788 0,7392 GRADED IMGMETBRO
ThssMpam4 ThssMpam4_retrieval_5X1000_CR 0,0707 0,5607 0,3998 0,2976 0,2598 0,2113
ThssMpam4 ThssMpam4_retrieval_SURFMATCH 0,0577 0,4555 0,3182 0,2364 0,2045 0,1556
ThssMpam4 ThssMpam4_retrieval_5000_NTI_CR 0,0722 0,5677 0,4008 0,3010 0,2633 0,2153
ThssMpam4 ThssMpam4_retrieval_5000_TI_CR 0,0724 0,5630 0,4046 0,3074 0,2701 0,2174
ThssMpam4 ThssMpam4_retrieval_5000_TI_NCR 0,0724 0,5610 0,4046 0,3071 0,2698 0,2173
VCTLab 1 0,0817 0,4605 0,3733 0,3200 0,2958 0,2548 NOFB IMGBRO
VCTLab 2 0,0828 0,4760 0,3822 0,3225 0,2914 0,2482 NOFB IMGBRO
VCTLab 3 0,0796 0,4615 0,3698 0,3128 0,2822 0,2430 NOFB IMGBRO
VCTLab 4 0,0726 0,4174 0,3434 0,2852 0,2624 0,2257 NOFB IMGBRO
VCTLab 5 0,0733 0,4119 0,3359 0,2874 0,2586 0,2187 NOFB IMGBRO
WIDE_IO WideIO 0,0633 0,4489 0,3341 0,2565 0,2252 0,1922
Mean 0,2285 0,6477 0,5476 0,4794 0,4509 0,4049
Std. Deviation 0,2086 0,1540 0,1862 0,2110 0,2227 0,2307


License

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Germany License.

Please cite this report for proper attribution:

@inproceedings{Zel12,
author = {Zellh\"ofer, David},
title = {An Extensible Personal Photograph Collection for Graded Relevance Assessments and User Simulation},
pages = {to appear},
publisher = {ACM},
isbn = {978-1-4503-1329-2},
series = {ICMR '12},
booktitle = {{P}roceedings of the 2nd {A}{C}{M} {I}nternational {C}onference on {M}ultimedia {R}etrieval},
year = {2012}
}

A technical report with similar content is available.

"© ACM, 2012. This is the author's version of the work. It is posted here by permission of ACM for
your personal use. Not for redistribution.
The definitive version was published in Proceedings of the 2nd ACM International Conference on Multimedia Retrieval,
{VOL#, ISS#, (DATE)} http://doi.acm.org/10.1145/nnnnnn.nnnnnn"

Organization

  • David Zellhöfer, Brandenburg University of Technology, Germany, david.zellhoefer[at]tu-cottbus.de
  • Workgroup of Database and Information Systems, Brandenburg University of Technology, Website

This research was supported by a grant of the Federal Ministry of Education and Research (Grant Number 03FO3072).

BTU Logo

AttachmentSize
PDF icon Manual for the Baseline System172.62 KB
Package icon Topics1.85 KB
Office spreadsheet icon Results81.5 KB
Package icon Ground truth218.01 KB