You are here

Home

Photo Annotation Task 2011: Results for the Example-based Evaluation

The table depicts the annotation performance averaged over all photos for each run in alphabetical order. The columns show the run identifier, the results evaluated with the example-based F-measure (F-ex), the results for the evaluation with the Semantic R-Precision (SR-Precision), and the configuration of the submission. The SR-Precision is a novel performance measure. It is derived from the example-based R-Precision measure. In contrast to R-Precision, it considers the Flickr Tag Similarity measure to determine the semantic relatedness of misclassified concepts. As the R-Precision, the SR-Precision is a performance measure which works on ranked predictions of the classifier, while the F-ex measure considers label set predictions. The SR-Precision assigns a score of 1 for perfect prediction.

Please note that the SR-Precision measure is only considered as a test for example-based evaluation. It poses the requirement of comparability of classifier scores which was not explicitly stated beforehand to the participants. By addressing this requirement, teams might achieve higher evaluation scores.

Summary of results of the example-based evaluation

Run F-ex SR-Precision Visual Textual Multi-modal
BPACAD_1308238535311__bpacad_avg_cns.txt 0.568004 0.72902450 X
BPACAD_1308261535886__avg_cssj.pred.txt 0.585842 0.70911366 X
BPACAD_1308263253401__bpacad_jensen.txt 0.502440 0.63528085 X
BPACAD_1308265398652__bpacad_wecssj.txt 0.593088 0.71928000 X
BPACAD_1308265910609__bpacad_wecssj_am2.txt 0.545341 0.70853215 X
BUFFALO_1308353494435__2011_June_17_final_1_XiDaCo.txt 0.526883 0.68274504 X
BUFFALO_1308353862419__2011_June_17_final_2_XiDaCo.txt 0.517385 0.67521644 X
BUFFALO_1308355303811__imageclef2011_XiDaCo_3.txt 0.494133 0.67102330 X
BUFFALO_1308355432416__imageclef2011_XiDaCo_4.txt 0.486641 0.66385630 X
BUFFALO_1308449288137__imageclef2011_XiDaCo_5.txt 0.471073 0.65038390 X
CAEN_1306759634594__GREYC_MultiFeat_Chi2SVM.txt 0.559980 0.69383204 X
CAEN_1308305392853__GREYC_BoW+FisherKernel.txt 0.600307 0.72691345 X
CAEN_1308313475389__GREYC_SVMOutput.txt 0.490982 0.65120530 X
CAEN_1308316457541__GREYC_BoW+FisherKernel+SVMOuput.txt 0.591824 0.72466100 X
CEALIST_1308300279159__CEALIST_text_Fsb.txt 0.478517 0.67578644 X
CEALIST_1308311302883__CEALIST_piria5_FsbRdsa.txt 0.503486 0.70041360 X
CEALIST_1308311395247__CEALIST_piria5_FsbRdsa_text.txt 0.497925 0.70438200 X
CEALIST_1308675752517__CEALIST_piria5_FsbRdsa_text2.txt 0.508043 0.71098950 X
CEALIST_1308676000353__CEALIST_fklsp_FsbRdsa_text2.txt 0.484388 0.69311965 X
DBIS_1308502854426__DBIS_AnnotationTask_run01.txt 0.503050 0.65548706 X
DBIS_1308502958123__DBIS_AnnotationTask_run02.txt 0.510672 0.66625820 X
DBIS_1308503209671__DBIS_AnnotationTask_run03.txt 0.518191 0.67118920 X
DBIS_1308503283580__DBIS_AnnotationTask_run04.txt 0.481454 0.65548706 X
DBIS_1308503370780__DBIS_AnnotationTask_run05.txt 0.463859 0.66850656 X
HHI_1308673665950__clef2011_test_o_t.txt 0.579904 0.71262264 X
HHI_1308673737018__clef2011_test_o+ss_t.txt 0.580804 0.71251965 X
HHI_1308673811451__clef2011_test_o+g_t.txt 0.588042 0.71764547 X
HHI_1308673923203__clef2011_test_o+ss_t+inf+excl.txt 0.580729 0.71252900 X
HHI_1308683266045__clef2011_test_o+ss_t+inf.txt 0.580792 0.71252900 X
IDMT_1308309796049__tags.txt 0.525352 0.67673475 X
IDMT_1308309978916__earlyfusion_rgbsift_tags.txt 0.541316 0.71278083 X
IDMT_1308310711437__IDMT_latefusion_rgbsift_tags.txt 0.551224 0.70141750 X
IDMT_1308310771509__IDMT_latefusion_rgbsift_tags_tagpost.txt 0.551918 0.70138430 X
IDMT_1308310966208__IDMT_earlyfusion_rgbsift_tags_tagpost.txt 0.541587 0.71212930 X
ISIS_1308686553452__runpa-UvA-multimodal-coreA-mkl.txt 0.521205 0.70777990 X
ISIS_1308686786417__runpa-UvA-multimodal-coreA.txt 0.622038 0.74187386 X
ISIS_1308687411165__runpa-UvA-coreA.txt 0.611786 0.73449880 X
ISIS_1308687611695__runpa-UvA-core.txt 0.608159 0.73210645 X
ISIS_1308687805156__runpa-UvA-corefast.txt 0.604933 0.72984165 X
LAPI_1306496688389__ANNOT_LDA_wo_post_lapli2.tex.txt 0.386226 0.55125690 X
LAPI_1308386433242__ANNOT_LDA_post_lapli2.tex.txt 0.390112 0.55125654 X
LIRIS_1308689153154__1text_model_1.txt 0.431725 0.67493284 X
LIRIS_1308689353311__2visual_text_model_2.txt 0.575757 0.71740430 X
LIRIS_1308689578403__text_model_3.txt 0.409709 0.67576570 X
LIRIS_1308689722316__4visual_model_4.txt 0.539452 0.72505510 X
LIRIS_1308689855977__5visual_text_model_5.txt 0.566935 0.71826094 X
MEIJI_1308314805408__meiji_vw_r.txt 0.471923 0.43197626 X
MEIJI_1308315068076__meiji_tag_r.txt 0.458523 0.49061212 X
MEIJI_1308496313915__meiji_vw_color_r.txt 0.451822 0.45208520 X
MEIJI_1308496549558__meiji_vw_color_tag_r.txt 0.423201 0.47993870 X
MEIJI_1308496872087__meiji_vw_tag_r-vw_r.txt 0.495162 0.46919692 X
MLKD_1307607230166__Sub2_Tags.txt 0.506114 0.65265820 X
MLKD_1307607669838__Sub1_Visual.txt 0.559546 0.69805810 X
MLKD_1307608333045__Sub3_Multi1.txt 0.558795 0.69819640 X
MLKD_1307609091226__Sub4_Multi2.txt 0.509443 0.66871190 X
MLKD_1307609601024__Sub5_Multi3.txt 0.516479 0.67086214 X
MRIM_1308226825708__LIG_run_test_final.txt 0.544138 0.70335567 X
MRIM_1308318230664__LIG_run_msvm_tags_final.txt 0.552276 0.70642180 X
MRIM_1308318529187__LIG_msvm_two_desc_final.txt 0.536680 0.69039620 X
MRIM_1308326177978__res_final_LIG_CG.txt 0.004937 0.27391845 X
MUFIN_1308261784179__MufinSubmission100.txt 0.461820 0.62794690 X
MUFIN_1308261985797__MufinSubmission101.txt 0.447870 0.61690420 X
MUFIN_1308262673276__MufinSubmission110.txt 0.442091 0.61257595 X
MUFIN_1308290147258__MufinSubmission120.txt 0.384356 0.55653490 X
NII_1308499021260__NII.Run5.cCV_MIXED.GlobalFeature.txt 0.298222 0.68731534 X
NII_1308499728632__NII.Run4.cCV_MIXED.LocalFeatureLight.txt 0.270543 0.69656230 X
NII_1308499994137__NII.Run3.cCV_MIXED.LocalFeatureLight+GlobalFeature.txt 0.270317 0.69666830 X
NII_1308577732866__NII.Run2.cCV_MIXED.LocalFeatureFull.txt 0.272203 0.70180240 X
NII_1308648063465__NII.Run1.cCV_MIXED.LocalFeatureFull+Global.txt 0.271886 0.70127910 X
REGIMVID_1308507656911__REGIMVID.txt 0.140913 0.39596778 X
TUBFI_1306846967062__tubfi_testrun1_purevisual.txt 0.550228 0.61771655 X
TUBFI_1307228738426__scores.txt 0.550709 0.63000554 X
TUBFI_1307276000026__scores.txt 0.565980 0.55861930 X
TUBFI_1307570187656__scores.txt 0.551744 0.62136513 X
TUBFI_1307625062344__scores.txt 0.548459 0.61896770 X
UNIKLU_1308339327587__UNIKLU_SVM_ALL_UNFILTERED.txt 0.469376 0.67023460 X
UNIKLU_1308339699147__UNIKLU_SVM_SELECTED_UNFILTERED.txt 0.468492 0.67155670 X
UNIKLU_1308340011026__UNIKLU_SVM_SELECTED_FILTERED.txt 0.468540 0.67155640 X
UNIKLU_1308422176166__UNIKLU_SVM_ALL_FILTERED.txt 0.469429 0.67022290 X

ImageCLEF - Cross-language image retrieval evaluations