You are here

Visual Concept Detection and Annotation Task 2009: Results Hierarchical Measure

In the table below you can find the average annotation performance over all photos of the testset for each run in alphabetical order. If you are interested in the score for each photo of your run, please have a look here.
We determined the annotator agreement between concepts. Therefore 11 persons annotated the same 100 photos. The list with the agreements is here. In the table you can compare the results for your run, when the annotator agreement is taken into account as scaling factor and when not.
________________________________________________________________________________________________________

Summary of Runs evaluated with the hierarchical measure

Run Average Annotation Score with Annotator Agreement Average Annotation Score without Annotator Agreement
apexlab_61_2_1245568188287.txt 0.75611556 0.73274606
apexlab_61_2_1245569401060.txt 0.64574885 0.6194745
apexlab_61_2_1245571723120.txt 0.64574885 0.6194745
AVEIR_81_2_1245664594607.txt 0.44242984 0.38653708
AVEIR_81_2_1245664654451.txt 0.50426006 0.46583846
AVEIR_81_2_1245664755158.txt 0.43001753 0.37243047
AVEIR_81_2_1245664820829.txt 0.39556795 0.33309698
bpacad_18_2_1245621915717.txt 0.5522943 0.52551454
bpacad_18_2_1245622065724.txt 0.63162863 0.58996934
bpacad_18_2_1245622367725.txt 0.6096587 0.5746476
bpacad_18_2_1245622473940.txt 0.61271924 0.5778141
bpacad_18_2_1245623805085.txt 0.68395746 0.6522653
CEA LIST_24_2_1245339305674.txt 0.70892227 0.67913336
CEA LIST_24_2_1245339411928.txt 0.6880514 0.65703386
CEA LIST_24_2_1245339505739.txt 0.6936255 0.6622523
CEA LIST_24_2_1245339660021.txt 0.6959019 0.66484046
CVIUI2R_22_2_1244628714641.txt 0.8092977 0.78838325
CVIUI2R_22_2_1244629050173.txt 0.80944806 0.78855246
FIR2_92_2_1245417583652.txt 0.8016111 0.7792978
FIR2_92_2_1245516363637.txt 0.8016377 0.77933127
FIR2_92_2_1245516525876.txt 0.79847556 0.7758949
FIR2_92_2_1245516822169.txt 0.79342705 0.7711678
IAM Southampton_30_2_1245438072355.txt 0.41258514 0.3893898
IAM Southampton_30_2_1245519187248.txt 0.39195824 0.3690196
IAM Southampton_30_2_1245519327555.txt 0.36823183 0.34262568
INAOE TIA_66_2_1245620227762.txt 0.74411815 0.7153912
INAOE TIA_66_2_1245630111443.txt 0.51931924 0.49777094
INAOE TIA_66_2_1245631132743.txt 0.72582835 0.6943392
INAOE TIA_66_2_1245631427651.txt 0.7296937 0.69904524
INAOE TIA_66_2_1245631711558.txt 0.73449016 0.7049101
ISIS University of Amsterdam_76_2_1245075326057.txt 0.7398107 0.71989775
ISIS University of Amsterdam_76_2_1245075554764.txt 0.7623391 0.7374101
ISIS University of Amsterdam_76_2_1245075919629.txt 0.7647352 0.73999774
ISIS University of Amsterdam_76_2_1245076320558.txt 0.749065 0.72315747
ISIS University of Amsterdam_76_2_1245076767051.txt 0.758124 0.73289967
KameyamaLab_21_2_1245594455534.txt 0.47097242 0.45414135
KameyamaLab_21_2_1245616956073.txt 0.78563696 0.762027
KameyamaLab_21_2_1245617358988.txt 0.52566075 0.50214314
KameyamaLab_21_2_1245617609041.txt 0.793902 0.7711168
KameyamaLab_21_2_1245617818119.txt 0.7961495 0.77342385
LEAR_44_2_1245581505805.txt 0.7225312 0.6940174
LEAR_44_2_1245581860906.txt 0.7312141 0.70315176
LEAR_44_2_1245581967963.txt 0.74848056 0.7218529
LEAR_44_2_1245582143586.txt 0.7702538 0.7463586
LEAR_44_2_1245582451309.txt 0.7312745 0.70348054
LIP6_17_2_1245498965150.txt 0.4395645 0.40859267
LIP6_17_2_1245579903668.txt 0.26109898 0.23885955
LIP6_17_2_1245610812916.txt 0.26151088 0.23977785
LIP6_17_2_1245617011478.txt 0.26016545 0.23786168
LIP6_17_2_1245618383113.txt 0.26147848 0.23979928
LSIS_14_2_1245595798047.txt 0.39337575 0.35644892
LSIS_14_2_1245595947674.txt 0.48181027 0.4526701
LSIS_14_2_1245596027969.txt 0.3655994 0.3359449
LSIS_14_2_1245664294478.txt 0.51217276 0.45836368
LSIS_14_2_1245664386697.txt 0.49277753 0.45700964
MMIS_33_2_1245434554581.txt 0.50254005 0.44883654
MMIS_33_2_1245586552541.txt 0.57598627 0.53034455
MMIS_33_2_1245601239738.txt 0.37549266 0.3015072
MMIS_33_2_1245611281967.txt 0.5760397 0.5304037
MMIS_33_2_1245674693001.txt 0.57525134 0.5295074
MRIM_50_2_1245573948330.txt 0.65106684 0.6093575
MRIM_50_2_1245574356224.txt 0.6875562 0.64959854
MRIM_50_2_1245574874836.txt 0.71565866 0.68281114
MRIM_50_2_1245578233616.txt 0.56599104 0.5299541
Random 0.38346142 0.35004056
TELECOM ParisTech_39_2_1245415171956.txt 0.38478595 0.3549008
TELECOM ParisTech_39_2_1245415387504.txt 0.33444405 0.31251764
UAIC_34_2_1244812428616_changed.txt 0.69028765 0.6550599
Wroclaw University of Technology_49_2_1245621871943.txt 0.43492436 0.41822502
Wroclaw University of Technology_49_2_1245622279557.txt 0.77879554 0.7526036
Wroclaw University of Technology_49_2_1245622900052.txt 0.46332258 0.44425774
Wroclaw University of Technology_49_2_1245622992589.txt 0.46332258 0.44425774
Wroclaw University of Technology_49_2_1245624864334.txt 0.7672884 0.7395571
XRCE_36_2_1245429345115.txt 0.81071573 0.78999674

___________________________________________________________________________________________________________

OLD, BUGGY Results: Summary of Runs evaluated with the hierarchical measure

Run Average Annotation Score with Annotator Agreement Average Annotation Score without Annotator Agreement
apexlab_61_2_1245568188287.txt 0.77994823 0.7585434
apexlab_61_2_1245569401060.txt 0.6588524 0.6336668
apexlab_61_2_1245571723120.txt 0.6588524 0.6336668
AVEIR_81_2_1245664594607.txt 0.475304 0.42211136
AVEIR_81_2_1245664654451.txt 0.5162502 0.47880462
AVEIR_81_2_1245664755158.txt 0.46305847 0.40817085
AVEIR_81_2_1245664820829.txt 0.428374 0.36858788
bpacad_18_2_1245621915717.txt 0.577842 0.5531843
bpacad_18_2_1245622065724.txt 0.64546704 0.604971
bpacad_18_2_1245622367725.txt 0.6194188 0.5851988
bpacad_18_2_1245622473940.txt 0.621747 0.5875682
bpacad_18_2_1245623805085.txt 0.70747864 0.67775553
CEA LIST_24_2_1245339305674.txt 0.75172716 0.72546273
CEA LIST_24_2_1245339411928.txt 0.73096365 0.7034727
CEA LIST_24_2_1245339505739.txt 0.7369552 0.70914716
CEA LIST_24_2_1245339660021.txt 0.73920727 0.7117068
CVIUI2R_22_2_1244628714641.txt 0.82751185 0.8080815
CVIUI2R_22_2_1244629050173.txt 0.8276921 0.8082839
FIR2_92_2_1245417583652.txt 0.8152637 0.7940429
FIR2_92_2_1245516363637.txt 0.8152935 0.7940799
FIR2_92_2_1245516525876.txt 0.81217587 0.79069155
FIR2_92_2_1245516822169.txt 0.8068478 0.7856627
IAM Southampton_30_2_1245438072355.txt 0.41897374 0.39630276
IAM Southampton_30_2_1245519187248.txt 0.39985585 0.37756625
IAM Southampton_30_2_1245519327555.txt 0.37403193 0.34889817
INAOE TIA_66_2_1245620227762.txt 0.7591791 0.73166347
INAOE TIA_66_2_1245630111443.txt 0.5329161 0.512457
INAOE TIA_66_2_1245631132743.txt 0.72801954 0.69661
INAOE TIA_66_2_1245631427651.txt 0.7323485 0.70182383
INAOE TIA_66_2_1245631711558.txt 0.7417855 0.7127507
ISIS University of Amsterdam_76_2_1245075326057.txt 0.7502514 0.73118794
ISIS University of Amsterdam_76_2_1245075554764.txt 0.7811672 0.7577831
ISIS University of Amsterdam_76_2_1245075919629.txt 0.7830596 0.759825
ISIS University of Amsterdam_76_2_1245076320558.txt 0.77050894 0.746365
ISIS University of Amsterdam_76_2_1245076767051.txt 0.7779823 0.7543934
KameyamaLab_21_2_1245594455534.txt 0.48170832 0.4657534
KameyamaLab_21_2_1245616956073.txt 0.8001519 0.7777083
KameyamaLab_21_2_1245617358988.txt 0.5472044 0.5254423
KameyamaLab_21_2_1245617609041.txt 0.8072495 0.7855501
KameyamaLab_21_2_1245617818119.txt 0.8085204 0.786806
LEAR_44_2_1245581505805.txt 0.7487974 0.7224225
LEAR_44_2_1245581860906.txt 0.7573569 0.7314461
LEAR_44_2_1245581967963.txt 0.77192163 0.74723846
LEAR_44_2_1245582143586.txt 0.7918007 0.76966405
LEAR_44_2_1245582451309.txt 0.7561616 0.73041534
LIP6_17_2_1245498965150.txt 0.44497892 0.41448137
LIP6_17_2_1245579903668.txt 0.26109898 0.23885955
LIP6_17_2_1245610812916.txt 0.26151088 0.23977785
LIP6_17_2_1245617011478.txt 0.26016545 0.23786168
LIP6_17_2_1245618383113.txt 0.26147848 0.23979928
LSIS_14_2_1245595798047.txt 0.3950422 0.35825703
LSIS_14_2_1245595947674.txt 0.48310107 0.45405138
LSIS_14_2_1245596027969.txt 0.36631873 0.33672667
LSIS_14_2_1245664294478.txt 0.54901516 0.4982652
LSIS_14_2_1245664386697.txt 0.50417936 0.4693403
MMIS_33_2_1245434554581.txt 0.5479666 0.49800622
MMIS_33_2_1245586552541.txt 0.6179764 0.57577974
MMIS_33_2_1245601239738.txt 0.42055705 0.35027474
MMIS_33_2_1245611281967.txt 0.6180272 0.5758361
MMIS_33_2_1245674693001.txt 0.6172693 0.5749729
MRIM_50_2_1245573948330.txt 0.6792413 0.6398938
MRIM_50_2_1245574356224.txt 0.71418947 0.6784494
MRIM_50_2_1245574874836.txt 0.7413273 0.71056616
MRIM_50_2_1245578233616.txt 0.5905188 0.5565225
Random 0.38431713 0.35097164
TELECOM ParisTech_39_2_1245415171956.txt 0.39030218 0.36085272
TELECOM ParisTech_39_2_1245415387504.txt 0.34283262 0.32156074
UAIC_34_2_1244812428616_changed.txt 0.7236283 0.69118863
Wroclaw University of Technology_49_2_1245621871943.txt 0.4453914 0.42950743
Wroclaw University of Technology_49_2_1245622279557.txt 0.79032654 0.7650345
Wroclaw University of Technology_49_2_1245622900052.txt 0.4727444 0.45442307
Wroclaw University of Technology_49_2_1245622992589.txt 0.4727444 0.45442307
Wroclaw University of Technology_49_2_1245624864334.txt 0.7775906 0.75067323
XRCE_36_2_1245429345115.txt 0.82949024 0.81032526
AttachmentSize
Plain text icon agreements.txt2.22 KB