ImageCLEFaware

Motivation

Images constitute a large part of the content shared on social networks. Their disclosure is often related to a particular context and users are often unaware of the fact that, depending on their privacy status, images can be accessible to third parties and be used for purposes which were initially unforeseen. For instance, it is common practice for employers to search information about their future employees online. Another example of usage is that of automatic credit scoring based on online data. Most existing approaches which propose feedback about shared data focus on inferring user characteristics and their practical utility is rather limited.

We hypothesize that user feedback would be more efficient if conveyed through the real-life effects of data sharing. The objective of the task is to automatically score user photographic profiles in a series of situations with strong impact on her/his life. Four such situations were modeled this year and refer to searching for: (1) a bank loan, (2) an accommodation, (3) a job as waitress/waiter and (4) a job in IT. The inclusion of several situations is interesting in order to make it clear to the end users of the system that the same image will be interpreted differently depending on the context.

The final objective of the task is to encourage the development of efficient user feedback, such as the YDSYO Android app.
The current functioning is summarized in this video.

News

More information will be added soon!

Preliminary Schedule

• 16.11.2020: registration opens for all ImageCLEF tasks
• 22.02.2021: development data release starts
• 15.03.2021: test data release starts
• 10.05.2021: deadline for submitting the participants runs
• 15.05.2021: release of the processed results by the task organizers (depends on the task)
• 28.05.2021: deadline for submission of working notes papers by the participants
• 11.06.2021: notification of acceptance of the working notes papers
• 02.07.2021: camera ready working notes papers
• 21-24.09.2021: CLEF 2021, Bucharest, Romania

Task description

Given the training dataset described below, participants will propose machine learning techniques which provide a ranking of test user profiles in each situation which is as close as possible to a human ranking of the test profiles.

Data

This is the first edition of the task. A data set of 500 user profiles with 100 photos per profile was created and annotated with an \say{appeal} score for a series of real-life situations via crowdsourcing.
Participants to the experiment were asked to provide a global rating of each profile in each situation modeled using a 7-points Likert scale ranging from \say{strongly unappealing} to \say{strongly appealing}.
The averaged \say{appeal} score will be used to create a ground truth composed of ranked users in each modeled situation.
User profiles are created by repurposing a subset of the YFCC100M dataset.

In accordance with GDPR, data minimization is applied and participants receive only the information necessary to carry out the task in an anonymized form.
Resources include (i) anonymized visual concept ratings for each situation modeled; (ii) automatically extracted predictions for the images that compose the profiles.
The final objective of the task is to integrate the most promising of the developed algorithms into YDSYO\footnote{\url{https://ydsyo.app}}, a mobile app that provides situation-related feedback to users.

Evaluation methodology

Participants to the task will provide an automatically ranking of user ratings for each situation which will be compared to a ground truth rating obtained by crowdsourcing (see "Data" section below).
The correlation between the two ranked list will be measured using Pearson's correlation coefficient.
The final score of each participating team will be obtained by averaging correlations obtained for individual situations.

Participant registration

Please refer to the general ImageCLEF registration instructions

Submission instructions

The submissions will be received through the crowdAI system.
Participants will be permitted to submit up to 10 runs. External training data is not allowed.

More information will be added soon!

Results

More information will be added soon!

CEUR Working Notes

• All participating teams with at least one graded submission, regardless of the score, should submit a CEUR working notes paper.
• Official detailed instructions for the CLEF 2021 working notes can be found here: http://clef2021.clef-initiative.eu/index.php?page=Pages/instructions_for_authors.html

References

Van-Khoa Nguyen, Adrian Popescu, and Jerome Deshayes-Chossart. "Unveiling Real-Life Effects of Online Photo Sharing." arXiv preprint arXiv:2012.13180 (2020).

Organizers

• Adrian Popescu <adrian.popescu(at)cea.fr>, CEA LIST, France
• Jérôme Deshayes-Chossart <jerome.deshayes-chossart(at)cea.fr>, CEA LIST, France
• Bogdan Ionescu, University Politehnica of Bucharest, Romania

Acknowledgements

This task is supported under project AI4Media, A European Excellence Centre for Media, Society and Democracy, H2020 ICT-48-2020, grant #951911.