Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics

Aleksandr Gushchin^1,2,3, Khaled Abud^1,2, Georgii Bychkov^1,2,3, Ekaterina Shumitskaya^1,2,3, Anna Chistyakova^1,3, Sergey Lavrushkin^2,3, Bader Rasheed⁴, Kirill Malyshev¹, Dmitriy Vatolin^1,2,3,4, Anastasia Antsiferova^2,3,4,

¹ Lomonosov Moscow State University
² MSU Institute for Artificial Intelligence
³ ISP RAS Research Center for Trusted Artificial Intelligence
⁴ Innopolis University

Paper Code arXiv

Abstract

Modern neural-network-based Image Quality Assessment (IQA) metrics are vulnerable to adversarial attacks, which can be exploited to manipulate search engine rankings, benchmark results, and content quality assessments, raising concerns about the reliability of IQA metrics in critical applications. This paper presents the first comprehensive study of IQA defense mechanisms in response to adversarial attacks on these metrics to pave the way for safer use of IQA metrics. We systematically evaluated 30 defense strategies, including purification, training-based, and certified methods --- and applied 14 adversarial attacks in adaptive and non-adaptive settings to compare these defenses on 9 no-reference IQA metrics. Our proposed benchmark aims to guide the development of IQA defense methods and is open to submissions; the latest results and code are at https://videoprocessing.ai/benchmarks/iqa-defenses.html

Key features

Our benchmark is the first to evaluate and compare different techniques for defending IQA metrics. Key features are:

Datasets: KonIQ‑10k, KADID‑10k, NIPS‑2017, AGIQA‑3K
Metrics: Meta‑IQA, MANIQA, CLIP‑IQA+, TOPIQ, Koncept, SPAQ, PAQ2PIQ, Linearity, FPR
Attacks: 14 white/black‑box, 3 budgets, adaptive vs. non‑adaptive
Defenses: 17 purification, 7 adversarial‑training, 6 certified
Subjective study (60k+ votes)
Open online leaderboard for continuous submissions

Evaluation scheme: The benchmark consists of four parts: datasets, IQA models, adversarial attacks, and adversarial defenses.

Adversarial defenses efficiency for IQA metrics in terms of SROCC (left) and D_score (right). Bars and dots are for adaptive and non-adaptive attacks, respectively. Each dot represents the result for each preset of defense. Red dots represent a selected preset for the adaptive case. Results are averaged across 9 IQA metrics and 14 attacks.

Results of subjective evaluation for purification methods.

Examples of artifacts caused by various defenses when MANIQA metric is attacked. We selectively zoom in on key parts of the images to highlight the details.

Results and submission

Latest results and steps to submit your method can be found here: https://videoprocessing.ai/benchmarks/iqa-defenses.html.

BibTeX

        @misc{gushchin2024guardiansimagequalitybenchmarking,
        title={Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics}, 
        author={Alexander Gushchin and Khaled Abud and Georgii Bychkov and Ekaterina Shumitskaya and Anna Chistyakova and Sergey Lavrushkin and Bader Rasheed and Kirill Malyshev and Dmitriy Vatolin and Anastasia Antsiferova},
        year={2024},
        eprint={2408.01541},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2408.01541}, 
        }