Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In this 2018 work at the International Conference on Machine Learning, Anish Athalye, Nicholas Carlini, and David Wagner define obfuscated gradients as a special case of gradient masking in which a defense makes the attack gradient unusable or misleading. They provide warning signs for this problem and techniques that change how an attack handles backward calculations, randomness, or unstable computations. By the end, you should understand those warning signs, what makes an attack adaptive, and why an evaluator must distinguish a failed attack from a successful defense.

Obfuscated gradients are a special case of gradient masking in which a defense’s construction makes the attack gradient unusable or misleading. The authors group them into shattered gradients, stochastic gradients, and vanishing or exploding gradients. In the evaluated cases, adaptive attacks completely circumvented most of the applicable defenses and partially circumvented another under each defense’s original threat model. These results show that unusable or misleading gradients did not establish robustness against adapted attacks in that setting.

The operational question is how an evaluator can detect an attack gradient that has become unusable or misleading and then adapt the attack. The diagnostic checklist flags several suspicious patterns. These include a one-step attack beating an iterative attack, or a black-box attack beating a white-box attack. Other warning signs appear when an unbounded attack fails to reach full success, random search succeeds where gradient-based optimization fails, or larger input distortion does not improve attack success. An adaptive attack is designed after the defense is fully specified and uses all knowledge allowed by the threat model.

An adaptive attack is designed after the defense is fully specified and uses all knowledge allowed by its threat model. BPDA, which stands for Backward Pass Differentiable Approximation, preserves the defense’s real forward computation but substitutes a differentiable approximation for the backward pass. EOT, which stands for Expectation Over Transformation, estimates gradients by sampling the randomized defense and optimizing its expected behavior instead of relying on one random outcome. Reparameterization changes the variables so optimization runs through a differentiable representation rather than an unstable sequence of calculations. As needed, the authors used iterative optimization attacks with 100 to 10,000 gradient-descent iterations.

Across the studied defenses, most relied on obfuscated gradients. Adaptive attacks completely circumvented all but one of those defenses under each defense’s original threat model, and partially circumvented the remaining defense. The result applies to the evaluated defenses rather than establishing how every later system will behave. Its logical force is narrower: a successful attack that stays within the stated threat model is enough to refute a claim that the defense stops every allowed attack.

For untargeted attacks, the evaluation used the complete MNIST and CIFAR-10 test sets. For targeted ImageNet attacks, the authors randomly selected 1,000 test images and assigned random target classes. Against the strongest tested input-transformation defense, at the smallest tested limit on how much the input could change, combining BPDA and EOT achieved 100 percent targeted attack success and left 0 percent defended accuracy. In that combined attack, BPDA supplied a usable backward calculation while EOT averaged behavior across the defense’s randomness.

One defense was only partly bypassed: BPDA achieved a 45 percent attack success rate. The authors attributed that shortfall to an imperfect projection step that failed to preserve every adversarial input within the image patterns the system could represent. The study also excluded defenses making provable security claims and a defense claiming security only against attackers with black-box access. These experiments therefore do not establish that those excluded claims fail, and the partial bypass should not be described as a complete one.

For evaluation teams, the recommended discipline starts with a realistic threat model. Clearly describe the attacker’s access, including what they know and can observe. Account for their ability to query the system and for any randomness in the defense, without imposing an artificial computation limit. Make robustness claims specific and testable. Where possible, publish the complete settings and supporting materials, including source code and pretrained models. When using several attacks, select the strongest result for each image and then average those results, rather than choosing the whole attack with the strongest average. Finally, treat adaptive evaluation like held-out testing: after every defense change, adapt the attack again.

The contribution is a diagnostic and attack toolkit for evaluations affected by obfuscated gradients: warning patterns plus techniques for approximating the backward pass, accounting for randomization, and changing unstable optimization variables. Operationally, model-security engineers and red teams can check those warning signs, choose an applicable attack technique, and adapt again whenever the defense changes. They should not infer that every model defense is ineffective or that the excluded defenses with provable security claims are invalid. The cautious takeaway is that one successful attack within the original threat model can refute a universal defense claim, while defenders must account for every attack that model allows.