Towards Evaluating the Robustness of Neural Networks

Towards Evaluating the Robustness of Neural Networks. Nicholas Carlini and David Wagner presented this work at the IEEE Symposium on Security and Privacy in 2017. They examine a deceptively difficult evaluation question: when a neural network appears robust to adversarially altered images, is it genuinely harder to fool, or has the attack simply failed to find nearby adversarial inputs? This episode focuses on how their optimization attacks tested that distinction, what happened to defensive distillation, and why attack failure is not proof of safety.

Here, robustness concerns how much an image must change before the model can be forced into a chosen target class. The researchers tested three ways of measuring pixel changes, labeled L0, L2 and L-infinity. A successful attack establishes an upper bound: it proves the model can be fooled with no more than the distortion the attack found. Failure establishes no matching safe boundary, because a stronger attack might still find a smaller change that works.

Defensive distillation provided the concrete test. An earlier evaluation found that its attack-success rate fell from 95% to 0.5%. Carlini and Wagner investigated whether increasing the defense’s training temperature really increased the L2 change required for an attack, or merely disrupted the calculations used by the earlier attack. That separates measurable robustness from an optimizer failing to get useful guidance.

The researchers built targeted attacks for L0, L2 and L-infinity distance. Their objective balanced pixel distortion against a penalty that pushed the chosen class’s raw output score, called a logit, above the alternatives. They repeatedly searched for a suitable balance and optimized each selected tradeoff. They also transformed the variables to keep pixels within valid bounds.

The evaluation used the first 1,000 test images from both MNIST and CIFAR-10, plus 1,000 correctly classified ImageNet images. Across distilled and undistilled MNIST and CIFAR-10 networks, all three new attacks achieved 100% success. In an extreme ImageNet example, the L-infinity attack forced a chosen target while changing each pixel by no more than a single least-significant bit. These successes establish attack-derived distortion bounds, not the exact smallest possible changes.

The diagnostic measurements pointed to an optimization obstacle. At training temperature 100, the average total magnitude of the model’s raw output scores rose from 5.8 in the undistilled model to 482. Probability outputs saturated and numerical gradients vanished, leaving a naive optimizer with little useful direction even though nearby adversarial inputs could still exist. Separate tests found that demanding greater confidence in the target class made examples optimized against one network more likely to fool another.

None of the three pixel-distance rules perfectly represents whether two images look similar to a person. The experiments also concerned image classifiers, assumed access to the models' internals and constrained changes using those distance rules. They were not end-to-end safety or operational-security tests. A successful attack supplies an upper bound on distortion, while neither success nor failure establishes the exact minimum change required in every setting.

For model-security teams, the operational lesson is to adapt evaluation attacks to the defense. If probability conversion has saturated, inspect raw output scores and search the tradeoff between distortion and reaching the target. Then confirm that final inputs remain valid, and vary target confidence when testing transfer between models. Compute cost also matters. No attack took more than a few minutes per instance, but the L2 and L-infinity attacks were typically 10 to 100 times slower than earlier attacks.

In their evaluated setting, Carlini and Wagner’s attacks turned defensive distillation’s reported 0.5% residual attack rate into 100% success. Their measurements linked the apparent protection to scaled outputs and vanishing gradients. Adversarial-machine-learning engineers and security red teams should treat resistance to one attack as provisional. They should not infer human-perceptual robustness, universal exploitability or end-to-end operational safety from these pixel-distance experiments alone.