The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark. Released through Cornell University’s arXiv in 2026, this work is by Jeremy Spence and colleagues. Can AI agents that perform well when source code is available recover program behavior from binaries, and can an evaluation block recognition shortcuts while still reflecting realistic software scale and protection?

Binary software is represented as raw bytes rather than readable source code. Reverse engineering, abbreviated as RE, reconstructs what those bytes make a program do, often before an analyst can assess vulnerabilities or exploits. Benchmark contamination means the target or its design may already be familiar from training data. An agent could then recognize the program and skip much of the actual analysis. This is especially damaging here because even knowledge of the program’s broad purpose can guide interpretation. Real binaries can also span thousands of lines of original code and carry multiple, sometimes custom, anti-analysis layers. A credible evaluation therefore needs unfamiliar targets and realistic complexity together.

SRE-Bench tests the missing bridge between source-code security work and binary analysis. The question is whether an agent can understand an unseen, complex and protected binary instead of relying on memorized structure. Malware, firmware and proprietary applications often reach security teams without source code. If success on source-code benchmarks does not transfer, organizations cannot safely use those scores as a proxy for binary investigation capability.

The researchers authored 19 private programs covering network protocols, firmware, games, file-format parsers and malware. They paired these programs with an in-house anti-analysis suite, compiled and optionally hardened them into different binary instances, and used deterministic graders. They evaluated 5 frontier models through the same Bash-only agent setup. Each attempt ran in an isolated container with standard reverse-engineering tools, but without reference solutions, grader code or scoring feedback. The assignment was defensive remediation: build a cleanup tool that removes the targeted artifacts while preserving benign data.

The strongest model reached a 61.4% per-instance score but fully recovered only 31.5% of the instances. Applying the anti-analysis suite cut that model’s measured capability roughly in half and reduced every weaker model to near zero. Removing symbols was also costly, suggesting that current agents lean more on word-level clues than on reasoning directly from machine instructions.

To check whether the intended benchmark design actually caused the difficulty, the researchers removed one requirement at a time. A small private program built from scratch was fully solved. A deployed program derived from gzip, whose overall design is widely represented in public data, was also fully recovered despite local modifications. Only the private compressor built at real-world scale separated the evaluated models. The researchers’ interpretation is that recognition can substitute for analysis: a private target can still be too easy when it is small, while a realistically sized target can be too familiar when derived from public software. Both scale and unfamiliarity affected the evaluation.

The benchmark contains 19 programs, which the researchers acknowledge is a small pool. A larger collection would produce more precise estimates for each domain, especially firmware. Some runs also produced no gradeable result because models refused tasks framed as cybersecurity work or lost context during long attempts; reported averages cover only gradeable runs. That means not every missing result reflects reverse-engineering ability alone. Finally, the anti-analysis suite does not claim to be unbreakable. Its purpose is to raise analysis cost in measurable increments for evaluation, not to serve as a security product. These boundaries make universal conclusions about binary workloads premature.

Operationally, security teams comparing autonomous analysis tools should evaluate binary work separately instead of inferring it from source-code results. Include stripped and protected builds, because removing names and adding anti-analysis mechanisms sharply reduced performance in this setting. For defensive testing, ask whether the agent can remove malicious artifacts without damaging benign state, not merely whether it can reconstruct original names, types or source-like output. This keeps the evaluation focused on defensive remediation and preserving benign data.

SRE-Bench contributes a contamination-controlled, realistically scaled way to measure agentic reverse engineering using privately authored targets and layered protection. Its results suggest that strong source-code security performance is not yet reliable evidence of binary capability. Detection engineers, malware analysts and teams evaluating autonomous security tools should request results on unseen, protected binaries and examine dependence on symbols. They should not treat one benchmark score as a guarantee across every binary environment, or treat the benchmark’s protection mechanisms as unbreakable defenses.