MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection. This 2026 work by Yue Wang and colleagues appears on arXiv. They build a common benchmark for detecting malicious Agent Skills and test learned text detectors alongside ready-made Skill scanners. The practical question is whether these tools still separate malicious from benign Skills when they encounter sources excluded from training. By the end, you should understand how the benchmark controls duplicates and reused structure, why random data splits can give an incomplete impression, and what the detector results do—and do not—justify.

An Agent Skill is an installable package that gives a language-model agent reusable capabilities. It may combine natural-language instructions with scripts, templates, resources and service configuration. Once installed, the Skill becomes procedural authority inside the agent’s workflow, making malicious content a distinct trust problem. Existing research collections do not line up cleanly: they contain everything from complete artifacts and constructed attacks to vulnerability records and scanner verdicts, backed by different kinds of evidence. Some repeat identical or formatting-equivalent content, reuse common scaffolding, or assign conflicting labels to the same normalized content. Simply merging their rows would overcount independent coverage and could place reused material on both sides of an evaluation split.

The benchmark addresses two linked questions. First, can these collections be combined without erasing what each artifact, label and evidence type actually means? Second, do detectors generalize to an unseen source without merely treating unfamiliar benign artifacts as attacks? Random splitting is useful as a reference, but it can distribute source conventions and structurally related variants across both training and testing. A held-out-source test keeps an entire source outside training, although provenance, construction, labeling and class composition then change together. The study therefore evaluates malicious detection and benign over-flagging together rather than relying on recall, false alarms or a summary score alone.

Wang and colleagues froze 13 public sources and produced thousands of Skills after normalizing equivalent formatting, with malicious examples outnumbering benign ones. During curation, they preserved the meaning of each source and its evidence while removing exact duplicates and copies that differed only in formatting. They treated broader structural reuse separately from attack meaning and excluded records with conflicting labels. Labeling was conservative: suspicious, vulnerable, dual-use or scanner-flagged records were not automatically treated as malicious ground truth. Threat categories came only from documented source labels and mappings, not guesses based on Skill text. The evaluation covered several learned text detectors and off-the-shelf scanners. It used random splits, splits separating related malicious structures, and held-out-source splits. Source identity and provenance remained available for auditing but were excluded from detector inputs.

Across the learned text models, Macro-F1 was higher with random splits but dropped when whole sources were held out. No evaluated detector combined high malicious recall with a low benign false-positive rate across those held-out sources. Differences in provenance and construction moved together with documentation, labeling and class composition, so the experiment cannot isolate one cause for the decline. It demonstrates a cross-source robustness problem, not proof that any single shortcut produced the result.

On held-out sources, 95.6% of malicious Skills were detected, but 62.4% of benign Skills were wrongly flagged as malicious. That is why a high attack-detection rate cannot be interpreted alone: a detector can appear effective by broadly rejecting unfamiliar artifacts. The off-the-shelf scanners occupied a different unsatisfactory operating range. Configurations with very few benign false alarms detected only a small fraction of malicious Skills, while the more sensitive scanner generated substantial benign false alarms. Across these evaluated configurations, high malicious detection and low benign over-flagging were not achieved together.

These results apply to the benchmark and detector configurations that were evaluated. First, the learned baselines and off-the-shelf scanners inspected static primary Skill artifacts, excluding package-level and runtime behavior. The evaluation did not test adversarial evasion or attack success, and one scanner ran without a language model. Second, the data had coverage constraints. The benign pool favored confidence in its labels over breadth, so it covered fewer sources than the malicious side. Many Core-contributing sources contained only one label and could not support a meaningful two-class held-out-source test. These constraints limit how confidently the findings carry into live environments, and the static text scores are not an upper bound on code-aware, behavioral, runtime or multimodal defenses.

For teams assessing pre-installation screening, the benchmark supports a stricter evaluation recipe. Preserve source and evidence provenance, remove formatting-equivalent duplicates, and keep structurally related malicious examples from being divided between training and testing. Test entire unseen sources and include trustworthy benign Skills drawn from multiple source conventions. Report both the malicious catch rate and benign over-flagging, because either measure alone can hide an unusable operating point. An operational inference, rather than something directly tested here, is to treat a scanner verdict as one signal rather than a final installation decision until the scanner has been evaluated on the Skill sources and benign packages your environment actually uses.

MaliciousSkillBench contributes a traceable, cross-source artifact benchmark and a controlled way to compare learned detectors with existing scanners. Its measurements show that conclusions drawn from random splits weaken when entire sources are withheld: learned models over-flag unfamiliar benign Skills, while scanners that avoid those false alarms lose malicious recall. Detection engineers, agent-platform defenders and software supply-chain teams should therefore demand source-aware testing and two-sided metrics. They should not infer that reliable malicious-Skill detection is solved, or treat these static results as the performance ceiling for runtime and behavior-aware defenses.