Detecting malicious agent skills — scores fall sharply once evaluation splits by source
Reusable skills that extend AI agents can also serve as a distribution channel for malicious behaviour. This paper builds a benchmark for detecting them and shows that detector scores shift substantially with how the data is split.
Paper overview (our summary)
- Field (arXiv category)cs.CR（+1）
- AuthorsYue Wang, Yi Liu, Gelei Deng, et al. (7)
- Submitted2026-08-20
- arXiv ID2608.19901v1
Key points
- Reusable skills that extend AI agents are noted as a direct distribution channel for malicious behaviour.
- Thirteen public sources were consolidated, reducing 8,414 raw records to 7,539 unique identities across 4,588 structural families.
- Learned detectors scored 0.882 to 0.932 under random splits but 0.653 to 0.665 under source-disjoint evaluation.
- The strongest retained 95.6 percent malicious recall while producing a 62.4 percent benign false positive rate on held-out sources.
- Off-the-shelf scanners reduced false positives only at the cost of sharply lower malicious recall.
1Skills as a distribution channel
Reusable packages of instructions are used to add capability to AI agents, sometimes including scripts, resources and service configuration. Convenient as that is, the same channel can distribute malicious behaviour. This site covers elsewhere how an effect on a development or distribution platform reaches outward to what it delivers to.
2Scores move with the split
With the same detectors and the same data, changing how training and evaluation are divided moves scores substantially. A random split can leave content resembling the training data on the evaluation side. Splitting by source leaves the evaluation side holding origins never seen in training. Practice resembles the latter, so numbers from a random split can overstate capability.
3Catching everything against crying wolf
The strongest detector caught 95.6 percent of malicious cases from unseen sources. It also wrongly flagged 62.4 percent of benign ones. In operational terms, nearly all the bad is stopped and so is more than six in ten of the good. The off-the-shelf scanners showed the same tension, reducing false positives only at the cost of much lower malicious recall.
4Measuring both at once
The authors conclude that reliable detection requires both broader cross-source coverage and evaluation measuring attack detection and benign over-flagging together. Look at one alone and either number can be made to look good. The same point about evaluation design driving conclusions appears in other research this site covers. This article is our own summary of public research information and does not warrant its contents.
Why it matters
Detection performance cannot be judged without measuring both missing threats and over-flagging together. Numbers from a random split can overstate capability against origins never seen before.
FAQ
Why do scores move with the split?
What does a 62.4 percent false positive rate mean?
Sources (primary)
Source: arXiv (descriptive metadata is CC0 public domain). Summaries are our own; see arXiv for the original text and PDF.
- arXiv abstract page (original, official)
- PDF (arXiv)
- arXiv ID: 2608.19901