Overview#
MalwareBench is the first end-to-end, cross-platform evaluation of frontier models on malware reverse engineering.
In each MalwareBench case, a model is given an obfuscated malware binary. It must complete a full investigation of the binary and produce two deliverables: a behavioral analysis and a reusable artifact extractor tested against held-out malware variants. This initial release contains 12 cases across Windows PE, Linux ELF, macOS Mach-O, and Android APK.
Previous benchmarks cover only a narrow slice of models’ potential for economic impact on malware reverse engineering. MalwareBench is designed to capture the broader picture in three ways:
- Task realism. Many benchmarks use artificial targets.[1] For example, CTF and CrackMe tasks require recovering an arbitrary secret from a toy program,[2] while SRE-Bench uses malware exhibiting synthetic, non-harmful behavior.[3] Most of the available corpora of such puzzles are saturated, without necessarily translating to high performance on real-world reverse-engineering tasks. MalwareBench uses real, functional malware, obfuscated using primitives drawn from techniques contributed by malware experts or documented in public catalogs like MITRE ATT&CK.[4]
- Long-horizon, end-to-end workflows. Classification tasks stop at family labels,[5] while decompilation and deobfuscation tasks[6][7] test intermediate steps such as payload recovery. MalwareBench requires models to go beyond these partial results and produce the same final deliverables that human analysts would produce during an incident response engagement. A typical MalwareBench rollout is an order of magnitude longer than rollouts on similar benchmarks[3][8][2], taking several hours and hundreds of millions of total tokens for models to complete.
- Measuring generalization. In other workflows, memorization typically still boosts real-world performance, but in malware reverse engineering, economic value is derived from analyzing never-before-seen malware variants. Any previously deployed malware has been thoroughly catalogued in centralized registries and its detection logic widely distilled into endpoint defenses. Malware encountered by reverse engineers during a live investigation has evaded these filters and is therefore unfamiliar. Thus, using known malware specimens for evaluation, as similar benchmarks do,[8] is an unreliable predictor of economic value. In MalwareBench, we construct malware cases through an attacker-defender curriculum to generate original variants that produce zero detections across all VirusTotal scanners.
While other malware benchmarks are saturated, frontier models perform poorly on MalwareBench. The best-performing model, GPT-6.1 Sol, scores only 7.6% overall (best@3). These results suggest that there is significant room for models to improve at malware analysis, and MalwareBench provides a useful measuring stick for further progress in this area.
Results#
vs
GPT-6.1 Sol²General Analyst¹ · Max4.4%$116.513.5h610.2K195.4M1,2542GPT-6 Astra²General Analyst¹ · xHigh4.3%$250.16.4h215K80.8M5513GPT-6.1 Sol²Codex · Max2.0%$112.67.3h532.9K81.1M5324GPT-6 Astra²Codex · Max1.9%$722.87.2h531.3K110.7M7065
Claude Opus 5.5² ³Claude Code · Max1.1%$158.57.2h437.8K123.5M3206
Gemini 3.8 Flash²Gemini CLI · High1.0%$12.22.4h341.4K94.9M5407
Grok 4.7²Grok Build · xHigh0.4%$69.62.3h318.6K102.2M1458GPT-6 LunaGeneral Analyst¹ · Max0.3%$8.116.3h871.5K346.6M2,3219GPT-6 LunaCodex · Max0.3%$2.23.0h366.9K58.8M395- General Analyst (GA) is our malware reverse-engineering harness. It uses custom context compaction, adds time tracking, and nudges models to refine and resubmit findings. Currently, we evaluate General Analyst only with OpenAI models. On average, models refused more frequently with custom harnesses, making it difficult to evaluate all models using General Analyst.
- If a guardrail or model refusal prevents completion of any rollout, we resample the rollout until one completes without a blocking refusal. Prior to resampling, Claude Opus 5.5 at max effort initially failed to complete due to safeguards 26% of the time, while other models' non-completion rates were low (below 10%), using each model's native harness.
- Claude Opus 5 was automatically selected by Claude Code as a fallback for rollouts blocked by safety filters.
Tasks#
Each task provides the agent with an obfuscated malware sample and a set of questions, evidence requirements, and output schemas. The agent works in a sandboxed environment with binary-analysis tools, internet access, and a fixed wall-clock budget of 20 hours.
The agent must submit findings with supporting evidence in submission.json, reconstruct requested files in recovered/, and write a standalone configuration extractor in extract_config.py. A separate verifier checks the findings and recovered bytes and then tests the extractor on held-out malware variants and benign files.
Case distribution
12 cases| Technique | Windows | Linux | macOS | Android | All |
|---|---|---|---|---|---|
| Embedded PayloadsT1027.009Enterprise | — | ||||
| Data from Local SystemT1005Enterprise | — | ||||
| Environmental KeyingT1480.001Enterprise | — | ||||
| File and Directory DiscoveryT1083Enterprise | — | ||||
| Encrypted/Encoded FileT1027.013Enterprise | — |
| Technique | Windows | Linux | macOS | Android | All |
|---|---|---|---|---|---|
| Ingress Tool TransferT1105Enterprise | — | ||||
| Native APIT1106Enterprise | — | ||||
| Process DiscoveryT1057Enterprise | — | — | |||
| CompressionT1027.015Enterprise | — | — | |||
| Exfiltration Over C2 ChannelT1041Enterprise | — |
Partially encrypts files, adds recovery data, renames targets, and leaves a ransom note.
Steals browser credentials, captures the desktop, checks processes, and retrieves files.
Maps processes and networks, transfers or deletes files, and runs programs remotely.
Profiles macOS through loadable modules, runs shell commands, and persists at login.
Steals messages and contacts, tracks GPS, and captures photos or microphone audio.
Captures the desktop, retrieves files, checks processes, and edits Windows registry data.
References#
-
Quesma. BinaryAudit: An Open-Source Benchmark for Evaluating AI Agents’ Ability to Find Backdoors Hidden in Compiled Binaries. GitHub repository, accessed October 22, 2026.
-
David and Gervais. CrackMeBench: Binary Reverse Engineering for Agents. 2026.
-
Spence et al. The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark. SRE-Bench, 2026.
-
MITRE ATT&CK. Obfuscated Files or Information (T1027). Accessed October 22, 2026.
-
Joyce et al. EMBER2024: A Benchmark Dataset for Holistic Evaluation of Malware Classifiers. KDD, 2025.
-
Gao et al. DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios. Findings of ACL, 2025.
-
Chen, Jin, and Lin. JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation. ACM CCS, 2025.
-
ARIMLABS. Malware Bench. Updated July 29, 2026.
Citation#
Please cite this work as:
@misc{liu2026malwarebench,
author = {Liu, Rex and Li, Maximilian and Mastretta, Andrea and Havaei, Rez and Sun, Michelle and Zhang, Eric},
title = {{MalwareBench: reverse-engineering low-cost, evasive malware into actionable intelligence}},
year = {2026},
url = {https://malwarebench.com/}
}