TraceLLM automates Ethereum forensics, but 66% accuracy is the honest ceiling

The most interesting AI security paper this month is not about jailbreaks or alignment. It is about Ethereum forensics, and it comes from researchers at the Hong Kong University of Science and Technology (Guangzhou) and the National University of Singapore. Their framework, TraceLLM, uses large language models to turn raw blockchain execution traces into human-readable incident reports, identifying attacker and victim addresses with 85.19% precision across 27 ground-truth cases. The paper, posted to arXiv on September 3, 2025, is the first to build a reproducible benchmark for joint trace-and-code-driven security analysis.

The underlying problem is real and expensive. Ethereum holds more than 63 billion USD in total value locked, according to DefiLlama figures cited in the paper, and over 78 million smart contracts deployed on mainnet. In the past two years alone, 218 attacks on DeFi protocols have caused cumulative losses surpassing 953 million USD, per SlowMist data. When a protocol gets drained, the forensic work is manual: analysts align transaction traces against contract code to reconstruct what happened, who attacked, and which function was exploited. TraceLLM aims to automate that reconstruction with a four-component pipeline: Parser, Detector, Extractor, and Analyzer.

The design choices matter as much as the headline numbers. Existing tools split into two camps: anomaly transaction detection, which flags suspicious transactions but cannot explain attack strategy, and code vulnerability detection, which fails on unverified contracts and rarely shows how a flaw was actually exploited. TraceLLM attacks the gap between them. Its Extractor combines a traditional decompiler with LLM-based refinement to reconstruct contract code when verified source is absent, improving decompilation precision by 8.52% over the widely used Etherscan decompiler. Its Analyzer then uses numerical and semantic trace features to find anomalous execution paths, feeding those paths alongside decompiled code to the LLM for report generation.

The empirical results are honest, which is refreshing. On 27 real-world incidents with expert ground-truth reports, TraceLLM identifies attacker and victim addresses with 85.19% precision and produces automated reports with 70.37% factual precision, beating the best proxy baseline by 25.93% in accuracy. Across 148 real-world Ethereum incidents, the numbers drop: 82.43% precision for attacker/victim identification and 66.22% expert-verified accuracy for full reports. The framework also constructs the first anomaly trace dataset, containing 11,228 execution paths, and identifies 83.92% of anomalous paths automatically.

Here is the take that matters for AI builders: the gap between 70.37% and 66.22% is the real story. TraceLLM performs materially better on curated, ground-truth cases than on the messy long tail of real incidents. That gap is not a bug in this specific framework. It is the structural ceiling of any LLM-based security analysis system that depends on decompiled code, noisy traces, and the model’s ability to reason about both simultaneously. Decompiled code is lossy. Traces are enormous. The LLM is doing the hardest kind of work, multi-source integration under uncertainty, and it does it well enough to be useful, not well enough to be trusted without review.

That distinction matters for how this gets deployed. TraceLLM is not a replacement for human analysts. It is a force multiplier that turns hours of manual trace alignment into minutes of report generation, with the caveat that roughly one in three reports across the generalizable set will contain factual errors. For a security team triaging 148 incidents, that is a genuinely useful tool: it lets analysts prioritize the 66% that are likely accurate and focus human attention on the rest. For anyone hoping to run TraceLLM unattended and act on its output automatically, the 66.22% figure is a warning.

The paper also signals a broader shift in how the AI economy treats security tooling. The same LLM capabilities that power code generation and agentic workflows are being repurposed for defensive forensics, and the economics are compelling. A human analyst reconstructing a single exploit can spend days. TraceLLM generates a structured report in minutes, for the cost of a few API calls. The 953 million USD in DeFi losses over two years is the addressable market. Every percentage point of report accuracy is worth real money to protocols, insurers, and security firms.

There is a policy angle too, though the paper does not develop it. Automated forensic reports will increasingly feed into insurance claims, legal disputes, and regulatory inquiries. A report with 66% factual precision is fine for internal triage. It is not fine as evidence in a court filing or a regulator’s enforcement action, unless the LLM’s reasoning is auditable and its errors are characterized. TraceLLM’s authors do not address liability or evidentiary standards. That gap will be filled by whoever builds the compliance layer on top.

The decompiler improvement deserves its own attention. An 8.52% precision gain over Etherscan’s decompiler, achieved by having an LLM refine the output, is a small but meaningful result with applications beyond blockchain. The same pattern, LLM refinement of lossy tool output, applies to binary analysis, log parsing, and any domain where a deterministic tool produces imperfect intermediates. That is arguably the most transferable contribution in the paper.

What to watch next: whether TraceLLM’s authors release the decompiled-code dataset and the 11,228-path anomaly trace dataset, and whether anyone builds a production service on the framework. The paper establishes a benchmark, which is valuable, but benchmarks only matter when others run against them. The 148-incident evaluation set, curated by hand, is the kind of asset that could become a standard reference for blockchain security LLM research.

The honest closing observation is that TraceLLM’s value is proportional to how honestly its users treat its failure rate. A tool that automates 66% of forensic report generation and flags the other 34% for human review is a real productivity gain for an industry that lost 953 million USD in two years. A tool that automates 100% of report generation and is trusted blindly is a liability. The paper’s authors seem to understand this. Their precision numbers are reported with ground-truth comparisons, and their generalizability results are presented as expert-verified, not as ground truth. That epistemic humility is the rarest feature in the current AI security landscape, and it is worth more than any single accuracy point.