AI finds Ethereum bug, but human review proves it

According to Nikos Baxevanis, who authored the field notes, the primary difficulty lay not in finding bugs, but in verifying their authenticity. Unlike traditional fuzzer tools that provide crash data for quick engineer confirmation, AI agents construct elaborate narratives, including severity ratings and code demonstrations, regardless of whether the bug is real or fabricated.

The Foundation identified three recurring types of false positives: crashes occurring only in test builds with safety checks not present in shipped software; attacks requiring manually planted values that external users cannot deliver; and formal verification proofs that demonstrate trivially true statements.

Furthermore, AI agents demonstrated weakness in identifying exploits that depend on a sequence of individually valid steps, a common method in recent crypto protocol exploits like those seen with Edel Finance and BONK. To address this, the Ethereum Foundation now uses AI to propose suspicious sequences for testing, while retaining traditional testing methods and human oversight for validation.