ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

Enterprise framework migration is one of the most expensive and brittle software engineering activities, and existing coding benchmarks don’t capture its real difficulty. Bug fixing and code generation benchmarks show impressive progress, but framework migration demands preserving runtime behavior, adapting build systems, and navigating dependency injection—not just translating source code. ScarfBench exposes this gap by requiring migrated Java applications to actually build, deploy, and pass behavioral validation, not just match a reference implementation.

ScarfBench focuses on migrations across Spring, Jakarta EE, and Quarkus, with 204 migration tasks spanning 34 applications and ~151K lines of code. The current frontier agents achieve less than 10% behavioral success, even though compile success rates are much higher. The benchmark reveals that configuration layers dominate agent effort, that agents are overconfident in self-assessing completion (Claude Code reported 29 successful builds out of 30 whole applications, but only 22 actually built), and that environmental issues like Docker cache inconsistencies and port connectivity problems frequently derail validation even after code migration is largely complete.

The key takeaway for builders is that the hard part of modernization isn’t writing translated Java code—it’s managing the web of dependencies across configuration, infrastructure, and runtime environments. Independent build and test validation remains essential, and agent self-assessment should not be trusted. ScarfBench provides a standardized way to measure progress toward autonomous application modernization, and practitioners should use it to evaluate solutions before deploying in production rather than relying on compile-success rates alone.