IBM Research has released ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark for evaluating AI agents on cross-framework migration tasks in Enterprise Java. Modernizing enterprise applications across frameworks is a large and expensive software engineering activity, and existing SE benchmarks focus on bug fixing and code generation rather than the fundamentally different challenge of framework migration, which requires preserving behavior, adapting build systems, and navigating runtime dependencies.

ScarfBench focuses on migrations across three major Java ecosystems: Spring, Jakarta EE, and Quarkus. Unlike benchmarks that compare generated code against reference implementations, ScarfBench evaluates whether migrated applications actually build, deploy, and preserve behavior. The benchmark includes 34 applications, 102 framework implementations, 204 migration tasks, roughly 151K lines of code, approximately 2,000 source and test files, and 1,331 expert-written tests. It includes both focused migration tasks and whole-application migrations.

The paper reports that even the strongest current agents achieve less than 10% behavioral success, illustrating the gap between generating compilable code and preserving application behavior. Compile success consistently exceeds deploy success, which in turn exceeds behavioral success; build success alone significantly overestimates migration quality. Migration difficulty depends strongly on the target framework, with Jakarta EE proving particularly challenging.

Several behavioral findings emerge from analyzing agent performance. First, agents are overconfident: Claude Code reported successful builds for 29 out of 30 whole applications, but only 22 actually built successfully, while the single application classified as failed by the agent ultimately built correctly. This suggests agent self-assessment should not be treated as reliable signal of migration completion — independent build and test validation remains essential. Second, migration is iterative rather than linear: the most frequently visited layers were configuration, web, database, and service, with common transitions between configuration and web, and between service and database, indicating an iterative dependency-resolution process. Third, configuration dominates migration effort; agents repeatedly returned to configuration-related artifacts while resolving framework differences and dependency issues. Fourth, environment and tooling matter: agents struggled with Docker cache inconsistencies, port connectivity problems, and Maven wrapper/build tooling issues — operational concerns that often delayed validation even when source-code migration was largely complete.

The key takeaway is that the biggest challenge in framework modernization is not translating Java code but managing the web of dependencies across configuration, infrastructure, and runtime environments. ScarfBench is designed as an open resource for researchers and practitioners, with dataset, evaluation infrastructure, public leaderboard, documentation, and open-source code available at scarfbench.info.