
Advancing Coding in Low-Resource Programming Languages

This post describes new model versions that are substantially better at building and migrating software in low-resource programming languages such as COBOL, Fortran, and Visual Basic. These languages have less code and documentation for models to learn from than Python or JavaScript, yet they support critical infrastructure. According to the post, COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions, while Fortran is used in weather forecasting and nuclear reactor safety analysis. Improving AI’s ability to work in these languages could help teams maintain and modernize software that essential services depend on.
The work is guided by feedback from developers and enterprise customers who said coding models often struggle with these languages. Maintaining and modernizing such software is difficult because documentation may be incomplete, original developers may have left, and important business rules may only exist in code. Whether fixing bugs or migrating to another language, teams need to preserve the behavior their systems depend on.
To measure progress, the team developed two evaluations. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in a specified programming language, organizing a codebase, managing state, handling errors, and adding features while preserving existing functionality. Database files serve as a shared interface across languages, and the reference SQLite implementation checks whether the resulting database has expected contents. The COBOL migration evaluation asks models to port COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving behavior, with hidden tests requiring byte-for-byte parity on observable outputs. Successful migration requires understanding both business logic and COBOL-specific behavior such as fixed-point arithmetic that discards excess decimal digits unless rounding is explicitly requested.
On MultiLangBench, GPT-6 Astra achieves an average test pass rate of 87.4% across 20 programming languages at maximum reasoning effort, up from 26.8% for GPT-5.6 Sol, with improvements in every language. Examples include COBOL rising from 10.9% to 72.1%, OCaml from 25.6% to 85.8%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%; Astra scores above 83% in 19 of 20 languages, with COBOL remaining the hardest at 72%. The lower-cost GPT-6 Luna also beats GPT-5.6 Sol in every language, averaging 40.3%, with examples like COBOL from 10.9% to 17.1%, OCaml from 25.6% to 34.1%, Fortran from 21.4% to 31.4%, and Visual Basic from 27.5% to 41.6%. Luna’s estimated average API cost is $0.77 per sample versus $3.48 for GPT-5.6 Sol, approximately 78% lower.
On the COBOL migration evaluation, GPT-6.1 Sol achieves 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol, while costing 39% less. At higher reasoning effort, accuracy reaches 97.9%, close to saturating the evaluation. The post notes that behavior-preserving migration is useful for teams maintaining older systems, though applying it to production software still requires expert review and testing against the system’s business rules and dependencies.
The MultiLangBench evaluation setup, test suite, and per-language results are being released on GitHub. The stated goal is for models to apply software engineering skills across more languages: understanding unfamiliar repositories, planning changes, using build tools, running tests, and debugging failures.


