The Locksmith Loop: A New Approach to Validating AI Code Migration

Migrating legacy COBOL to Java is a nightmare. You need extensive testing, but often lack test data and struggle to cover all corner cases. A new arXiv paper (2607.28271) proposes an agentic test-synthesis method called the "Locksmith Loop" that tackles this head-on. It's an iterative loop that penetrates program branches, performs parity-preserving mutations, and identifies "Locked Paragraphs" to push coverage further.

The paper reports impressive results: across three COBOL-Java case studies (two open-source, one internal production-like), the method reached nearly complete branch coverage on the open-source programs and 91.90% branch coverage on the internal one. The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases.

How It Works

The Locksmith Loop starts by preparing two runtime environments: the COBOL source and the generated Java target. Both are instrumented with mocks and executed off-mainframe on commodity hardware. Then an iterative agentic loop performs two key steps:

  1. Witness Search: Searches over input mocks to penetrate program branches.
  2. Parity-Preserving Mutations: Mutates inputs while ensuring the COBOL and Java outputs remain equivalent.

When routing boundaries are reached, an analyzer identifies a "Locked Paragraph" — a condition preventing deeper exploration. The loop then adjusts to unlock it.

The paper's key innovation is using a deterministic oracle: the original COBOL program itself. This allows automated validation of the AI-generated Java without human-written test oracles.

Results and Metrics

  • Case 1: Open-source program, 430 source lines — reached nearly complete coverage.
  • Case 2: Open-source program, 2,114 source lines — reached nearly complete coverage.
  • Case 3: Internal production-like COBOL program, 4,114 source lines — reached 91.90% branch coverage.

All accepted test cases passed deterministic parity checks, meaning the Java output matched the COBOL output exactly for those inputs.

Why This Matters for Developers

If you've ever dealt with legacy COBOL systems, you know the pain of migration. This method could automate a huge chunk of the validation process, saving weeks of manual testing. It also demonstrates a novel pattern: using the original system as a test oracle for AI-generated code.

Practical Implications

This isn't just for COBOL-to-Java. The concept of using a deterministic oracle to validate AI-generated code could apply to other migration scenarios, such as Python-to-Go or JavaScript-to-TypeScript. The Locksmith Loop's agentic approach could be adapted to other languages and frameworks.

How to Apply This

If you're working on code migration, consider:

  • Instrumenting both source and target environments with mocks to enable off-mainframe testing.
  • Using the original program as a deterministic oracle to validate generated code.
  • Implementing a loop that searches for inputs to penetrate branches, then mutates them to maintain parity.
  • Identifying 'locked paragraphs' — conditions that prevent deeper exploration — and adjusting your search strategy.

Next Steps

The paper is available on arXiv (2607.28271). If you're tackling a migration project, read it and consider how the Locksmith Loop could apply. For tool builders, this approach could be integrated into CI/CD pipelines to automatically validate AI-generated code.

Don't wait for the AI to be perfect. Use the old code as your safety net, and let the loop do the heavy lifting.